View unanswered posts    View active topics

All times are UTC - 6 hours





Post new topic Reply to topic  [ 12 posts ] 
Print view Previous topic   Next topic  
Author Message
Search for:
PostPosted: Mon Dec 16, 2019 12:55 pm 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
I've recently tried switching to the SD JSON grabber. Every time I run it I see the following:

Code:
WARNING: Perl module JSON::XS not installed.  JSON encode/decode performance will be poor.
I've been away from linux for a few years, but I thought modules could be installed with CPAN. However, this is what happens when I try to install the module:
Code:
cpan[1]> install JSON::XS
Reading '/home/mythtv/.cpan/Metadata'
  Database was generated on Thu, 12 Dec 2019 13:41:03 GMT
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
Reading '/home/mythtv/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................DONE
Fetching with LWP:
http://www.cpan.org/modules/02packages.details.txt.gz
Reading '/home/mythtv/.cpan/sources/modules/02packages.details.txt.gz'
  Database was generated on Sat, 14 Dec 2019 13:55:11 GMT

Fetching with LWP:
http://www.cpan.org/modules/03modlist.data.gz
Reading '/home/mythtv/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /home/mythtv/.cpan/Metadata
Running install for module 'JSON::XS'
Checksum for /home/mythtv/.cpan/sources/authors/id/M/ML/MLEHMANN/JSON-XS-4.02.tar.gz ok
Scanning cache /home/mythtv/.cpan/build for sizes
............................................................................DONE
'YAML' not installed, will not store persistent state
---- Unsatisfied dependencies detected during ----
----       MLEHMANN/JSON-XS-4.02.tar.gz       ----
    Canary::Stability [build_requires]
Running install for module 'Canary::Stability'
Checksum for /home/mythtv/.cpan/sources/authors/id/M/ML/MLEHMANN/Canary-Stability-2013.tar.gz ok
Configuring M/ML/MLEHMANN/Canary-Stability-2013.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Canary::Stability
Writing MYMETA.yml and MYMETA.json
  MLEHMANN/Canary-Stability-2013.tar.gz
  /usr/bin/perl Makefile.PL INSTALLDIRS=site -- OK
Running make for M/ML/MLEHMANN/Canary-Stability-2013.tar.gz
Can't exec "make": No such file or directory at /usr/share/perl5/core_perl/CPAN/Distribution.pm line 2213.
  MLEHMANN/Canary-Stability-2013.tar.gz
  make -- NOT OK
  No such file or directory
  MLEHMANN/JSON-XS-4.02.tar.gz
  Has already been unwrapped into directory /home/mythtv/.cpan/build/JSON-XS-4.02-1
Warning: Prerequisite 'Canary::Stability => 0' for 'MLEHMANN/JSON-XS-4.02.tar.gz' failed when processing 'MLEHMANN/Canary-Stability-2013.tar.gz' with 'make => NO'. Continuing, but chances to succeed are limited.
Configuring M/ML/MLEHMANN/JSON-XS-4.02.tar.gz with Makefile.PL
Can't locate Canary/Stability.pm in @INC (you may need to install the Canary::Stability module) (@INC contains: /usr/lib/perl5/5.26/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/5.26/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/5.26/core_perl /usr/share/perl5/core_perl .) at Makefile.PL line 4.
BEGIN failed--compilation aborted at Makefile.PL line 4.
Warning: No success on command[/usr/bin/perl Makefile.PL INSTALLDIRS=site]
  MLEHMANN/JSON-XS-4.02.tar.gz
  /usr/bin/perl Makefile.PL INSTALLDIRS=site -- NOT OK
Failed during this command:
 MLEHMANN/Canary-Stability-2013.tar.gz        : make NO
 MLEHMANN/JSON-XS-4.02.tar.gz                 : writemakefile NO '/usr/bin/perl Makefile.PL INSTALLDIRS=site' returned status 512
Can someone point me in the right direction to install this module? If I'm going to be using the JSON grabber, and I think soon we all are, then I'd rather not have "poor" performance every day.

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Mon Dec 16, 2019 4:39 pm 
Offline
Joined: Sat Jan 06, 2007 7:08 pm
Posts: 125
i'm no perl expert, but i poked around a little and it seems like JSON:XS may be provided by the pacman package

Code:
core/perl-list-moreutils-xs


i have this package on my system, and i am not seeing the error you are getting

the contents of the package:

Code:
 find . -ls
8631684566        0 drwx------    6 welner           staff                 192 Dec 16 17:26 .
8631684570        0 drwxr-xr-x    4 welner           staff                 128 May 30  2018 ./usr
8631684571        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/lib
8631684576        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/lib/perl5
8631684577        0 drwxr-xr-x    4 welner           staff                 128 May 30  2018 ./usr/lib/perl5/5.26
8631684579        0 drwxr-xr-x    2 welner           staff                  64 May 30  2018 ./usr/lib/perl5/5.26/core_perl
8631684578        0 drwxr-xr-x    4 welner           staff                 128 May 30  2018 ./usr/lib/perl5/5.26/vendor_perl
8631684581        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/lib/perl5/5.26/vendor_perl/auto
8631684582        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/lib/perl5/5.26/vendor_perl/auto/List
8631684583        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/lib/perl5/5.26/vendor_perl/auto/List/MoreUtils
8631684584        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/lib/perl5/5.26/vendor_perl/auto/List/MoreUtils/XS
8631684585      248 -r-xr-xr-x    1 welner           staff              125176 May 30  2018 ./usr/lib/perl5/5.26/vendor_perl/auto/List/MoreUtils/XS/XS.so
8631684580        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/lib/perl5/5.26/vendor_perl/List
8631684586        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/lib/perl5/5.26/vendor_perl/List/MoreUtils
8631684587        8 -r--r--r--    1 welner           staff                3347 Dec 18  2017 ./usr/lib/perl5/5.26/vendor_perl/List/MoreUtils/XS.pm
8631684572        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/share
8631684573        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/share/man
8631684574        0 drwxr-xr-x    3 welner           staff                  96 May 30  2018 ./usr/share/man/man3
8631684575        8 -r--r--r--    1 welner           staff                2190 May 30  2018 ./usr/share/man/man3/List::MoreUtils::XS.3pm.gz
8631684568       16 -rw-r--r--    1 welner           staff                6426 May 30  2018 ./.BUILDINFO
8631684569        8 -rw-r--r--    1 welner           staff                 696 May 30  2018 ./.MTREE
8631684567        8 -rw-r--r--    1 welner           staff                 418 May 30  2018 ./.PKGINFO
macbookether:perl-list-moreutils-xs-0.428-1-x86_64.pkg

_________________
DH87MC i7-4770 16GB ram Xonar Essence ST geforce 710 LinHes 8.6


Top
 Profile  
 
PostPosted: Tue Dec 17, 2019 9:24 am 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
Thanks for the reply. I have that same package on my system but still receive the error
Code:
WARNING: Perl module JSON::XS not installed.  JSON encode/decode performance will be poor.
I tried reinstalling the perl-list-moreutils-xs package again, but that didn't help. It's a minor issue I guess I'll get around to later.

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Tue Dec 17, 2019 7:21 pm 
Offline
Joined: Thu Sep 27, 2007 5:44 pm
Posts: 580
Where are you seeing the warning? Are you running something from the command line or seeing it in the logs?


Top
 Profile  
 
PostPosted: Wed Dec 18, 2019 8:35 am 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
knappster wrote:
Where are you seeing the warning? Are you running something from the command line or seeing it in the logs?

I've been troubleshooting the JSON grabber and see it in the terminal when I manually run the grabber command. It's not a huge deal, but since the whole mythfilldatabase process is already long and memory-intensive, I wanted to see if improving the JSON performance improved grabbing at all.

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Wed Dec 18, 2019 8:33 pm 
Offline
Joined: Thu Sep 27, 2007 5:44 pm
Posts: 580
I guess I'm not sure that I know how to run the json grabber manually. Maybe you and I are even talking about different things. I believe that mythfilldatabase is still responsible for filling my database, but it is using the tv_grab_zz_sdjson_sqlite from here setup with these instructions:
https://www.mythtv.org/wiki/XMLTV#Sched ... t_Grabbers

I am tuning with antenna, so I don't think I need a lot of efficiency, but I don't see any warnings in my mythfilldatabase log, either.


Top
 Profile  
 
PostPosted: Thu Dec 19, 2019 1:06 pm 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
knappster wrote:
I guess I'm not sure that I know how to run the json grabber manually.


I think we're talking about the same thing. The grabber is run during the instructions you linked with
Code:
tv_grab_zz_sdjson_sqlite --days 0 --config-file $HOME/.mythtv/SD.xmltv
It can also be run without any command line arguments, in which case it will grab the max number of days available. When mythfilldatabase calls it, either there is no log for the grabber itself, or it is going somewhere I haven't found yet. That's why there's nothing in any of the normal logs.

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Fri Dec 20, 2019 6:07 am 
Offline
Joined: Thu Sep 27, 2007 5:44 pm
Posts: 580
Got it. If it makes you feel any better, I get the same warning if I run that command. I only have 18 channels in my database, though, so I'm not sure how valuable a speedup would be on my system. How comfortable are you with building packages in Arch? It looks like this is contained in perl-json-xs:
https://www.archlinux.org/packages/comm ... l-json-xs/

You could request an addition to the repository instead so that you do not taint your system with self-built packages (it's burned me a few times in the past...)


Top
 Profile  
 
PostPosted: Fri Dec 20, 2019 2:41 pm 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
knappster wrote:
Got it. If it makes you feel any better, I get the same warning if I run that command. I only have 18 channels in my database, though, so I'm not sure how valuable a speedup would be on my system. How comfortable are you with building packages in Arch? It looks like this is contained in perl-json-xs:
https://www.archlinux.org/packages/comm ... l-json-xs/

You could request an addition to the repository instead so that you do not taint your system with self-built packages (it's burned me a few times in the past...)
Thanks. I'm not sure it's worth taking the risk that I mess up the system. My 361 channels take almost 2 hours to fully process. Mythfilldatabase reports complete on the Backend Status page within 15 minutes, but the program runs much longer. I was hoping to speed it up a bit, but I'm not convinced this is the best way to do that. I'd probably be better off weeding out the QVC-type and foreign-language channels that I will never be tempted to watch or record.

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 
PostPosted: Sat Dec 21, 2019 5:55 am 
Offline
Joined: Thu Sep 27, 2007 5:44 pm
Posts: 580
Hmmm... I'm not sure how it scales, but my 18 channels takes 4 minutes and it runs 3 times. I think I've got it misconfigured because I have 3 antennas and I thought I'd have to setup a different xml file for each, but I think it's ultimately downloading the same data 3 times and I had to manually delete the unused channels from each source in the backend to achieve what I wanted. Again, for my limited channels it's not a huge performance hit so I just didn't want to mess with it.

Sorry to digress. What I was getting at is that even if it was 4 minutes for 18 channels, it would still trend to 80 minutes for 361 channels. If I'm correct about my inefficiencies, then it would be less than 30 minutes. Are you monitoring the runtime through top, logs, or another method? The information I reported is from my mythfilldatabase log.

The daily task that tends to run the longest for me is myth_mtc. It runs database optimization and data backup and can eat up some resources during this time. Still, it only looks like it takes about 20 minutes to complete and it is terrific to have the automatic backups that it provides, even though I hope to never need them!


Top
 Profile  
 
PostPosted: Sat Dec 21, 2019 11:44 am 
Offline
Joined: Sat Jan 06, 2007 7:08 pm
Posts: 125
i incorrectly stated in a previous post that i was not seeing this error. as i have been following along with the replies, and the commands in question have come out, i see that running 'mythfilldatbase' does not show the error message. if i run:

Code:
tv_grab_zz_sdjson_sqlite --days 0 --config-file /home/mythtv/.mythtv/hdhr0.xmltv


i do indeed see the error that doomedtx is seeing

with 20 channels, it takes less than a second for the above command to complete execution, and mythtfilldatabase takes about 3 minutes to run. i also have a similar configuration as knappster with unique xmltv files for each of the 4 tuners in my hdhomerun tuner. don't know if it is correct or not, but that's what i ended up with after crossing over to the json grabber, and it works well enough.

this caused me to revisit some perl stuff i tried doing several years ago. i attempted to get the logitech media server running on my mythtvbox. i encountered many headaches with perl modules and dependencies. that experience taught me to stay away from modifying perl on mythtv. i ended up building a totally separate perl installation just for the logitech media server to use.

_________________
DH87MC i7-4770 16GB ram Xonar Essence ST geforce 710 LinHes 8.6


Top
 Profile  
 
PostPosted: Mon Dec 23, 2019 11:21 pm 
Offline
Joined: Wed Apr 12, 2006 3:05 pm
Posts: 252
Location: GA, USA
knappster wrote:
Are you monitoring the runtime through top, logs, or another method?


I've been keeping a terminal open with top running to see what was going on. The Backend status page on mythweb reports that mythfilldatabase completed in about 15 minutes, but in top I can see that mythfilldatabase runs for closer to 90 minutes. The load on the machine stays very high during this time. I really only noticed because I was having troubles with the grabber initially, and I was trying to eliminate anything that might have been causing the problem. The grabber seems to be functioning normally, so I'm going to leave it alone and try not to look at the status too much while it's running.

_________________
LinHES R8.6 BE/FE GIGABYTE GA-X48-DS5 / EVGA GeForce 9800 GTX+ / 4GB DDR2 RAM


Top
 Profile  
 

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 


All times are UTC - 6 hours




Who is online

Users browsing this forum: No registered users and 15 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group

Theme Created By ceyhansuyu