LinHES Forums
http://forum.linhes.org/

tv_grab_uk_rt Slooooow !
http://forum.linhes.org/viewtopic.php?f=17&t=13008		 Page 1 of 1
Author:  red321 [ Sun Dec 17, 2006 7:14 am ]
Post subject:  tv_grab_uk_rt Slooooow !
Hi all,

I have just started (trying ) to use tv_grab_uk_rt. I have both dvb-s and dvb-t, which cover similar channel lists, but the eit on dvb-s is only now and next, whilst the dvb-t is 7 days. to use both cards and to give prefrence to some of the channels on dvb-s that I have poor reception on dvb-t I need channum, callsign and the epg to be the same for the two channel sets.

There seems to be no way to populate the DVB-s channels with the dvb-t guide data :-(, so I turned to xmltv. Running tv_grab_uk_rt works, but takes forever. I am only grabbing 15 chnnels or so, with 14 days of data. and the run takes about 2 hours ! Worse still, I need to run it twice, once for each videosource (dvb-t and dvb-s).

Anybody else using tv_grab_uk_rt , and got any tips ?
Author:  alien [ Mon Dec 18, 2006 3:08 am ]
Post subject: 
One to two hours is what it takes me to run tv_grab_de_tvtoday.

You might take a look at tv_grab_dvb. I've never used it, but it looks like it is designed to do what you are looking for.

As for the multiple passes, I have never been able to fix it. However, it isn't a big problem for me as grab the data on another PC and then load it from a file on my USB stick, so it doesn't download twice.
Author:  red321 [ Mon Dec 18, 2006 12:25 pm ]
Post subject: 
OK, thanks, sounds like there is nothing wrong with my setup, its just the way it is .....

What would be cool is a way to map channids together, so that they can use the same epg, whatever the source, but That is obviously a new development. 8)
Author:  bruce_s01 [ Tue Dec 19, 2006 7:08 am ]
Post subject: 
Quote:
There seems to be no way to populate the DVB-s channels with the dvb-t guide data Sad, so I turned to xmltv. Running tv_grab_uk_rt works, but takes forever. I am only grabbing 15 chnnels or so, with 14 days of data. and the run takes about 2 hours ! Worse still, I need to run it twice, once for each videosource (dvb-t and dvb-s).


I'm doing a similar thing, I have a channel list for Freeview and a list for my Telewest STB, I'm using XMLTV to populate both of the programme listings to ensure no duplicate entries, with the possibility of programme names being slightly different from each source if I was using DTT EIT info.
This procedure for me does not take too long at all, definitely a lot shorter than 2 hours. What kind of internet connection do you have for MythTV box?

Bruce S.
Author:  red321 [ Thu Dec 21, 2006 2:53 pm ]
Post subject: 
bruce_s01,

I have a good Broadband connection. Problem seemed to be the huge amount of mysql activity that went on to populate all th etables, rather thean the actual download of the rt data. (although that was slower than I would have expected.

just a snapshot of top now is typical
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20902 mythtv 19 10 24340 20m 2100 R 76.7 2.0 16:10.33 tv_grab_uk_rt
4711 root 0 0 112m 77m 5532 S 4.3 7.6 11:34.67 X
17726 mythtv 0 0 153m 84m 18m S 4.0 8.3 30:03.47 firefox-bin
4815 mythtv 0 0 9136 4872 3244 S 0.3 0.5 0:04.69 x-window-manage
4862 mythtv 0 0 271m 38m 14m S 0.3 3.8 1:26.39 java


loads of cpu, been running for the realtime equivelent of 16 mins.

Mythbackend shows

Last mythfilldatabase run started on 2006-12-21 19:22 and is currently running. (it is now 20:54 )

As I use auto shutdown through mythwelcome, it tries to grab EPG before shutdown, but that takes so long to grab the data that it eventually decides it time to record another show before it gets a chance to actually shut down !
Author:  bruce_s01 [ Sat Dec 23, 2006 7:50 am ]
Post subject: 
This is quite strange.
From my mythbox:
Code:
Last mythfilldatabase run started on 2006-12-23 06:06 and ended on 2006-12-23 06:09. Successful.
 There's guide data until 2007-01-05 04:00 (13 days).

While it is running can you run "free" at the command prompt and paste it here. I'm thinking it could be you are running out of memory and having to page out to Swap. From my machine:
Code:
bruce@homenet_pvr:~$ free -m -t
             total       used       free     shared    buffers     cached
Mem:          1008        982         25          0          3        438
-/+ buffers/cache:        541        467
Swap:         1913         77       1836
Total:        2922       1060       1862

Do you use the rest of the network functionality such as MythWeather or the News Feeds? When I was using a Linksys router, I had problems with connectivity reliability, the weather info wouldn't update and news feeds wouldn't download and IIRC the guide data download was slow, though not as slow as yours. When I changed my router, there was an immediate improvement in connection speed. IIRC there was particular feature of the Linux networking stack that doesn't play well with certain routers, this may have been the problem.

Bruce S.
Author:  red321 [ Sat Dec 23, 2006 1:59 pm ]
Post subject: 
Code:
             total       used       free     shared    buffers     cached
Mem:          1008        991         17          0          8        538
-/+ buffers/cache:        443        564
Swap:         1019          0       1019
Total:        2028        991       1037


hmmm. I did however find a problem with my router. That does not seem to have improved anything though.

WHat I do see, which I dont understand, is lots of updates that look the same, but diffrently formatted....


Code:
removing existing program: east.bbc1.bbc.co.uk Flubber 2006-12-30T13:25:00 - 2006-12-30T14:50:00
inserting new program    : east.bbc1.bbc.co.uk Flubber Sat Dec 30 13:25:00 2006 - Sat Dec 30 14:50:00 2006


This coul dbe causing lots of unnecesary mysql action.


Has anyone found a way, even as a cron job, of cloning all the programm data gathered through eit from dvb-t to use on another set of channid's in my case dvb-s channels ?
Author:  bruce_s01 [ Mon Dec 25, 2006 10:38 am ]
Post subject: 
Quote:
Code:
removing existing program: east.bbc1.bbc.co.uk Flubber 2006-12-30T13:25:00 - 2006-12-30T14:50:00
inserting new program    : east.bbc1.bbc.co.uk Flubber Sat Dec 30 13:25:00 2006 - Sat Dec 30 14:50:00 2006

It appears this is something that the uk grabber does, don't ask me why :) but it does seem to generate a lot more activity that it should:
Code:
removing existing program: scotland.bbc2.bbc.co.uk The Battle of the River Plate 2006-12-28T12:35:00 - 2006-12-28T14:30:00
inserting new program    : scotland.bbc2.bbc.co.uk The Battle of the River Plate Thu Dec 28 12:35:00 2006 - Thu Dec 28 14:30:00 2006


What is the spec of your system BTW?

Bruce S.
Author:  red321 [ Mon Dec 25, 2006 11:56 am ]
Post subject: 
updated sig.
Author:  bruce_s01 [ Mon Dec 25, 2006 5:37 pm ]
Post subject: 
Quote:
hmmm. I did however find a problem with my router. That does not seem to have improved anything though.

After you fixed this problem, have you tried Mythweather etc, just to see how they act?

Just to compare (from my /var/log/mythtv/mythbackend.log.1):
Code:
2006-12-25 06:04:38.244 mythbackend: Running mythfilldatabase
2006-12-25 06:04:38.246 Running mythfilldatabase
2006-12-25 06:04:38.812 Using runtime prefix = /usr
2006-12-25 06:04:38.945 New DB connection, total: 1
2006-12-25 06:04:38.990 Connected to database 'mythconverg' at host: localhost
2006-12-25 06:04:39.057 mythfilldatabase: Listings Download Started
2006-12-25 06:04:39.076 New DB connection, total: 2
2006-12-25 06:04:39.077 Connected to database 'mythconverg' at host: localhost
2006-12-25 06:04:40.869 Updating source #1 (Freeview) with grabber tv_grab_uk_rt
2006-12-25 06:04:40.875 ----------------- Start of XMLTV output ----------------
-
2006-12-25 06:04:40.876 New DB connection, total: 3
2006-12-25 06:04:40.877 Connected to database 'mythconverg' at host: localhost
All data is the copyright of the Radio Times website
<http://www.radiotimes.com> and the use of this data is restricted to
personal use only.

finding channels
RT id 2007 (Five Life) not known in channel ids file
RT id 2008 (Five US) not known in channel ids file
channel weekly.filmfour.channel4.com (1041) not seen on site at /usr/bin/tv_grab
_uk_rt line 216.
grabbing
credits element: Wide character in print at /usr/lib/perl/5.8/IO/Handle.pm line
147.
2006-12-25 06:07:18.274 ------------------ End of XMLTV output -----------------
-
2006-12-25 06:07:19.553 New DB connection, total: 4
2006-12-25 06:07:19.554 Connected to database 'mythconverg' at host: localhost
2006-12-25 06:07:23.172 Updating icons for sourceid: 1
2006-12-25 06:07:23.228 New DB connection, total: 5
2006-12-25 06:07:23.234 Connected to database 'mythconverg' at host: localhost
removing existing program: channel4.com Rudolph the Red-Nosed Reindeer and the I
sland of the Misfit Toys 2006-12-25T07:00:00 - 2006-12-25T08:20:00
inserting new program    : channel4.com Rudolph the Red-Nosed Reindeer and the I
sland of the Misfit Toys Mon Dec 25 07:00:00 2006 - Mon Dec 25 08:20:00 2006

2006-12-25 06:07:32.451 New DB connection, total: 6
2006-12-25 06:07:32.452 Connected to database 'mythconverg' at host: localhost
removing existing program: channel4.com Christmas Carol: the Movie 2006-12-25T13
:00:00 - 2006-12-25T14:30:00
inserting new program    : channel4.com Christmas Carol: the Movie Mon Dec 25 13
:00:00 2006 - Mon Dec 25 14:30:00 2006
....lots of updates....
Code:
removing existing program: scottishtv.co.uk Chitty Chitty Bang Bang 2007-01-01T1
6:15:00 - 2007-01-01T18:45:00
inserting new program    : scottishtv.co.uk Chitty Chitty Bang Bang Mon Jan 1 16
:15:00 2007 - Mon Jan 1 18:45:00 2007

Updated programs: 1011  Unchanged programs: 9328
2006-12-25 06:08:56.359 Updating source #2 (Telewest) with grabber tv_grab_uk_rt
2006-12-25 06:08:56.401 ----------------- Start of XMLTV output ----------------
-
All data is the copyright of the Radio Times website
<http://www.radiotimes.com> and the use of this data is restricted to
personal use only.

finding channels
RT id 2007 (Five Life) not known in channel ids file
RT id 2008 (Five US) not known in channel ids file
channel weekly.filmfour.channel4.com (1041) not seen on site at /usr/bin/tv_grab
_uk_rt line 216.
grabbing
credits element: Wide character in print at /usr/lib/perl/5.8/IO/Handle.pm line
147.
2006-12-25 06:09:38.456 mythbackend: Running housekeeping thread
2006-12-25 06:09:38.461 mythfilldatabase still running, skipping checks.
2006-12-25 06:12:29.525 ------------------ End of XMLTV output -----------------
-
2006-12-25 06:12:37.506 Updating icons for sourceid: 2
removing existing program: bravo.co.uk The Negotiator 2006-12-25T21:00:00 - 2006
-12-25T23:45:00
inserting new program    : bravo.co.uk The Negotiator Mon Dec 25 21:00:00 2006 -
 Mon Dec 25 23:45:00 2006
...lots of updates....
Code:
removing existing program: uk-gold.flextech.telewest.co.uk Cast Away 2007-01-05T
23:00:00 - 2007-01-06T01:50:00
inserting new program    : uk-gold.flextech.telewest.co.uk Cast Away Fri Jan 5 2
3:00:00 2007 - Sat Jan 6 01:50:00 2007

2006-12-25 06:14:38.464 mythfilldatabase still running, skipping checks.
Updated programs: 1486  Unchanged programs: 12761
2006-12-25 06:14:43.032 Data fetching complete.
2006-12-25 06:14:43.041 Adjusting program database end times.
2006-12-25 06:14:43.485     0 replacements made
2006-12-25 06:14:43.489 mythfilldatabase: Listings Download Finished
2006-12-25 06:14:43.490 Marking generic episodes.
2006-12-25 06:14:43.899     Found 10
2006-12-25 06:14:43.901 Marking repeats.
2006-12-25 06:14:44.287     Found 0
2006-12-25 06:14:44.287 Unmarking new episode rebroadcast repeats.
2006-12-25 06:14:44.675     Found 0
2006-12-25 06:14:44.798
===============================================================
| Attempting to contact the master backend for rescheduling.  |
| If the master is not running, rescheduling will happen when |
| the master backend is restarted.                            |
===============================================================
2006-12-25 06:14:44.802 Connecting to backend server: 127.0.0.1:6543 (try 1 of 5
)
2006-12-25 06:14:44.809 Using protocol version 26
2006-12-25 06:14:44.814 MainServer::HandleAnnounce Monitor
2006-12-25 06:14:44.816 adding: homenet_pvr as a client (events: 0)
2006-12-25 06:14:44.853 Reschedule requested for id -1.
2006-12-25 06:14:44.862 mythfilldatabase run complete.
2006-12-25 06:14:44.906 Unknown socket closing
2006-12-25 06:14:45.108 Scheduled 45 items in 0.3 = 0.15 match + 0.10 place
2006-12-25 06:14:45.113 scheduler: Scheduled items: Scheduled 45 items in 0.3 =
0.15 match + 0.10 place


You don't seem to be running out of memory with 1Gb :) and your processor is OK, so I'm scratching my head with what's going on. I'm wondering if it's some problem with either your net connection or it's the UK grabber.
BTW my internet connection is running on Telewest 2Mb cable, so it's nothing outstanding.
Author:  tohc1 [ Tue Dec 26, 2006 10:54 am ]
Post subject: 
red321 wrote:
OK, thanks, sounds like there is nothing wrong with my setup, its just the way it is .....

What would be cool is a way to map channids together, so that they can use the same epg, whatever the source, but That is obviously a new development. 8)


Can't help with your fill database problem but if you make sure that all the channel names/ids/xmltv tag are identical for the same channel on each source you should only see only one listing in the program guide.

I have both an analogue tv source and a freeview source and the common channels (ie BBC,ITV and channel four) only show up once in the epg.

You'l probably have to do this all manualy so you may find it easier to use the channel setup screen in mythweb.
Author:  bruce_s01 [ Tue Dec 26, 2006 4:39 pm ]
Post subject: 
Apparently pgr is having similar problems to you in this thread so I'm wondering if you two are having the same problem which has something to do with BBC or ITV regional variations.

I suggested going though and removing and adding XMLTV ID entries from the backend and the .xmltv file(s) in ~/.mythtv/ in turn and running mythfilldatabase to see what happens. To see what XMLTV id is giving problems.

You can use the web interface to change the XMLTV entries for the back end (http://mythbox_ip/mythweb/settings/channels).

HTH,

Bruce S.
Author:  red321 [ Wed Dec 27, 2006 8:01 am ]
Post subject: 
bruce,

You fixed this for me by telling me to edit ~/.mythtv/ !

Doh!

I have been havinng problems with mythtv-setup recently...http://mysettopbox.tv/phpBB2/viewtopic.php?t=12426&start=0&postdays=0&postorder=asc&highlight=

So I had run tv_grab_uk_rt --configure which configures /home/mythtv/.xmltv/tv_grab_uk_rt.conf

I had then manually configured the sources table in mysql (UKTV and SKY in my case) to use tv_grab_uk_rt.....job done :oops:

However running mythfilldatabase --manual actually configures two files
/home/mythtv/.mythtv/UKTV.xmltv
/home/mythtv/.mythtv/SKY.xmltv

Both of these existed from previous runs, The UKTV.xmltv was the full list of available channels, and that was causing the slow performance. :oops:


So when I copied over tv_grab_uk_rt.conf to UKTV.xmltv and SKY.xmltv all was well.
takes about 6 mins for each of the UKTV and SKY runs.

I was put off the scent because one of my other grabbers is tv_grab_nl, which DOES use the /home/mythtv/.xmltv/tv_grab_nl.conf file, even when run through mythfilldatabase . Oh well.. :-)


Thanks for your help !
Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/