LinHES Forums
http://forum.linhes.org/

Using iPodder to download music and videos
http://forum.linhes.org/viewtopic.php?f=6&t=5609		 Page 1 of 3
Author:  alewman [ Sun Aug 07, 2005 3:02 pm ]
Post subject:  Using iPodder to download music and videos
Pretty much ever since I got KnoppMyth up and going, I've been wondering if there was a way that I could easily populate MythVideo with various shows found on the net, and MythMusic with podcasts and similar that I might enjoy listening to from the recliner. Originally, I used Azureus with its RSS plugin to identify and download shows. Unfortunately, it isn't really flexible, not terribly scalable, required signifigant resources, and only worked with torrent feeds. Another solution was Torrentocracy but that required manual intervention in order to download shows, so I never used it for more than testing.

Recently, I discovered that the podcasting client iPodder can download more than just mp3s, and more importantly, that it can be used as a command line application. It is well known that it can download files directly and from torrents, and that it saves the files it gets in well organized directories. This is fairly ideal.

Armed with this information (and a lot of caffeine), this morning I set out to create the mythtv rss aggrigator of my dreams...

The end result of this project provides a way to periodically (via a crontab) download RSS based content into /myth/video and /myth/music. New feeds can be added by adding the rss feed information into a text file.

Before starting, make sure that you have wxpython installed on your system. It is a dependency of iPodder and does not come installed on with R5A16.

Code:
# apt-get update; apt-get install wxpython2.6-0


The first thing to do is install the iPodder software. Since it doesn't appear in Debian's apt-get repository, I grabbed the current copy from sourceforge and installed it into the default location (under /opt).

As root: wrote:
# cd
# wget http://unc.dl.sourceforge.net/sourcefor ... .1.tar.bz2
# tar -xjf iPodder-linux-2.1.tar.bz2
# cd iPodder-linux
# ./install.sh


Accept the defaults and iPodder will be installed in /opt/iPodder

As it comes, iPodder will dump all of the files it gets into one directory. Since we want iPodder to be able to store the downloads into two directories (/myth/video,/myth/music) we will need to tweak its source code a little bit in order to give us the flexibility we need.

The first thing to do is get rid of the default podcasts that are always loaded up when you start iPodder. We don't want the 'welcome' mp3s showing up in myth.

We will need to be root in order to make the following changes. Also, be aware that indentation is VERY important in the python language we will be editing. I don't know how to get this forum to display indents correctly, so just make sure that the code that you change/add is indented the same as code around the changes.

Edit /opt/iPodder/gui/skin.py and look for the section of code that looks like:
/opt/iPodder/gui/skin.py wrote:
DEFAULT_SUBS = [('Default Channel', \
'http://radio.weblogs.com/0001014/categories/ipodderTestChannel/rss.xml'), \
('iPodder News', 'http://ipodder.sf.net/podcasts/ipodder-users.xml')]


and change that to:
/opt/iPodder/gui/skin.py wrote:
DEFAULT_SUBS = []


Now we need to add a few features to the command line version of iPodder. The next thing we do will add a command like option for choosing where the iPodder state file is stored.

Edit: /opt/iPodder/ipodder/configuration.py and look for:
/opt/iPodder/ipodder/configuration.py wrote:
parser.add_option('-f', '--favorites',
dest = 'favorites_file',
action = 'store',
type = 'string',
default = None,
help = "Override: specify which favorites file to use")


and add a new block right below that that looks like:
/opt/iPodder/ipodder/configuration.py wrote:
parser.add_option('-s', '--statedb',
dest = 'state_db_file',
action = 'store',
type = 'string',
default = None,
help = "Override: specify which state_db file to use")


Just because the command line iPodder will allow us to pass in a parameter stating that we'd like to over-ride the the location of the state database doesn't mean it will actually do it. The next change in this same file will tell iPodder to allow the location of the state_db and the favorites_file to be overridable.

In the file, look for a section that starts off like:
/opt/iPodder/ipodder/configuration.py wrote:
configOptions = [
# ('key',default, exposed)
('appdata_dir', None, True),


Beneath this entry, add the following new items:
/opt/iPodder/ipodder/configuration.py wrote:
('favorites_file', None, True),
('state_db_file', None, True),


Now the last thing we need to do is allow these variables to actually be overridden. It turns out that with the way the code is written, even if you
specify different favorites file, iPodder will just use the defaults values anyway.


Therefore, in the same file, a little lower down look for these lines:
Code:
        self.favorites_file = join(appdata, "favorites.txt")
        self.state_db_file = join(appdata, "iPodder.db")


and change them to:
Code:
        if self.favorites_file is None:
            self.favorites_file = join(appdata, "favorites.txt")
        if self.state_db_file is None:
            self.state_db_file = join(appdata, "iPodder.db")


The indentation here is very important. the self.* lines listed should be indented from the new if self.* lines.

Save the file.

One last thing that was discovered is that the players.py file need to be set as world readable so issue this command to make that so:

Code:
# chmod 644 /opt/iPodder/ipodder/players.py


Now that we are done making changes to the iPodder source code, we are ready to try out our modification.

For the rest of this tutorial, change to the mythtv user:
Quote:
# su mythtv
$ cd


get iPodder to create a preference directory for us.
Quote:
$ python /opt/iPodder/iPodder.py


This will create the /home/mythtv/iPodderData directory that will contain preference data for iPodder for our mythtv user.
You may want to view the ipodder.cfg file in that directory and adjust some variables such as max_scan_jobs and max_download_jobs to suite your greedy needs.

Create a preference file called videofeeds.txt that contains urls of all of our video rss feeds, one per line. For example:
/home/mythtv/iPodderData/videofeeds.txt wrote:
# Call for Help Torrents
http://cfh.myownlittleserver.us/bt/rss.php
# commandN XviD
http://feeds.feedburner.com/command-n/xvid
# Revision3 - Systm
http://revision3.com/systm/feed/large.x ... orrent.xml
# Revision3 - Diggnation
http://revision3.com/diggnation/feed/small.mov


Also create one for the music feeds. For example:
/home/mythtv/iPodderData/musicfeeds.txt wrote:
# this WEEK in TECH
http://feeds.feedburner.com/twit
# OCRemix Feed
http://remix.supertux.com/songs/rss.php
# Revision3 - Diggnation
http://revision3.com/diggnation/feed/high.mp3


Test to see if these work by issuing the following commands:
Quote:
$ python /opt/iPodder/iPodder.py --favorites=/home/mythtv/iPodderData/videofeeds.txt --statedb=/home/mythtv/iPodderData/video.db --downloads=/myth/video

$ python /opt/iPodder/iPodder.py --favorites=/home/mythtv/iPodderData/musicfeeds.txt --statedb=/home/mythtv/iPodderData/music.db --downloads=/myth/music


If things are going well, you will have new directories with mp3s and xvids and such in /myth/video and /myth/music. To add new feeds, just reedit the preference files and put in the RSS urls at the end.

Now things are working well enough, we'll have cron schedule iPodder to run a couple of times a day so that it can fetch any new content that comes out. To do that we'll create a new script that will contain all of our iPodder commands. A script will allow us to serialize our iPodder commands so that two instances of iPodder won't run at the same time and try to listen on the same port.

Create a shell script in mythtv's home directory that looks like:
/home/mythtv/runipodder.sh wrote:
#!/bin/sh
#
# This file will sequentially run instances of iPodder
# This is needed because of conflicts with the bit torrent port.

python /opt/iPodder/iPodder.py --favorites=/home/mythtv/iPodderData/videofeeds.txt --statedb=/home/mythtv/iPodderData/video.db --downloads=/myth/video > /dev/null 2>&1
python /opt/iPodder/iPodder.py --favorites=/home/mythtv/iPodderData/musicfeeds.txt -statedb=/home/mythtv/iPodderData/music.db --downloads=/myth/music >> /dev/null 2>&1


If you want to log the results of the iPodder runs to a file, be sure to change /dev/null to the name of the log file.

Make the script executable with the following command:
Quote:
$ chmod 755 /home/mythtv/runipodder.sh


Now we just need to add it to the scheduling system. We'll have our new command execute at 2:00 am and again at 2:00 pm.

Quote:
$ crontab -e


Add this line to mythtv's crontab
Quote:
0 2,14 * * * /home/mythtv/runipodder.sh > /dev/null 2>&1


That's it! The new content will appear automagically with it is released.

That said, there are some things to do to make this solution more useful.

The first thing would be to change the default view of MythVideo entries to "Listings". This will make browsing videos by directory possible in MythVideo. The setting is in: Utilities/Setup->Setup->Media Settings->Videos Settings->General Settings->Default View.

Another thing to do in the same area is to turn off "Show Unknown File Types" as otherwise we will see a lot of .torrent files in MythVideo.

Setting "Video List browses files" on the same screen is useful because this will allow the new content to show up automatically when browsing MythVideo instead of having to wait for it to be manually scanned in like normal.

If you add quicktime feeds, or other feeds that the default mplayer doesn't know how to play, just follow the instructions on this page to build a more functional mplayer that can play those media types.

If you do download more exotic media like quicktimes, you'll want to add those extentions to the filter so those files are displayed. The setting is in: Utilities/Setup->Setup->Media Settings->Videos Settings->File Types.

Since iPodder downloads every media file listed in a feed, you'd need to use a service like http://feedshake.com to create a feed for you based on another feed that is filtered in some way. This would allow you to download say, only "Dr. Who" from a feed that lists dozens of different shows that you don't care about.

And thats about all I can think of right now. Since I just set this up this morning, I haven't had enough time to kick the tires in on everything I've mentioned, but so far, things seem like they should work.

-Aubrey
Author:  alewman [ Wed Aug 10, 2005 3:44 am ]
Post subject: 
Well, I've been using the modified iPodder and have good results to report. It seems to work, and has been downloading new audio and video content on a daily basis automatically since I set it up.

One thing that I didn't make clear was that if you want to delete a feed, you'll need to remove it from the favorites (feed) text file *and* delete the matching database file. If you just delete the feed from the favorites file, iPodder will put it back in the favorites file for you when the next run happens as the information is still stored in the state database file.

I am not happy with using MythMusic with this internet content as before MythMusic can see any of the content, I have to go into the utilities are and scan for new music. MythVideo had an option to scan automatically whenever you launched the module which is nice, but I haven't found the same for MythMusic yet. Having to manually scan for new music isn't a horrible thing, it is just not ideal.

Anyway, I have packaged up the iPodder installer with the modifications mentioned above and put it into this file. If you did want to check this out, you could use this package and install iPodder like normal, then go right into setting up the feeds.

Edit: I've updated the link to fix the players.py issue.

-Aubrey
Author:  chart3 [ Fri Aug 12, 2005 10:29 pm ]
Post subject:  looks interesting
Your progress is interesting. Several weeks ago I started working on using iPodder to do something like what you've accomplished. Unfortunately, my progress to date has been sporadic at best, so I must commend you on your efforts. Have you done any testing on any other OS's or do you have any idea of how well your modifications may perform on other OS's?


Thanks,

Curtis
Author:  alewman [ Fri Aug 12, 2005 11:45 pm ]
Post subject: 
Wow, after 5 days someone finally replied to my experiment!!

I don't think so. The windows version of iPodder is in a differnt format. And from looking at it, the file/directory structure seems to be a bit different as well.

That said, if you knew how to build the windows version of iPodder, I'd think that the code snippets I provided should work fine.

But I know almost nothing about Python. You'd probably get better results from asking about this in the iPodder support forums.

Also, be aware that I used cron to schedule the feed checks. You'd need to come up with a replacement for that on a non unix platform.

-Aubrey
Author:  xtopher [ Sat Aug 13, 2005 11:11 am ]
Post subject:  one thing
first off thanx for doning this i have wanted this for a while, IpTV is awsome.
********edited from the original post after trying pre-configured file.******************

I know nothing about python. i followed your instructions and i am getting an error at the
Code:
python /opt/iPodder/iPodder.py


this is the error
Code:
sh-3.00$ python /opt/iPodder/iPodder.py
Traceback (most recent call last):
  File "/opt/iPodder/iPodder.py", line 38, in ?
    from ipodder import configuration
  File "/opt/iPodder/ipodder/configuration.py", line 15, in ?
    import players
ImportError: No module named players


any ideas?

thanx in advance for any help you can offer.

PS: if you use the code button on the fourm you can include the indents.
Author:  alewman [ Sat Aug 13, 2005 6:38 pm ]
Post subject:  Re: one thing
xtopher wrote:
Code:
sh-3.00$ python /opt/iPodder/iPodder.py
Traceback (most recent call last):
  File "/opt/iPodder/iPodder.py", line 38, in ?
    from ipodder import configuration
  File "/opt/iPodder/ipodder/configuration.py", line 15, in ?
    import players
ImportError: No module named players


This means that your account can't open /opt/iPodder/ipodder/players.py. The way to fix this would be to run this command as root before trying again:

Code:
# chmod 644 /opt/iPodder/ipodder/players.py


I should have run into this problem before, but didn't. It must be because I once ran iPodder as the root user, which COULD see that file, and once the file is read, a matching .pyc file is created that could be read by all users. I think that this problem is a bug with the current release of iPodder.

xtopher wrote:
PS: if you use the code button on the fourm you can include the indents.


Thanks! Ya, I should have seen that before, but thanks for pointing it out!

-Aubrey
Author:  xtopher [ Sat Aug 13, 2005 9:33 pm ]
Post subject: 
thanx for the info but it seems there are more problems. it might just be my box. i have had problems with it before.


new error
Code:
sh-3.00$ python /opt/iPodder/iPodder.py
xmms couldn't be imported
Traceback (most recent call last):
  File "/opt/iPodder/iPodder.py", line 38, in ?
    from ipodder import configuration
  File "/opt/iPodder/ipodder/configuration.py", line 16, in ?
    import feeds
  File "/opt/iPodder/ipodder/feeds.py", line 17, in ?
    from gui.skin import DEFAULT_SUBS
  File "/opt/iPodder/gui/__init__.py", line 2, in ?
    from gui import images
  File "/opt/iPodder/gui/images.py", line 4, in ?
    from wx import ImageFromStream, BitmapFromImage
ImportError: No module named wx

I ran it in root and mythtv same error.
Author:  alewman [ Sun Aug 14, 2005 5:39 pm ]
Post subject: 
What KnoppMyth release are you on? I'm guessing it isn't R5A16.

Anyway, I'd answer you more fully, but I'm currently rebuilding my Myth box with R5A16 because I had a little accident with apt-get upgrade. :-) But I'll give an answer right when I get things squared away again.

-Aubrey
Author:  LaGaffe [ Mon Aug 15, 2005 1:41 am ]
Post subject:  Same error
Hello everybody,

Thanks for your tutorial! I have tried it and got to the same error as xtopher.

Quote:
xmms couldn't be imported
Traceback (most recent call last):
File "/opt/iPodder/iPodder.py", line 38, in ?
from ipodder import configuration
File "/opt/iPodder/ipodder/configuration.py", line 16, in ?
import feeds
File "/opt/iPodder/ipodder/feeds.py", line 17, in ?
from gui.skin import DEFAULT_SUBS
File "/opt/iPodder/gui/__init__.py", line 2, in ?
from gui import images
File "/opt/iPodder/gui/images.py", line 4, in ?
from wx import ImageFromStream, BitmapFromImage
ImportError: No module named wx


I am on Knoppmyth R5A15.1 - I suppose that could be the problem.

Any help would be appreciated.

Thanx

Stefan

Warning: I don't have the slightest clue about Python, I am even spelling it as Phyton every time I try to run a script :?
Author:  alewman [ Mon Aug 15, 2005 3:15 am ]
Post subject: 
Interestingly enough, I ran into this issue with my clean R5A16 install. The solution was to install the wxpython package.

Code:
# apt-get install wxpython2.6-0


That cleared things right up.

Sorry about the issues...

-Aubrey
Author:  LaGaffe [ Mon Aug 15, 2005 5:32 am ]
Post subject:  Working!
That fixed it!
The first files are arriving on my box now.

Thanks again for the tutorial.

Stefan
Author:  xtopher [ Mon Aug 15, 2005 7:54 am ]
Post subject: 
its me.. pain in the but again

new error

Code:
sh-3.00$ /home/mythtv/runipodder.sh                                            xmms couldn't be imported
Successfully loaded config file /home/mythtv/iPodderData/ipodder.cfg
Performing a self-check of the state database...
Self-check complete.
Loaded 0 feeds from the state database.
Loaded 0 new feeds from /home/mythtv/iPodderData/favorites.txt
Bloglines not configured.
No feeds defined! Adding the default feeds.
Wrote 0 entries to /home/mythtv/iPodderData/favorites.txt
Pass #1: downloading feeds and looking for enclosures
Figuring out which feeds to scan...
We have 0 feeds to scan.
Scanning...
Pass #1 ended with 0 enclosures discovered.
Pass #2: downloading enclosures...
/home/mythtv/runipodder.sh: line 7: --favorites=/home/mythtv/iPodderData/videofeeds.txt: No such file or directory
/home/mythtv/runipodder.sh: line 8: syntax error near unexpected token `newline'
/home/mythtv/runipodder.sh: line 8: `--statedb=/home/mythtv/iPodderData/video.db --downloads=/myth/video > ''

any ideas.
Author:  alewman [ Mon Aug 15, 2005 8:41 am ]
Post subject: 
Yes actually.

It looks like the command line used to launch iPodder in the shell script is wrapped into multiple lines. Try putting the command on one long line.

-Aubrey
Author:  benjohnson [ Sun Sep 25, 2005 9:52 am ]
Post subject: 
With the new 2.1.9 ipodder, they moved the command line py file to core.py in the /opt/iPodder/ipodder directory. Unfortunately the didn't seem to actually test this for command line operation. It can't find many of its imports. What I did to fix this was to copy the entire contents of /opt/iPodder/ipodder up one directory to /opt/iPodder/ Then I could run python /opt/iPodder/core.py in a crontab job.

I tried the scheduler, but unfortunately it seems that you have to actually leave iPodder running. I haven't tried using feed manager yet, but it looks like a good way to control your podcasts through podnova, rather then actually running the ipodder gui.

BTW I am running the stock version, I didn't see the need to separate my podcasts into audio and video feeds, cause I don't listen to the podcasts on my myth boxes anyway. I either mount the /myth share on my 'doze box or copy the audio podcasts to my mp3 player.
Author:  kmkittre [ Fri Sep 30, 2005 11:32 am ]
Post subject: 
This is so cool, I was looking to do something like this for systm3 episodes... Thanks for doing all the leg-work alewman, and I'll probably be setting this up this weekend (I also liked your show and podcast selections...)
Page 1 of 3 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/