View unanswered posts    View active topics

All times are UTC - 6 hours





Post new topic Reply to topic  [ 71 posts ] 
Go to page Previous  1, 2, 3, 4, 5  Next

Print view Previous topic   Next topic  
Author Message
Search for:
PostPosted: Wed Oct 19, 2005 10:55 am 
Offline
Joined: Thu Sep 30, 2004 11:29 am
Posts: 2419
Location: Mechanicsburg, PA
Thanks! I was able to successfully test mythlink.sh - dead_wood will have to wait, since I don't think I understand the reasons behind its existence yet. I made one minor change to mythlink.sh and submitted it to cesman, though I do have a few recommendations for you guys. I am not the code police by any means, but I think the recommendations will help others maintain or extend your code more easily.

First, the change: You can just pipe the results of the mysql query into the perl call, instead of saving the results in a file and calling the perl code on the file. E.g.

mysql [yaddayadda] > file
perl -e 'perlcode' file
rm file

=>

mysql [yaddayadda] | perl -e 'perlcode'


Second, I think it's far more maintainable if such a large chunk of perl were in its own file. There are two ways to go about this. 1) split off the current giant perl expression into its own file, e.g. mythlink.pl, and call it from mythlink.sh, or 2) turn mythlink.sh into mythlink.pl and change the few shell commands in mythlink.sh into system calls in perl.

Third, to further increase maintainability, please put
Code:
use strict;
in all perl scripts. This makes it harder to accidentally run into scope-related problems, among other things.

Fourth, consider using more descriptive variable names, e.g. $original_filename vs. $ofn; $human_readable_file_link vs. $nfn. It takes several minutes off the time a new programmer takes to get up to speed on your code.

Fifth, please comment non-obvious code sections like your substitutions within the link name. I don't know many people who can instantly understand long regexps, and because there are sometimes inconsistencies between how expressions work in one languages vs. another, commenting helps the reader a lot.

Sixth, you can escape single quotes instead of resorting to the ascii code for it :) I had to run examples through the code before I knew what \047 was :)

Thanks very much, guys!

_________________
KnoppMyth R5.5
MythiC Dragon v2.0
Join the KnoppMyth Frappr!


Top
 Profile  
 
 Post subject:
PostPosted: Wed Oct 19, 2005 7:11 pm 
Offline
Joined: Sun Jun 12, 2005 10:55 pm
Posts: 3161
Location: Warwick, RI
Hi Human,

Even if it is a "little rough" around the edges, it helps me and I think others now also. We didn't write the original code, however did add the small enhancements. I do appreciate the pointers and will try to use them as guidence in future tweaks.

Basicly, what the mythlink.sh does is read the database for recordings and then make a alternate name.mpg to link to the real .nuv. This is handy in trying to find something as the label used for the .nuv can be daunting to say the least. It also shows up nicely in lineighborhood and the other os can see them too.

Here is a part of what I posted to Marc while he was trying to understand what I was after.
Quote:
Do you follow what I am trying to do? I have found the mythlink script covers all four possiblities of data.

A & B are good
A no database entry * no recorded entry
B database entry * recorded entry

C & D are not good
C database entry * no recorded entry (echo " " > /myth/tv/missing-file-name.nuv)
D no database entry * recorded entry (delete /myth/tv/skipped-file-name.nuv)

End result, a much higher WAF and the pretty files will update properly.


The WAF is because the system stays cleaner and therefore works. She deleates a file it goes away. Any dead wood that builds up is cleaned out so you have the space returned for more recordings. This may take months for some as it depends on the amount of recording / deleting that is done.

Best way to prove it to your self is to setup a record schedule that does a couple dozen recordings with over writes. (expire & rerecord) Let it run for a week or so and then see how many meg of storage is lost that you can't see. By using the pretty, I got the clues as if things didn't match, it would abort making the links. If you run it manually, it would spit out a list that it wasn't happy to see and the rest is history.

The second script, "dead_wood" I coined that from my younger farming days :) Marc created once he understood what I was rambling on about. Being on the causious side (wise) he moves the deadwood into a holding TRASH bin. If after a week you find it really is trash, it is then deleted. One could use the information generated with pretty alone, however because of timing, could delete some files prematurely. (been there - done i)

I feel honored that it is worthy of consideration, but the credit really goes to Marc as he did the work.

By the way, if you run mythlink.sh, you can see what it thinks is deadwood ( ls /tmp/rm-dead-files ) I am going to try and modify it a little more so I can have a log saved in /home/mythtv of what it does. Use Marc's deadwood script for the graceful removel.

Thanks
Mike


Top
 Profile  
 
 Post subject:
PostPosted: Wed Oct 19, 2005 7:38 pm 
Offline
Joined: Thu Sep 30, 2004 11:29 am
Posts: 2419
Location: Mechanicsburg, PA
mjl wrote:
Basicly, what the mythlink.sh does is [...]


Yep, I know :) I did QA on it, so I had to learn how it worked beforehand. It doesn't look like a working version made it into the next public release (that's the reason I asked you guys for an official version), but I've already submitted a fix for the second release. It's basically the version you posted for me, with a pipe instead of a temp file.

I was even thinking of pushing a change to mythweb to link to the "pretty" names of files instead, so people copying the URLs of content on their backend boxes will have human-readable names to work with. Some logistics need to be worked out, of course.

mjl wrote:
Best way to prove it to your self is to setup a record schedule that does a couple dozen recordings with over writes. (expire & rerecord) Let it run for a week or so and then see how many meg of storage is lost that you can't see.


Good plan - I'll try that!

mjl wrote:
I feel honored that it is worthy of consideration, but the credit really goes to Marc as he did the work.


Well done, Marc :)

_________________
KnoppMyth R5.5
MythiC Dragon v2.0
Join the KnoppMyth Frappr!


Top
 Profile  
 
 Post subject:
PostPosted: Wed Oct 19, 2005 11:52 pm 
Offline
Joined: Tue Jan 18, 2005 2:07 am
Posts: 1532
Location: California
Quote:
I was even thinking of pushing a change to mythweb to link to the "pretty" names of files instead, so people copying the URLs of content on their backend boxes will have human-readable names to work with. Some logistics need to be worked out, of course.


Human, the one downside to this change, if I have properly understood it, is that you won't be able to play recently recorded shows until mythlink runs. Since mythlink runs every 30 minutes, you could have to wait up to 29 minutes to start playing a new recording. Not the end of the world, but a bit annoying, espeically if you want to watch a show that is still recording.

An alternative is if myth named the original files in "/myth/tv" using "pretty" rules like the ones used by mythlink. This would eliminate the need for mythlink script all together. Just a thought...


Marc


Top
 Profile  
 
 Post subject:
PostPosted: Thu Oct 20, 2005 6:07 am 
Offline
Joined: Sun Jun 12, 2005 10:55 pm
Posts: 3161
Location: Warwick, RI
Hi & good morning,

Marc, you are correct about the running every 30 minutes, however that is just what is set in the crontab. I have mine 5 ,35 but most recordings would normally start on the hour / half hour so delay window would be less than 5 minutes for normal records. I doubt very seriously that part would be an issue as if the recording was that important, the person(s) would already be on the couch waiting.

Good to see the creative throughts running again :)
Mike


Top
 Profile  
 
 Post subject:
PostPosted: Thu Oct 20, 2005 8:51 am 
Offline
Joined: Thu Sep 30, 2004 11:29 am
Posts: 2419
Location: Mechanicsburg, PA
marc.aronson wrote:
you won't be able to play recently recorded shows until mythlink runs


Currently, yes, but recorded_programs.php can be told to run mythlink.sh before displaying anything. That'll fix it :)

marc.aronson wrote:
An alternative is if myth named the original files in "/myth/tv" using "pretty" rules like the ones used by mythlink.


That would be nice, but it'd require getting the MythTV maintainers to agree to it, and the change would likely be nontrivial in terms of code changes. Even the mythweb idea would require getting the KnoppMyth maintainers to agree, but I think that's more likely to happen.

_________________
KnoppMyth R5.5
MythiC Dragon v2.0
Join the KnoppMyth Frappr!


Top
 Profile  
 
 Post subject:
PostPosted: Sat Nov 12, 2005 1:25 pm 
Offline
Joined: Sun Oct 16, 2005 9:24 pm
Posts: 61
Thanks for the fantastic enhanced mythlink.sh and dead_wood scripts. They are working quite well on my R5A22 install.

However, the dead_wood script has uncovered a bug in the KnoppMyth restore feature. I quite accidently restored the MySQL database which truly messed up my system since the backup was a week old, and although all the recordings were still there, the ones recorded during the preceeding week were not accessible through MythTV. However, the ones that were still listed in MythTV and had not been deleted, are labelled as orphaned recordings by the dead_wood script.

I assume this is actually a problem with KnoppMyth and not the script because KnoppMyth is also treating these recovered recordings strangely - it is ignoring them as far as "storage options" go. For instance, I have set up a recording to save four episodes of a particular show and then delete the oldest when a fifth episode comes up. MythTV is ignoring the presence of the restored recordings and not deleting them when a new show is recorded.

Any ideas?

Chris


Top
 Profile  
 
 Post subject:
PostPosted: Sat Nov 12, 2005 2:57 pm 
Offline
Joined: Tue Jan 18, 2005 2:07 am
Posts: 1532
Location: California
Chris:

First a disclaimer: What I am about to say about Myth is based on inference, as I am not a Myth developer. Having said this, I suspect it is substaintally correct.

What you have encountered is neither a bug in Myth or in the script. Myth's understanding of what is recorded is not based on the "nuv" files in the file system -- it is based on what is recorded in the data base. As long as Myth is controlling both the database and the "/myth/tv" directory, it keeps them in sync. When you restored the data base to a backup that was 1-week old, the data base no longer contained any info on recordings created during that "missing" week. However the data base restore does not impact the "/myt/tv" directory, so the files still physically exist in the filesystem.

The reason that I implemented the script with a "trash can" and an aging process was to allow easy recovery from an undesirable delete. While I had not envisioned this particular scenario happening when I created the scripts, this is exactly the type of protection I was trying to ensure. ie: You do have the option to save the NUV file for replay outside of myth tv.

Finally, I have used PC-based DVR software that was designed to behave the way you are thinking about things. BeyondTV has both a data base and files-in-the-filesystem as Myth does. However, if BeyondTV encounters a file in the directory it manages that is not in the data base, and if the file follows the standard BeyondTV filenaming convention, beyondTV will automatically update the data base so that the file system and the data base remain in sync.

To be clear my obervations are not meant to be a criticism of either system. In the end I prefer Myth to BeyondTV for several reasons and I am not trying to suggest that anyone switch, but I thought you might find this interesting as the folks at BeyondTV were thinking about this issue the same way you are.


Marc


Top
 Profile  
 
 Post subject:
PostPosted: Sat Nov 12, 2005 3:36 pm 
Offline
Joined: Sun Jun 12, 2005 10:55 pm
Posts: 3161
Location: Warwick, RI
Hi Marc,
Nicely stated 8) Miracles always take longer to fix.
Mike


Top
 Profile  
 
 Post subject:
PostPosted: Sat Nov 12, 2005 6:52 pm 
Offline
Joined: Sun Oct 16, 2005 9:24 pm
Posts: 61
Marc:

Thanks for the very clear statement about the behaviours of MythTV and BeyondTV. I guess I'm the one who was not being clear. I'm speaking only about the recordings that were already in the database when I made the backup. Those are the recordings which MythTV still sees and are still on the disc, but that are being treated as orphans by the dead_wood script and not being overwritten by newer recordings as described in my defined MythTV storage options.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Nov 15, 2005 7:37 pm 
Offline
Joined: Tue Jan 18, 2005 2:07 am
Posts: 1532
Location: California
Chris:

My apologies -- I mis-understood the problem. This is indeed puzzeling. It almost sounds like a problem with either the backup or the restore, although I have used both those procedures without problem. (I did use the updated backup/restore scripts posted in the forum, not the ones that come with the original distribution.)

If you can still reproduce the problem, runing the script will create the following files:

/tmp/nuv-db-list
/tmp/nuv-files

If you can get me the contents of those files I may be able to assess what is going on. Also, give me the name of 2 shows, and their corresponding NUV file names, where the problem is happening. Thanks!

Marc


Top
 Profile  
 
 Post subject:
PostPosted: Tue Nov 15, 2005 11:04 pm 
Offline
Joined: Sun Oct 16, 2005 9:24 pm
Posts: 61
Marc:

I've sent you the files you requested by "private message" rather than clutter up this thread with unnecessary garbage. Let me know if you need me to send them again.

Chris


Top
 Profile  
 
 Post subject:
PostPosted: Tue Nov 15, 2005 11:45 pm 
Offline
Joined: Tue Jan 18, 2005 2:07 am
Posts: 1532
Location: California
Chris, that was helpful, but I am going to torment you with a request for more informaton :-).

1. Please PM me a copy of the file "/tmp/mythclean.db".

2. Please PM me a copy of the script you are running.

3. Have you made any changes to the script?

Thanks.

marc


Top
 Profile  
 
 Post subject:
PostPosted: Wed Nov 16, 2005 10:54 pm 
Offline
Joined: Tue Jan 18, 2005 2:07 am
Posts: 1532
Location: California
Mike:

I've been working on the problem that Chris encountered and I think I know what happened. When you re-posted my script, your posting contained the following lines:

Quote:
mysql -uroot mythconverg -B --exec \
"select chanid,DATE_FORMAT(starttime,'%Y%m%d%H%i%s'),DATE_FORMAT(endtime,'
%Y%m%d%H%i%s') \
from recorded;" >/tmp/mythclean.db


The problem here is that you accidently introduced a line break on the second line that was not in my original posting of the script. I suspect this happened as a side-effect of the way you did a copy & paste.

Could I ask you to go back and edit your posting to correct this so that it doesn't effect anyone else in the future? I would do this myself, but I can't edit other people's posts. Probably best if you do a total re-do of the posted script just in any other line breaks were accidently introduced. Thanks!

marc


Top
 Profile  
 
 Post subject:
PostPosted: Thu Nov 17, 2005 6:57 am 
Offline
Joined: Sun Jun 12, 2005 10:55 pm
Posts: 3161
Location: Warwick, RI
Hi Marc,
Never noticed it, sorry but I did go pull the cr out so should be good to go. I even add an alias to /etc/profile to help stop that from occurring when going the other way also.
# echo "alias nano='nano -w'" >> /etc/profile #big help.

Have a great day.
Mike


Top
 Profile  
 

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 71 posts ] 
Go to page Previous  1, 2, 3, 4, 5  Next



All times are UTC - 6 hours




Who is online

Users browsing this forum: No registered users and 18 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
Powered by phpBB® Forum Software © phpBB Group

Theme Created By ceyhansuyu