View unanswered posts    View active topics

All times are UTC - 6 hours





Post new topic Reply to topic  [ 51 posts ] 
Go to page Previous  1, 2, 3, 4  Next

Print view Previous topic   Next topic  
Author Message
Search for:
 Post subject:
PostPosted: Thu Jan 11, 2007 7:40 pm 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
For step 3 I'd just use a modified form of the manual backup conversion step from the upgrade hints:
Code:
cd /
tar zcvf /myth/backup/savedfiles.tar.gz ./etc ./root

Also, rather than making a seperate file in step 2, just make two backups, and the first one will serve as a fallback if something else goes wrong. The rest look fine. The list of things restored in /etc is purposely pretty darn minimal. A bigger thing might be the stuff in the DB, especially settings.

As for "What would you gain"... I dunno... I'm an upgrading fool. My DB goes back to June 1st, 2004, and predates my current hardware. ;-)


Top
 Profile  
 
PostPosted: Sun May 13, 2007 12:12 pm 
Offline
Joined: Tue Mar 28, 2006 8:26 pm
Posts: 804
Location: Minneapolis, MN
There are times when I run a backup and can't stay in the room to listen for the "backup complete" sounder. In those cases, I would like to manually run the check_backup script when I return to the room so I can be sure it went ok.

Is there a new style of check_backup.sh script to deal with the *.gz backup files? (used in R5E50 and R5F1)

The posted versions of check_backup.sh are coded for the *.bz2 backup files.

Thanks!
Eric

_________________
KnoppMyth R5.5, Asus A8N-VM CSM (nvidia 6150 onboard video), AMD Athlon 64 dual-core 4200+, two 1GB sticks DDR 400, HD-3000 HDTV card, PVR-150 card, Iguanaworks RS-232 IR receiver/transmitter, Pioneer DVR-110 DVD burner


Top
 Profile  
 
PostPosted: Fri Jul 13, 2007 6:32 pm 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
neutron68 wrote:
Is there a new style of check_backup.sh script to deal with the *.gz backup files? (used in R5E50 and R5F1)

See my posting in this thread from Sat Aug 26, 2006 12:36 pm, the checkbackup script has been available out of the box since R5D1.

You can also just run the mythbackup script manually from a root shell session and see the results there.


Top
 Profile  
 
 Post subject: already on our machines?
PostPosted: Sat Jul 14, 2007 9:34 am 
Offline
Joined: Tue Mar 28, 2006 8:26 pm
Posts: 804
Location: Minneapolis, MN
tjc, I believe you talking about the message that says
Quote:
Upcoming in R5D1, the backup and restore scripts now automatically sanity check the results. The builtin sanity checks are much faster than the original rough version posted here, and can also be run manually. The current versions, which can be found in /usr/local/bin are:

If so, then the current scripts are right on our machines now and they should be compatible with the current gz archives.

Thanks for your work and expertise!

_________________
KnoppMyth R5.5, Asus A8N-VM CSM (nvidia 6150 onboard video), AMD Athlon 64 dual-core 4200+, two 1GB sticks DDR 400, HD-3000 HDTV card, PVR-150 card, Iguanaworks RS-232 IR receiver/transmitter, Pioneer DVR-110 DVD burner


Top
 Profile  
 
 Post subject:
PostPosted: Sat Jul 14, 2007 9:55 am 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
The current scripts are compatible with files that are uncompress, compressed with gzip, or compressed with bzip2.

The default compression used is gzip but there is a gzip_to_bzip2.sh script to squeeze the extra 20% out of your backups if you're really tight on space.

Since R5D1 every backup and restore gets checked to make sure that all the files requested which are in the file system end up in the archive on a backup, or all the request files which are in the archive end up in the file system on a restore. The checkbackup and checkrestore scripts just expose this verification mechanism so it can be used separately.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Mar 14, 2008 8:11 pm 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
Updates in the coming release:

- New idle.sh script intended to be used for scheduled backups to avoid having them disrupt recordings or other activity. Click here to get it now
Code:
root@black2:~/scripts/backup# md5sum idle.sh
51cef4ae295a4204438a57565260d9de  idle.sh
root@black2:~/scripts/backup# sha1sum idle.sh
5cb3fc19f2961069071419317c206de339af06b9  idle.sh

See these threads for more details:
http://knoppmyth.net/phpBB2/viewtopic.php?t=17988
http://knoppmyth.net/phpBB2/viewtopic.php?t=18000

- On a similar note, the backup rollover list is now longer to reduce the risk of daily automated backups destroying the last backup from when the system was in a known good state if you're away for a couple weeks.

- Changed the way server restarts are done to be a bit more robust.

- The mybackup and mythrestore scripts now have blacklists of files and directories that can cause problems which they filter out of the supplemental lists.

- Suppressed some harmless messages from tar that seem to panic people.

- Made the mythrestore script warn about ambiguous cases with multiple backup files using different compression. Unlike some of the other warnings, this _should_ worry people since it's one of the more common ways to get a bad restore.

- Made the sanity checking smarter in several ways. In particular DB checking is now less prone to false alarms and handles both ISAM and InnoDB tables.


Top
 Profile  
 
 Post subject: Deep backup script
PostPosted: Sun Mar 16, 2008 11:12 am 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
The following is the backup wrapper script I use before doing any risky experiments on my KM production box. It assumes that you've got idle.sh, that the script is being run as root, that there is a backup disk mounted on /backup which is big enough to mirror the whole contents of my "/" and "/myth" trees, and that the "/backup/root" directory exists (not to be confused with "/root" BTW). Ideally, I'd also shutdown as many of the system daemons as possible to minimize files getting changed while the copy is happening, but generally they're only writing to directories we don't really care about anyway like /var/log.

Code:
root@black2:~/scripts# cat deepbackup.sh
#!/bin/bash

nice /usr/local/bin/idle.sh -t 60 || {
    echo "Deep backup aborted."
    exit 1
}

echo "Cleaning up any lingering junk..."
#find / -name '*.orig' -print | xargs rm
find / -type f -name '*~' -print | xargs rm
find /myth/tmp -type f -print | xargs rm

echo "Making a standard system backup..."
/usr/local/bin/mythbackup || {
    echo "Deep backup aborted."
    exit 1
}

echo "Syncing image to backup disk...."
/usr/bin/rsync -av --delete /myth /backup
/usr/bin/rsync -avx --delete / /backup/root

echo "Deep backup completed."
exit 0


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 13, 2008 12:33 pm 
Offline
Joined: Thu Feb 23, 2006 2:41 pm
Posts: 68
I've installed idle.sh, but not enstated it in any other scripts yet. I just use it to manually get a quick check on the system state.
Right now I'm trying to figure out what has gone wrong with my system since commenting out the bind-address line lots of bad things have happened.
I am also trying to restore to a backup from March very unsuccessfully, so I ran checkbackup to see how it looks, the list is huge, but it ends with this:
Code:
Warning, file lists are not identical!

Checking for the existance of the DB dump file...
Error, missing DB dump - '/myth/backup/mythconverg.sql'

The backup is bad or already out of date!


So, I did a rollback.sh and checkbackup from the terminal which looked a lot more realistic, and then restore system from the gui,
edit: I aslo tried mythrestore from the command line and the output was this:
Code:
No new style backup file found! Trying to do an old style restore.
Doing any needed file updates...
Which doesn't make sense because R5F27 is the sole creator of these backup files-
but I still don't have any cards and the checkbackup looks the same after the restore as it did after the rollback:

Code:
Checking for the existance of the backup tar file...
Error, missing tar file - '/myth/backup/savedfiles.tar'.

Checking for the existance of the DB dump file...
Using file /myth/backup/mythconverg.sql.gz
DB dump file exists. Checking the compression...
Compression looks OK. Checking DB dump contents...
Generating a list of tables and record counts in the DB dump...
Generating a list of tables and record counts in the live DB...
Comparing live versus saved tables...
3,5c3,5
< capturecard 5
< cardinput 0
< channel 244
---
> capturecard 2
> cardinput 3
> channel 239
7,8c7,8
< credits 186006
< customexample 1
---
> credits 176887
> customexample 0
11,12d10
< displayprofilegroups 6
< displayprofiles 246
15d12
< dvdbookmark 0
24,29c21,25
< housekeeping 4
< inputgroup 0
< inuseprograms 0
< jobqueue 0
< jumppoints 104
< keybindings 804
---
> housekeeping 3
> inuseprograms 3
> jobqueue 44
> jumppoints 34
> keybindings 266
46,49c42,45
< oldfind 55
< oldprogram 16026
< oldrecorded 2191
< people 61314
---
> oldfind 136
> oldprogram 14854
> oldrecorded 2352
> people 58950
52c48
< pidcache 237
---
> pidcache 239
54d49
< powerpriority 0
56,58c51,53
< program 89158
< programgenres 141202
< programrating 39620
---
> program 86788
> programgenres 136814
> programrating 37978
60c55
< record 48
---
> record 47
62,70c57,64
< recorded 120
< recordedcredits 111
< recordedfile 0
< recordedmarkup 937
< recordedprogram 121
< recordedrating 38
< recordedseek 578980
< recordingprofiles 49
< recordmatch 808
---
> recorded 137
> recordedcredits 320
> recordedmarkup 1053
> recordedprogram 136
> recordedrating 51
> recordedseek 639061
> recordingprofiles 41
> recordmatch 874
73,74c67
< settings 1177
< storagegroup 1
---
> settings 466
76d68
< upnpmedia 0
80c72
< videometadata 37
---
> videometadata 36
Warning, table lists are not identical!

The backup is bad or already out of date!


I think my backup directory might be useful since there is a discrepancy around savedfiles.tar, I have a feeling this is the root of my restore issue:
Code:
total 342180
-rw-r--r-- 1 root root 14716272 Mar 27 16:03 mythconverg.sql.gz
-rw------- 1 root root 14755956 Mar 27 16:00 mythconverg.sql.gz.1
-rw------- 1 root root 14969465 Feb 22 19:15 mythconverg.sql.gz.3
-rw------- 1 root root 13703903 Feb  2 17:20 mythconverg.sql.gz.4
-rw------- 1 root root 13956019 Feb  2 10:35 mythconverg.sql.gz.5
-rw------- 1 root root 15090311 Jan 24 23:53 mythconverg.sql.gz.6
-rw------- 1 root root  8824125 Dec 30 11:21 mythconverg.sql.gz.7
-rw------- 1 root root 46616157 Mar 27 16:01 savedfiles.tar.gz.1
-rw------- 1 root root 42889643 Feb 22 19:16 savedfiles.tar.gz.3
-rw------- 1 root root 42415863 Feb  2 17:21 savedfiles.tar.gz.4
-rw------- 1 root root 42210775 Feb  2 10:36 savedfiles.tar.gz.5
-rw------- 1 root root 40099209 Jan 24 23:54 savedfiles.tar.gz.6
-rw------- 1 root root 39687270 Jan  1 10:21 savedfiles.tar.gz.7

_________________
Athlon 64 X2 4200+ 2GB Ram
MSI K8N Neo4-F NF4 939
ASUS N EN6200TC256/TD/64M
500gb SATA
1x Plextor PX-M402U
1x HD5000
R5.5 Upgrade from F27


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 13, 2008 1:33 pm 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
Most of your problems seem to stem from the missing savedfiles.tar.gz which explains everything but the difference in the table lists. To get this back into some kind of reasonable state start by removing the mythconverg.sql.gz file and then move the *.1 files to the base name.
Code:
rm mythconverg.sql.gz
mv mythconverg.sql.gz.1 mythconverg.sql.gz
mv savedfiles.tar.gz.1 savedfiles.tar.gz

Also make sure you're doing this as root. If you're using su to become root make sure you use "su -" so it acts like a login shell. This avoids any potential problems with paths and aliases set for the mythtv (or other) user.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 13, 2008 3:14 pm 
Offline
Joined: Thu Feb 23, 2006 2:41 pm
Posts: 68
tjc wrote:
Also make sure you're doing this as root. If you're using su to become root make sure you use "su -" so it acts like a login shell. This avoids any potential problems with paths and aliases set for the mythtv (or other) user.


Thank you, I had not read that piece of advice before. The mythrestore was a little more predictable this time after completing the changes to the backup files you recommended, the last lines look like:
Code:
./var/www/mythweb/.htaccess
/bin/tar: ./etc/asound.conf: Not found in archive
/bin/tar: ./etc/default/aumix: Not found in archive
/bin/tar: ./var/lib/alsa/asound.state: Not found in archive
/bin/tar: Error exit delayed from previous errors
Completed the restore of files.
Starting the DB restore, this can take a while...
Clearing out the existing skeleton...
Recreating the db...
Restoring the data (long)...
Doing any needed db updates...
Completed the DB restore.
Sanity checking your restore...

Checking for the existance of the backup tar file...
Using file /myth/backup/savedfiles.tar.gz
Backup tar file exists. Checking the compression...
Compression looks OK. Checking backup tar file contents...
Generating a list of the backup contents...
/bin/tar: Skipping to next header
/bin/tar: Skipping to next header
/bin/tar: Skipping to next header
/bin/tar: Skipping to next header
/bin/tar: ./etc/asound.conf: Not found in archive
/bin/tar: ./etc/default/aumix: Not found in archive
/bin/tar: ./var/lib/alsa/asound.state: Not found in archive
/bin/tar: Error exit delayed from previous errors
Generating a list of the directory contents...
/usr/bin/find: ./etc/asound.conf: No such file or directory
/usr/bin/find: ./etc/default/aumix: No such file or directory
/usr/bin/find: ./var/lib/alsa/asound.state: No such file or directory
Comparing directory versus backup contents...
Live and saved file lists match.

Checking for the existance of the DB dump file...
Using file /myth/backup/mythconverg.sql.gz
DB dump file exists. Checking the compression...
Compression looks OK. Checking DB dump contents...
Generating a list of tables and record counts in the DB dump...
Generating a list of tables and record counts in the live DB...
Comparing live versus saved tables...
Live and saved table lists match.

Restore passes all checks.
Doing any needed file updates...
Restarting MythTV server: mythbackendNo /usr/bin/mythbackend found running; none killed.
.
.


It seems to have restored appropriately, I have my input cards back, and the mythweb status actually has stuff in the back in the encoder status and schedule boxes.

This was a fresh installed R5F27 without using older versioned backups to upgrade. I don't know that the backup files being off were necessarily linked to the only real thing I've done to the box in a a good while which was commenting out the bind-address to enable a second frontend on the network.
I suppose it's a little late now, how do the backup files get out of sync like that normally? Is there a shortcut finding the log entries that might help you in the future?

Thank you again. Now to enstate a more regular backup routine using...

arri

_________________
Athlon 64 X2 4200+ 2GB Ram
MSI K8N Neo4-F NF4 939
ASUS N EN6200TC256/TD/64M
500gb SATA
1x Plextor PX-M402U
1x HD5000
R5.5 Upgrade from F27


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 13, 2008 4:05 pm 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
Looks like there may be something funky with your tar files. You don't normally see "Skipping to next header" messages unless the file has been damaged somehow. The "not found" messages are the generally ignorable ones. Those three files are optional, however, I don't suppress the warnings because that could hide something nasty.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 13, 2008 5:02 pm 
Offline
Joined: Thu Feb 23, 2006 2:41 pm
Posts: 68
Follow up: I had my fun (shame on me for not doing a backup more often) cleaning up the recordings and such, optimized the database, rebooted, and then ran mythbackup as su - It appears as the the /bin/tar is still a little unhappy.
the last few lines of the backup:
Code:
./var/www/mythweb/.htaccess
/bin/tar: ./var/lib/alsa/asound.state: Cannot stat: No such file or directory
/bin/tar: Error exit delayed from previous errors
Sanity checking your backup...

Checking for the existance of the backup tar file...
Using file /myth/backup/savedfiles.tar.gz
Backup tar file exists. Checking the compression...
Compression looks OK. Checking backup tar file contents...
Generating a list of the backup contents...
/bin/tar: ./var/lib/alsa/asound.state: Not found in archive
/bin/tar: Error exit delayed from previous errors
Generating a list of the directory contents...
/usr/bin/find: ./var/lib/alsa/asound.state: No such file or directory
Comparing directory versus backup contents...
Live and saved file lists match.

Checking for the existance of the DB dump file...
Using file /myth/backup/mythconverg.sql.gz
DB dump file exists. Checking the compression...
Compression looks OK. Checking DB dump contents...
Generating a list of tables and record counts in the DB dump...
Generating a list of tables and record counts in the live DB...
Comparing live versus saved tables...
Live and saved table lists match.

Backup passes all checks.


It does appear the backup went well by this at least:
Code:
root@mythtv:/myth/backup# ls -l
total 384228
-rw------- 1 root root 10726316 Apr 13 15:55 mythconverg.sql.gz
-rw------- 1 root root 14755956 Mar 27 16:00 mythconverg.sql.gz.1
-rw------- 1 root root 14969465 Feb 22 19:15 mythconverg.sql.gz.4
-rw------- 1 root root 13703903 Feb  2 17:20 mythconverg.sql.gz.5
-rw------- 1 root root 13956019 Feb  2 10:35 mythconverg.sql.gz.6
-rw------- 1 root root 15090311 Jan 24 23:53 mythconverg.sql.gz.7
-rw------- 1 root root  8824125 Dec 30 11:21 mythconverg.sql.gz.8
-rw------- 1 root root 46994154 Apr 13 15:56 savedfiles.tar.gz
-rw------- 1 root root 46616157 Mar 27 16:01 savedfiles.tar.gz.1
-rw------- 1 root root 42889643 Feb 22 19:16 savedfiles.tar.gz.4
-rw------- 1 root root 42415863 Feb  2 17:21 savedfiles.tar.gz.5
-rw------- 1 root root 42210775 Feb  2 10:36 savedfiles.tar.gz.6
-rw------- 1 root root 40099209 Jan 24 23:54 savedfiles.tar.gz.7
-rw------- 1 root root 39687270 Jan  1 10:21 savedfiles.tar.gz.8

_________________
Athlon 64 X2 4200+ 2GB Ram
MSI K8N Neo4-F NF4 939
ASUS N EN6200TC256/TD/64M
500gb SATA
1x Plextor PX-M402U
1x HD5000
R5.5 Upgrade from F27


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 13, 2008 5:29 pm 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
That's a harmless warning. The version of the backup script in the next release eats the "/bin/tar: Error exit delayed from previous errors" stuff that needlessly worries most people but leaves the missing file warnings.

You'll notice that the script says "Backup passes all checks." which is what you should really be paying attention to. We've spent a bunch of time over the last few revisions tweaking this so that the final status is as reliable as possible. There is still one well documented issue (a false alarm caused by directories or files replaced with a symlink) but other than that if it complains at the end you should pay attention and if it doesn't you're golden.
arriflex wrote:
Code:
./var/www/mythweb/.htaccess
/bin/tar: ./var/lib/alsa/asound.state: Cannot stat: No such file or directory
/bin/tar: Error exit delayed from previous errors
Sanity checking your backup...

Checking for the existance of the backup tar file...
Using file /myth/backup/savedfiles.tar.gz
Backup tar file exists. Checking the compression...
Compression looks OK. Checking backup tar file contents...
Generating a list of the backup contents...
/bin/tar: ./var/lib/alsa/asound.state: Not found in archive
/bin/tar: Error exit delayed from previous errors
Generating a list of the directory contents...
/usr/bin/find: ./var/lib/alsa/asound.state: No such file or directory
Comparing directory versus backup contents...
Live and saved file lists match.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 13, 2008 5:50 pm 
Offline
Joined: Thu Feb 23, 2006 2:41 pm
Posts: 68
Very informative, thank you for all yours and everyone's hard work.

arri

_________________
Athlon 64 X2 4200+ 2GB Ram
MSI K8N Neo4-F NF4 939
ASUS N EN6200TC256/TD/64M
500gb SATA
1x Plextor PX-M402U
1x HD5000
R5.5 Upgrade from F27


Top
 Profile  
 
PostPosted: Sat Jun 07, 2008 3:12 pm 
Offline
Joined: Thu Mar 25, 2004 11:00 am
Posts: 9551
Location: Arlington, MA
Features of the modern backup and restore scripts:
    - A single unified and inclusive backup file for all file system configuration. The backup consists of two files, the first contains a full dump of the MythConverg database, and the second includes all the configuration files from the file system.

    - Inclusive backup, and Selective restoration of files. This means that the backup tries to gather everything it can in the way of configuration files, even if they're not needed for a default restore, and that the restore script picks and chooses just what it needs. As a result any special configuration changes you needed to make should be available in the backup for you to refer to or restore.

    - Multi-level rollover of backup files that keeps up to 20 old backups. This is intended to facilitate safe automatic scheduled backups. You can freely purge the old backups if you are very tight on disk space, but unless you go insane with the supplemental backup list or fill your home directories with junk, the maximum space used is less than one 30 minute SDTV recording.

    - Customizable supplemental backup and restore lists. This allows you to automatically include extra directories or files in every backup, or automatically have the restore extract additional files.

    - Automatic checking of backup and restore results. Every backup and restore includes automated comparisons to make sure that the database and file system and the contents of the backup files agree.

    - A restore blacklist and exclusions for safety in the face of silly user tricks. People do the dumbest stuff, even when they're been repeatedly and explicitly told not to. When warning signs aren't enough, making the fence higher can help convince them that it's easier and safer to go around the the pasture with the bull in it rather than through it.

    - Utility scripts to help you manage and automate your backups. These include scripts to test your backup and restore results, "rollback" to older backups, recompress your backup files to save space, optimize your database, and check if you can do a backup without interfering with other activities on the machine.

What is backed up and restored by default:

Please note that these lists are inclusive. That means that if you see the name of a directory like ./home in a list it generally includes that directory and everything it contains, files, links, subdirectories (and all of their contents), ... when backing up or restoring files. The exceptions are anything which is blacklisted for safety reasons.

Also note that it is perfectly normal for the scripts to warn about optional files like ./etc/asound.conf or ./etc/default/aumix which are not found.

    - The default (builtin) backup list is:
      ./etc
      ./home
      ./root
      ./var/lib/alsa/asound.state
      ./var/www/mythweb/.htaccess
    - The default (builtin) restore list is:
      ./etc/X11/xorg.conf
      ./etc/apache2/apache2.conf
      ./etc/asound.conf
      ./etc/default/aumix
      ./etc/htdigest
      ./etc/lirc
      ./etc/localtime
      ./etc/mplayer/mplayer.conf
      ./etc/mythtv/modules
      ./etc/timezone
      ./home
      ./root
      ./var/lib/alsa/asound.state
      ./var/www/mythweb/.htaccess
    - The restore blacklist (things which the scripts will refuse to restore for safety reasons) is:
      ./dev
      ./etc/fstab
      ./etc/modules
      ./lib
      ./myth
      ./myth/backup
      ./proc
      ./var/lib
      ./home/mythtv/.Xauthority
      ./root/.Xauthority
The asymmetry here is both intentional and purposeful. Having all of your /etc directory saved and available for reference is good, but blindly restoring all of it would be very bad. The inclusive backup and selective restore approach gives us the best of both worlds, and the extra space used is insignificant compared to even a few minutes of video or an albums worth of music.


Where to find things and what they are:
    - The backups and supplemental lists are stored in the /myth/backup directory.

    - The database dump file /myth/backup/mythconverg.sql.gz is a gzip compressed file of SQL commands which can be run to recreate the database layout and contents.

    - The file system backup file /myth/backup/savedfiles.tar.gz is a gzip compressed tar archive which can be unpacked to restore the saved files either individually or in bulk.

    - The old archived dump and backup files have names with a numeric suffix from ".1" to ".19" appended. For example /myth/backup/mythconverg.sql.gz.1 and /myth/backup/savedfiles.tar.gz.1 would be your most recent archived backup. If you do a new backup those would be renamed with a ".2" suffix before your current backup became ".1". If you do a rollback ".1" becomes current, ".2" becomes ".1" and so on. These "roll off" at both ends.

    - The two optional list files /myth/backup/backup.list and /myth/backup/restore.list are configuration files which contain the names of any files or directories the user wishes to have backed up or restored in addition to the builtin default lists.

    - The scripts themselves are kept in /usr/ocal/bin along with most of the other KnoppMyth custom utilities. They are:

      /usr/local/bin/backupcommon - library of common settings and funcions
      /usr/local/bin/checkbackup - utility to check a backup
      /usr/local/bin/checkrestore - utility to check a restore
      /usr/local/bin/file_editor.py - library for file editing
      /usr/local/bin/gzip_to_bzip2.sh - utility to recompress files from gzip to bzip2 format. Can reduce the size of the compressed files by ~10-20% at the cost of a lot of time and CPU power.
      /usr/local/bin/idle.sh - utility to check for other activity on the system
      /usr/local/bin/mythbackup - the main backup script
      /usr/local/bin/mythrestore - the main restore script
      /usr/local/bin/optimize_db.sh - utility to optimize and repair your database tables.
      /usr/local/bin/restore_fixups.sh - utility used to make certain automatic updates during the restore
      /usr/local/bin/rollback.sh - utility to discard the current backup and make the most recent archived backup current.

Using the supplemental backup and restore lists:

The backup and restore scripts allow you to specify additional files or directories to be included in your backup or restore, beyond what is already specified by the builtin lists. This is done using the two optional list files "/myth/backup/backup.list" and "/myth/backup/restore.list" described above.

    - These files do not exist unless you create them. If one or both of the files doesn't exist, no additional entries are added to the corresponding list.

    - The files should only contain the names of anything you want to add to the builtin lists for backup or restore.

    - The list files should be owned by and only writable for root.

    - Each of the files is a whitespace separated list containing the names of any additional objects to be backed up or restored when the corresponding script is run. The convention is to list one object per line.

    - All the entries in these lists should start with a "./" followed by the full path name. This is a traditional safety mechanism which allows you to unpack a root directory based tarball under another directory without unintended consequences.

    - Wildcards are not currently supported.

    - Avoid redundant and blacklisted entries. They will only cause confusion and generate strange warning messages. Putting "./etc/fstab" in you backup.list is silly because the default list already includes "./etc" and everything under it. Putting it in your restore.list is silly because it's blacklisted for good reason.

    - Whenever you add things to the restore.list you run the risk of blowing away a modern working (if not optimal) version of some file with a stale one. It is far better to error to the side of caution and extract the file later manually. There are directions for doing this safely below.

Given the default backup list described above, about the only things extra I've found a need to add to the backup are some custom scripts in the "/usr/local/bin" directory. As a result my current backup.list file looks like this:
Code:
root@black2:~# cat /myth/backup/backup.list
./usr/local/bin

Please note - Because they are already included in the backup by the default list, there is no need to specifically mention any files under the /etc, /root or /home directories.

Being selective means that the set of extras I add to the restore is a bit longer. My current list looks like this:
Code:
root@black2:~# cat /myth/backup/restore.list
./etc/exports
./etc/hosts
./usr/local/bin/category_fixer.sh
./usr/local/bin/copy_channel.sh
./usr/local/bin/update_database.sh
./usr/local/bin/deepbackup.sh
./usr/local/bin/settings.sh

Again when it comes to adding things to the restore list, it is much better to error to the side of caution. You can always extract files from the backup later, using something like this sequence of commands:
Code:
cd /
mv /etc/fstab /etc/fstab.installed
tar zxvf /myth/backup/savedfiles.tar.gz ./etc/fstab

You can even create a simple general purpose script for that.

Things to keep in mind:
    - The backup and restore scripts require and try to enforce an "idle" system state while they're running, but there are any number of ways for people to violate this condition. Please make sure you disable any "auto restart" scripts or remote systems which access the DB or file system to prevent conflicts.

    - /root and /home/mythtv should be fairly clean or the backups can get very large. For example, having a Subversion tree under one of these can cause serious backup bloat. This is true for any backup system, but keeping a rollover stack makes it that much more important.

    - Be careful about what you put in the supplemental backup and restore lists. Both in the size of the lists and their content. For example while restoring ./etc/hosts is probably quite safe, restoring all of ./etc is a really bad idea. Backing up ./usr/local/bin is perfectly reasonable, but trying to backup ./myth with these scripts would be amazingly, incredibly, spectacularly bad (because the backup files are written to /myth/backup and it would cause a feedback loop). The scripts do their best to prevent horrors like this but anyone sufficiently stubborn and crazy could thwart their efforts.

    - The backup uses gzip compression for new files, but will also automatically recognize and handle files which are uncompressed, or which use bzip2 compression. The user must be careful that this doesn't lead to ambiguity. Having more than one of "mythconverg.sql" "mythconverg.sql.bz2", and "mythconverg.sql.gz" means that the script has to guess which one to use, and may get it wrong. The order of preference is ".gz", followed by ".bz2", and finally "". While mythrestore checks for and warns about this, you really don't want to find out about it like that.

    - The scripts try to provide progress messages for any steps which can take a long time, but don't panic if something takes a few minutes. On the other hand, unless your hardware is very slow, your database is huge, or you've got an incredible amount of junk in your home directory, anything over 15 minutes should raise an eyebrow.

    - The scripts were designed and written with simplicity, robustness, speed, predictability and ease of description as guiding principles, and the design trade-offs follow. The script don't have any command line options to keep usage simple. They're list driven because lists are simple to set up and explain. They use gzip compression because it's much. much faster than bzip2. They back up everything and let restore sort it out because that's both simpler and more robust. The goal was to have a couple more moving pieces than a large rock, and we've only added more complicated behavior (like automated compression selection) when absolutely necessary.

    - Always verify the results of your backup! I strongly recommend making critical backups (such as the one for an upgrade) from the command line as root so that there is no questions about whether it succeeded or if there were any errors.

    - The scripts aren't magical. For upgrades:

      - If you don't do a backup there won't be anything to restore.

      - You can't do a successful upgrade without a valid and up to date backup.

      - The less time between when you make the backup and when you actually do the upgrade, the better. If the delay is more than a couple hours, you should redo the verification step to make sure that the backup is still current.

      - If you don't have the /myth partition mounted before entering the 2nd stage install (because you forgot to update /etc/fstab or reconfigure an LVM volume and mount it), the restore script won't be able to find the backup files.

      - If you can't use the "Auto Upgrade" option of the installer, or fake it by creating the appropriate semaphore file (/home/mythtv/.upgrade) after using the manual option, the 2nd stage setup won't run the restore automatically. (BTW - you can still do it manually, but some of the post restore configuration won't get done).

Converting an old style backup (from KM version R5B7 or before):

Please note that all the commands shown here must be run as root. See this wiki page for instructions on how to BecomeRoot.

    - Make sure you don't have an old "/myth/backup/savedfiles.tar.bz2" or "/myth/backup/savedfiles.tar.gz" file.
    Code:
    rm -f /myth/backup/savedfiles.tar*


    - Make an old style backup using the old mythbackup script installed on your system. Again I strongly recommend doing this from the command line rather than through the menus. This will help you ensure that there aren't any undetected failures which will cause you grief later.
    Code:
    mythbackup


    - Make a new style tarball. Doing this effectively converts an old style backup to a new style backup. Note that the leading ./ on these three directory names is important. Don't forget it, or your backup/restore won't work properly!
    Code:
    cd /
    tar zcvf /myth/backup/savedfiles.tar.gz ./etc ./root ./home


    - Check that the backup worked! Get the most recent check_backup.sh script from the MythBackupAndRestoreHowTo page on the wiki. and run it:
    Code:
    check_backup.sh


    - Run the following command to check that you don't have multiple compression formats for a given backup filename which will cause confusion later:
    Code:
    ls -l /myth/backup/savedfiles.tar* /myth/backup/mythconverg.sql* |
     sed '/\.[0-9]$/d'

    It's OK if the savedfiles.tar and mythconverg.sql compression (or lack thereof) are different, however, the files should both show the same current date. On my system it produces these healthy results:
    Code:
    root@black2:/# ls -l /myth/backup/savedfiles.tar* /myth/backup/mythconverg.sql* | sed '/\.[0-9]$/d'
    -rw------- 1 root root 13715983 Dec 30 14:20 /myth/backup/mythconverg.sql.gz
    -rw------- 1 root root  7958370 Dec 30 14:21 /myth/backup/savedfiles.tar.gz


    - Optional! Setup your supplemental backup and restore lists as described above.


Last edited by tjc on Sat Jun 28, 2008 10:48 pm, edited 3 times in total.


Top
 Profile  
 

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 51 posts ] 
Go to page Previous  1, 2, 3, 4  Next



All times are UTC - 6 hours




Who is online

Users browsing this forum: Google [Bot] and 18 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group

Theme Created By ceyhansuyu