 |
knoppmyth.net
The home of KnoppMyth.
|
| View previous topic :: View next topic |
| Author |
Message |
tjc
Joined: 25 Mar 2004
Posts: 9117
Location: Arlington, MA
|
 Posted: Fri May 19, 2006 6:32 pm Post subject: Taking advantage of the enhanced backup and restore scripts |
 |
|
This post is out of date and mainly of historical intrest. Please see the current writeup below http://knoppmyth.net/phpBB2/viewtopic.php?p=112490#112490
Cecil asked me to write up some usage notes about the new backup and restore scripts which are making their public debut in R5C7. These scripts were intended to be drop in replacements for the old ones. As a result they should generally be transparent to folks who don't do any personal backup hacking. For later KnoppMyth releases (R5D1, R5E50) please see the updates notes further down this thread.
Features of the new backup and restore scripts:
- A single unified and inclusive backup file for all filesystem stuff.
- Inclusive backup, and Selective restoration of files. This means that the custom version of /etc/X11/XF86Config-4 or /etc/webmin/miniserv.conf that you worked so hard on is automatically included in the backup.
- Multi-level rollover of backup files that keeps up to 10 old backups. You can freely purge these old backups if space is an issue. At the moment I've got a current backup and 6 sets of rollover files which are using a bit over 250Mb. That's roughly the equivalent of 7 minutes of video capacity.
- User customizable supplemental backup and restore lists. Not only is your custom /etc/X11/XF86Config-4 automatically saved by the backup, but you can have it automatically restored as well.
- Reverts to old style restore behavior if no new style backup is found. If this happens you will hear three beeps instead of the usual two.
What is backed up and restored by default:
- The default (builtin) backup list is: "./root ./home ./etc"
- The default (builtin) restore list is: "./root ./home ./etc/mythtv/modules ./etc/lirc"
Note that the asymmetry here is both intentional and purposeful. Having all of your /etc directory available for reference is good, blindly restoring all of it would be bad. This approach gives us the best of both worlds.
Things to keep in mind:
- /root and /home/mythtv should be fairly clean or the backups can get very large. For example, having a Subversion tree under one of these can cause serious backup bloat. This was already true with the old scripts, but keeping a deeper rollover stack makes it that much more important.
- Be careful about what you put in the supplemental backup and restore lists. Both in the size of the lists and their content. For example while restoring ./etc/hosts is probably quite safe, restoring ./etc/fstab is questionable, and ./etc/modules is probably a pretty bad idea. Trying to backup ./myth with these scripts would be amazingly, incredibly, spectacularly bad (because the backup files are written to /myth/backup and it would cause a feedback loop).
- The supplemental list files should be owned by and only writable for root.
- The compression of the SQL dump file done during rollover takes a fair amount of time. Give it a couple minutes before you assume that it's hung.
- Always check the results of your backup! This really goes for any backup script...
- Remember this isn't that magical  . If you don't do a backup there won't be anything to restore. If you can't use the "Auto Upgrade" option of the installer, or fake it by creating the appropriate semaphore files after using the manual option, the 2nd stage setup won't run the restore automatically (you can do it manually). If you don't have the /myth partition mounted before entering the 2nd stage install (because you forgot to update /etc/fstab or reconfigure an LVM volume and mount it), the restore script won't be able to find the backup files. These were all equally true with the old scripts.
Preparations for the switchover (from KM version R5B7 or before):
- Make sure you don't have an old "/myth/backup/savedfiles.tar.bz2" file.
- Do an old style backup using mythbackup.
- Make a new style tarball. Doing this effectively converts an old style backup to a new style backup. Note that without a new style tarball the restore lists described below don't work, because we just run the old script in that case.
| Code: | cd /
tar jcvf /myth/backup/savedfiles.tar.bz2 ./etc ./root ./home |
- Optional! Setup your supplemental backup and restore lists as described below. Remember, for the restore list to work, the script needs to find a new style unified backup, and the objects listed need to be included in that backup.
Using the supplemental backup and restore lists:
The new scripts allow you to specify additional files or directories to be backed up or restored, beyond what is specified by the builtin lists. This is done using two optional list files "/myth/backup/backup.list" and "/myth/backup/restore.list". These files do not exist unless you create them, and then should only contain the names of anything you want to add to the builtin lists for backup or restore. If one or both of the files doesn't exist, no additional entries are added to the corresponding list.
Each of the files is a whitespace seperated list containing the names of any additional objects to be backed up or restored when the corresponding script is run. The convention is to list one object per line. All the entries in these lists should start with a "./" followed by the full path name. This is a traditional safety mechanism which allows you to unpack a root directory based tarball under another directory without unintended consequences.
As described above, the default (builtin) backup list is "./etc ./root ./home". Given those three directories, about the only things I've found a need to add are personal scripts in the "/usr/local/bin" directory. As a result my current list looks like this:
| Code: | root@black2:~# cat /myth/backup/backup.list
./usr/local/bin/azntv_to_98.sh
./usr/local/bin/update_database.sh
./usr/local/bin/pickATheme.sh
./usr/local/bin/change_tuner0.sh
./usr/local/bin/change_tuner1.sh
./usr/local/bin/delayed_settings.sh |
Please note - Because they are already included in the backup by the default list, there is no need to specifically mention any files under the /etc, /root or /home directories.
As described above, the default (builtin) restore list is "./root ./home ./etc/mythtv/modules ./etc/lirc". This is intended to be as close as possible to what the old style backup and restore did. There are a number of additional files which I find it is generally safe and helpful to restore. As a result my current list looks like this:
| Code: | root@black2:~# cat /myth/backup/restore.list
./etc/X11/XF86Config-4
./etc/hosts
./etc/lvm/lvm.conf
./etc/webmin/miniserv.conf
./etc/exports
./usr/local/bin/azntv_to_98.sh
./usr/local/bin/update_database.sh
./usr/local/bin/pickATheme.sh
./usr/local/bin/change_tuner0.sh
./usr/local/bin/change_tuner1.sh
./usr/local/bin/delayed_settings.sh |
When it comes to adding things to the restore list, it is far better to error to the side of caution. You can always get the files out of the backup bundle later, using something like this sequence of commands:
| Code: | cd /
mv /etc/fstab /etc/fstab.installed
tar jxvf /myth/backup/savedfiles.tar.bz2 ./etc/fstab |
Going backwards:
If you need to revert from a release which uses the new style backup to a previous one which does not:
- The old style backup scripts should still be available as "/usr/local/bin/mythbackup.old". You should run that to get a backup compatible with the old style restore found in previous releases.
- Failing that, the new single file backup actually contains everything that the old style files did. You can use this command to extract "the usual stuff" from it.
| Code: | cd /
tar jxvf /myth/backup/savedfiles.tar.bz2 ./root ./home ./etc/mythtv/modules |
- The DB dump/restore process is identical, barring known content and format discontinuities like the "one way" upgrade to the MythTV 0.19 DB layout.
Last edited by tjc on Sat Jun 07, 2008 2:31 pm; edited 10 times in total |
|
| Back to top |
|
 |
tjc
Joined: 25 Mar 2004
Posts: 9117
Location: Arlington, MA
|
| Posted: Fri May 19, 2006 7:00 pm Post subject: |
|
|
Attached is a script called "check_backup.sh" for verifying that a new style backup looks healthy and complete. It provides a fairly high degree of confidence that your freshly made backup really did what was expected and contains more or less what it should.  I recommend running it either immediately after you backup or before you reboot to start an upgrade. Hopefully it will help you avoid that sinking feeling at the end of an upgrade, when you realize that your backup failed horribly without you noticing, and there isn't anything you can do about it but cry...  If you cut and paste this from here please make sure you don't end up with spaces after the '\' line continuations.
(Note: In R5D1 and beyond this type of checking is done automatically everytime you backup or restore.)
| Code: | #!/bin/bash
#----------------------------------------------------------------------------
DATABASE="mythconverg"
DATABASE_DIR="/var/lib/mysql/$DATABASE"
BACKUP_LIST="./root ./home ./etc"
BACKUP_DIR="/myth/backup"
BACKUP_EXTRAS="$BACKUP_DIR/backup.list"
BACKUP_TAR="$BACKUP_DIR/savedfiles.tar.bz2"
BACKUP_SQL="$BACKUP_DIR/$DATABASE.sql"
#----------------------------------------------------------------------------
[ -f "$BACKUP_EXTRAS" ] &&
BACKUP_LIST="$BACKUP_LIST $(tr '\n' ' ' <$BACKUP_EXTRAS)"
echo "Checking for the existance of the backup files..."
[ -f "$BACKUP_TAR" ] ||
{ echo "Error, missing tar file - '$BACKUP_TAR'." ; exit 1 ; }
[ -f "$BACKUP_SQL" ] ||
{ echo "Error, missing db dump - '$BACKUP_SQL'." ; exit 1 ; }
echo "Both files exist."
echo "Checking the integrity of the tar file compression..."
bzip2 --test $BACKUP_TAR ||
{ echo "Error, bad compressed tarball - '$BACKUP_TAR'" ; exit 1 ; }
echo "Compression looks OK."
echo
STATUS=0
LIVE_FILES=/tmp/live_files_$$
SAVED_FILES=/tmp/saved_files_$$
echo "Generating a list of the backup contents..."
tar jtf /myth/backup/savedfiles.tar.bz2 $BACKUP_LIST | sed '/\/$/s///' |
sort -u >$SAVED_FILES
echo "Generating a list of the directory contents..."
cd /
find $BACKUP_LIST \( -type d -or -type f -or -type l \) -print |
sort -u >$LIVE_FILES
echo "Comparing directory versus backup contents..."
if diff $LIVE_FILES $SAVED_FILES ; then
echo "Live and saved file lists match."
else
echo
echo "Warning, file lists are not identical!"
STATUS=1
fi
rm $LIVE_FILES $SAVED_FILES
echo
LIVE_TABLES=/tmp/live_tables_$$
SAVED_TABLES=/tmp/saved_tables_$$
echo "Checking DB dump"
echo "Generating a list of tables and record counts in the DB dump..."
awk '/CREATE TABLE/ { tbl = $3; gsub("`","",tbl); records[tbl] = 0; } \
/INSERT INTO/ { tbl = $3; gsub("`","",tbl); records[tbl] += 1; } \
END { for (tbl in records) print tbl, records[tbl]; }' < $BACKUP_SQL |
sort >$SAVED_TABLES
echo "Generating a list of tables and record counts in the live DB..."
for tbl in $((cd $DATABASE_DIR ; ls *.MYD) | sed '/\.MYD$/s///') ; do
mysql mythconverg -sBe "select '$tbl', count(*) from $tbl"
done | tr -s '\t' ' ' | sort > $LIVE_TABLES
echo "Comparing live versus saved tables..."
if diff $LIVE_TABLES $SAVED_TABLES ; then
echo "Live and saved table lists match."
else
echo
echo "Warning, table lists are not identical!"
STATUS=1
fi
rm $LIVE_TABLES $SAVED_TABLES
echo
if [ $STATUS = 0 ] ; then
echo "Backup passes all checks."
else
echo "The backup is bad or already out of date!"
fi
exit $STATUS
|
Updated to a much more polished and faster version of the script based on the ones found in R5D1.
Last edited by tjc on Sat Aug 26, 2006 6:36 pm; edited 5 times in total |
|
| Back to top |
|
|
mihanson
Joined: 25 Sep 2005
Posts: 972
Location: Los Angeles
|
| Posted: Sat Jul 22, 2006 6:07 pm Post subject: Check_Backup.sh script |
|
|
tjc,
I'm having some trouble with the check_backup.sh script...
| Code: | root@mythbox:/myth/backup# sh check_backup.sh
: command not foundne 2:
: command not foundne 9:
: No such file or directorymyth/backup
: command not foundne 12:
Checking saved files
Checking the integrity of the compressed file...
: No such file or directory.h/backup
: command not foundne 17:
'heck_backup.sh: line 70: syntax error near unexpected token `do
'heck_backup.sh: line 70: `for tbl in $(cat $EXPECTED_TABLES) ; do
|
| Code: | root@mythbox:~# ls -l /myth/backup/savedfiles.tar.bz2
-rw-r--r-- 1 root root 9749055 Jul 22 17:52 /myth/backup/savedfiles.tar.bz2
|
I'm running R5B7 and I created the savedfiles.tar.bz2 with
| Code: |
root@mythbox:/# tar jcvf /myth/backup/savedfiles.tar.bz2 ./etc ./root ./home
|
I'm probably doing something stupid, but I can't figure it out...
_________________
Mike
My Hardware Profile |
|
| Back to top |
|
|
mihanson
Joined: 25 Sep 2005
Posts: 972
Location: Los Angeles
|
| Posted: Sat Jul 22, 2006 6:13 pm Post subject: Nevermind ... |
|
|
Nevermind tjc... I figured it out... I originally pasted that script into WordPad and then copied it to my mythbox. I just copied it again, but this time directly into pico and it seems to work now. Sorry about that! 
_________________
Mike
My Hardware Profile |
|
| Back to top |
|
|
tjc
Joined: 25 Mar 2004
Posts: 9117
Location: Arlington, MA
|
| Posted: Sat Jul 22, 2006 6:20 pm Post subject: |
|
|
| If you cut and paste from the forums you have to watch out for extra spaces at the ends of the lines. This becomes a problem on the continuation lines. I've got a sed one liner posted around here some where that fyou can run to fix this. Ah, here it is: http://mysettopbox.tv/phpBB2/viewtopic.php?t=10001&postorder=asc&start=19 |
|
| Back to top |
|
|
mihanson
Joined: 25 Sep 2005
Posts: 972
Location: Los Angeles
|
| Posted: Sun Jul 23, 2006 10:47 am Post subject: Some warnings |
|
|
Should I be concerned about these?
| Code: | Warning, your record counts don't match for table channel!
Expected = 132, Actual = 111
Warning, your record counts don't match for table credits!
Expected = 81374, Actual = 82647
Warning, your record counts don't match for table inuseprograms!
Expected = 0, Actual = 1
Warning, your record counts don't match for table mythlog!
Expected = 135, Actual = 146
Warning, your record counts don't match for table oldprogram!
Expected = 4954, Actual = 4936
Warning, your record counts don't match for table oldrecorded!
Expected = 307, Actual = 308
Warning, your record counts don't match for table people!
Expected = 31453, Actual = 31318
Warning, your record counts don't match for table pidcache!
Expected = 320, Actual = 319
Warning, your record counts don't match for table program!
Expected = 36327, Actual = 36415
Warning, your record counts don't match for table programgenres!
Expected = 51356, Actual = 51401
Warning, your record counts don't match for table programrating!
Expected = 14700, Actual = 14805
Warning, your record counts don't match for table recorded!
Expected = 48, Actual = 49
Warning, your record counts don't match for table recordedmarkup!
Expected = 369708, Actual = 363624
Warning, your record counts don't match for table recordedprogram!
Expected = 43, Actual = 46
Warning, your record counts don't match for table recordmatch!
Expected = 109, Actual = 119
|
About 75% come back ok...
_________________
Mike
My Hardware Profile |
|
| Back to top |
|
|
tjc
Joined: 25 Mar 2004
Posts: 9117
Location: Arlington, MA
|
| Posted: Sun Jul 23, 2006 7:37 pm Post subject: |
|
|
| I would make a new backup and then test it. It looks like you updated the program guide, made ~3 recordings, and were watching a recording between when you made the backup and when you tested it. |
|
| Back to top |
|
|
opel70
Joined: 11 Apr 2006
Posts: 223
Location: Los Angeles, CA
|
| Posted: Mon Jul 24, 2006 8:06 am Post subject: |
|
|
This may be obvious to most, but is there a way to drop the new backup / restore scripts onto an R5B7 install? Seems like it would be nice to be able to at least use this backup prior to doing my next upgrade.
Thanks,
Tim |
|
| Back to top |
|
|
mihanson
Joined: 25 Sep 2005
Posts: 972
Location: Los Angeles
|
| Posted: Mon Jul 24, 2006 10:03 am Post subject: |
|
|
You can get the script rather easily if you have R5C7 on a CD-R...
| Quote: | - Boot from the CD as usual, but take option 6 (Exit) from the menu. This wil ask for confirmation then drop you to a command prompt.
- Type mount /dev/hda1 and hit enter. You can now access your root partition on /mnt/hda1. (Note from mihanson: this assumes your installation of R5B7 is on hda1. Substitute sda1 or whatever if you have a non-standard install)
|
- Move the old script. mv /mnt/hda1/usr/local/bin/mythbackup /mnt/hda1/usr/local/bin/mythbackup.R5B7
- Type cp -a /usr/local/bin/mythbackup /mnt/hda1/usr/local/bin/ and hit enter.
This should copy the new backup script to your existing install. Just pop the CD and reboot.
This info was adapted from tjc's post here:
http://mysettopbox.tv/phpBB2/viewtopic.php?t=8697&highlight=
_________________
Mike
My Hardware Profile |
|
| Back to top |
|
|
tjc
Joined: 25 Mar 2004
Posts: 9117
Location: Arlington, MA
|
| Posted: Mon Jul 24, 2006 7:08 pm Post subject: |
|
|
There are directions in the "Before you upgrade" section of the "Hints for upgradting to R5C7" on how to modernize an older backup for upgrade purposes. It's essentially a one liner.
I would strongly urge you to take a look at them, as they contain a great deal of useful information for making a smooth transition to R5C7. http://mysettopbox.tv/phpBB2/viewtopic.php?t=10116 |
|
| Back to top |
|
|
mac
Joined: 21 Feb 2006
Posts: 396
Location: Dushanbe, Tajikistan
|
| Posted: Mon Jul 24, 2006 9:32 pm Post subject: |
|
|
| Quote: | | - Multi-level rollover of backup files that keeps up to 10 old backups. You can freely purge these old backups if space is an issue. At the moment I've got a current backup and 6 sets of rollover files which are using a bit over 250Mb. That's roughly the equivalent of 7 minutes of video capacity. |
I like this.
Every now and again someone accidently does a restore when they don't intend to.
Your cat playing with remote or walking on keyboard could do it.. etc. And you would
be left a bunch of orphan shows and other strangeness.
An idea for future releases of your backup script could be the option:
"Revert to the previous state"
where a backup is done before every restore that can be used to revert. |
|
| Back to top |
|
|
tjc
Joined: 25 Mar 2004
Posts: 9117
Location: Arlington, MA
|
| Posted: Tue Jul 25, 2006 9:31 pm Post subject: |
|
|
I agree with you completely about dangers, but I see a couple problems with the suggested approach.
First it could be kind of time consuming (run time not development time). Second, the restore always works from the "current" backup so doing another one before doing a restore would both turn it into a no-op and play poorly with upgrades where this is most commonly used. Changing this would require substantial re-engineering of something that was intended to so simple that it only has 2 or 3 more "moving parts" than a large rock.
On the other hand, one of the things on my to-do list is adding some kind of safety prompt when the restore is run from the menu. Possibly requiring you to enter the root password or the like. Given the potential for destruction it really seems like something that should be a privileged operation. |
|
| Back to top |
|
|
mac
Joined: 21 Feb 2006
Posts: 396
Location: Dushanbe, Tajikistan
|
| Posted: Wed Jul 26, 2006 12:03 am Post subject: |
|
|
I can see your point with noop and not wanting to add the extra logic to something that
was meant to be simple. |
|
| Back to top |
|
|
randomhtpcguy
Joined: 07 Nov 2005
Posts: 153
|
| Posted: Fri Aug 11, 2006 4:56 pm Post subject: |
|
|
I would like to know how to restore "roll-back" to say two backups ago.
More specifically i need to go back until a certain table that dissappeared and another which was corrupted are backed up.
I would like to be able to restore single tables from the backups but a complete roll-back would be ok if I could roll-back everything except the program metadata like the show name and category etc. so i don't have to manually rename orphaned files by guessing.
What should I do?
thanks. |
|
| Back to top |
|
|
tjc
Joined: 25 Mar 2004
Posts: 9117
Location: Arlington, MA
|
| Posted: Fri Aug 11, 2006 6:12 pm Post subject: |
|
|
How are your Linux admin skills? The trick goes somthing like this in outline.
- Uncompress the db backup you're interested in. Two back would be "/myth/backup/mythconverg.sql.bz2.2".
- Extract the create and insert statements for the tables you're interested in into another file. Let's do both of these in one step...
| Code: | bzcat /myth/backup/mythconverg.sql.bz2.2 |
sed -n -e '/CREATE TABLE `recordingprofiles`/,/;/p' \
-e '/INSERT INTO `recordingprofiles`/p' >>extracted.sql |
Obviously you'll need to substitute the names of the tables you're interested in here.
- Feed that file of SQL commands into mysql as input.
| Code: | | mysql mythconverg <extracted.sql |
Let me know which parts you need more details on... Pay extra special attention to the punctuation characters there. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
