Rotated backup

Years ago I 'invented' a wheel, I mean rotated backup scheme. My idea was to store three types of copies:

• Monthly, stored for ever


• Weekly, overwrote every month (a copy for every 1st, 2nd, 3rd, 4th and sometimes 5th Friday of the month


• Daily, overwrote every week (a copy for every weekday)

This schema provides 5 (7 including weekends) last copies, and 5 weekend copies (last month) and a copy from the beginning of every month. The advantage is that I can preserve some space and still have quite good resource of files from the past. The disadvantage is that I can't restore file file created 10 days ago and deleted 9 days ago (if it wasn't Friday :-)).

Usage:
/usr/local/bin/rotated_backup.sh server directory
invoked from backup server assuming that directory fo copies is called /var/BACKUP. How its moved on a tape or DVD is another story.

If you prefer to follow symbolic links, add 'h' to tar command.

#!/bin/bash

. /usr/local/bin/password
LC_ALL=C
DATE=`date +%F`
DAYOFMONTH=`date +%d`
DAYOFMONTH=${DAYOFMONTH#0}
DAYOFWEEK=`date +%a`
let WEEKOFMONTH=$DAYOFMONTH/7+1
MONTH=`date +%m`

LOCALSTORAGE=/var/BACKUP
SERVER=$1
DIRTOBACKUP=$2
DIRNAME=${DIRTOBACKUP##*/}
PARENTDIR=${DIRTOBACKUP%/*}

# Don't write undescore if its root subdirectory
# (PARENTDIR is /, means empty)
if [[ "$PARENTDIR" != "" ]]
then
  PARENTDIR="${PARENTDIR##*/}_"
else
  PARENTDIR="root_"
fi

if [[ "$DIRNAME" != "" ]]
then
  DIRNAME="${DIRNAME##*/}_"
fi

DAILYARCHNAME=${LOCALSTORAGE}/${SERVER}_${PARENTDIR}${DIRNAME}${DAYOFWEEK}.tgz
WEEKLYARCHNAME=${LOCALSTORAGE}/${SERVER}_${PARENTDIR}${DIRNAME}WEEK_${WEEKOFMONTH}.tgz
MONTHLYARCHNAME=${LOCALSTORAGE}/${SERVER}_${PARENTDIR}${DIRNAME}MONTH_${MONTH}_${DATE}.tgz

# Check argument list. Two are required.
: ${2?"Usage: $0 servername dir_to_backup"}

# Check the order of arguments (no slashes in servers name,
# at least one slach in directory name)
if [[ "$SERVER" == *"/"* ]]
then
  echo "The first argument is a server and can't contain slashes."
  exit 1
fi

if [[ "$DIRTOBACKUP" != "/"* ]]
then
  echo "The second argument is a directory and must contain at least one slashe, the leading one."
  exit 1
fi

# ==== FUNCTIONS ====
function getbackup {
  # Backup
  if [[ -e $1 ]]
  then
    mv $1 ${1}_old
  fi
# PASS is in /usr/local/bin/password
# sourced at the beginning of the script
  ssh $SERVER tar zc $DIRTOBACKUP | openssl des3 -salt -k $PASS > $1

  # Error checking
  if [[ -e $1 ]]
  then
    if [[ -s $1 ]]
    then
      echo "Back up is OK (${1})"
      rm -f ${1}_old
    else
      echo "Back up failed. File is epmty (${1})"
      mv ${1}_old $1
      exit 2
    fi
  else
    echo "Backup failed. File not created (${1})"
    mv ${1}_old $1
    exit 3
  fi
}

# ==== EXECUTION ====
if [ "$DAYOFMONTH" == "01" ]; then
  echo "Monthly backup"
  getbackup $MONTHLYARCHNAME
elif [ "$DAYOFWEEK" == "Fri" ]; then
  echo "Weekly backup"
  getbackup $WEEKLYARCHNAME
else
  echo "Daily backup"
  getbackup $DAILYARCHNAME
fi

exit 0

Incremental backup

If you have really huge amount of data to copy, its good to short daily backup and copy only changed files. There are some limit of the technique, as it relays on timestamp, but quite effective.

ssh $SERVER tar zc --listed-incremental=/var/log/daily.snar $DIRTOBACKUP | openssl des3 -salt -k $PASS > $1

In case of using this schema, the backup function for daily and other types will be different.

• Daily backup: take /var/log/daily.snar and start copying


• weekly backup: delete /var/log/daily.snar and perform with new on (neccessary for Monday backup)


• monthly backup: can simply omit this parameter. The next daily backup perform as usual - incremental to the last one.

---

If you have another idea about good backup schemes, or you just felt inspired to write something, leave a comment, please.

Virtual tapes for every daily, weekly and monthly copy

Timeline of rotated backup

Have you any impression about it? Do something! :-)
Leave a comment please or write me an e-mail, please.

Nightly commits /etc into SVN

Simple script committing changes in /etc/postfix and /etc/network into SVN repository. Can be scheduled every night - it checks for changes and in doesn't try to commit if system is untouched.

/etc -> SVN


#!/bin/bash

export LC_ALL=C
export EDITOR=/usr/bin/vim
tmpconf=/var/tmp/conf2svn
root=/etc

for srv in serverA serverB serverC
do
  for dir in network postfix
  do
    rm -rf ${tmpconf}/${srv} > /dev/null 2>&1
    mkdir -p ${tmpconf}/${srv} > /dev/null 2>&1
    cd ${tmpconf}/${srv}
    svn checkout --quiet svn://svn.mydomain.com/repos/conf/${srv}/${dir} ./${dir}
    scp -q -r ${srv}:${root}/${dir} .
    cd ${tmpconf}/${srv}/${dir}
    svn diff | grep "" > /dev/null
    if [[ $? -eq 0 ]]
    then
      svn commit -m "Commiting last changes in live servers' configuration (/etc)."
    fi
  done
done

rm -rf $tmpconf > /dev/null 2>&1


If you import /etc/something into SVN, delete /etc/something (be careful) and check something out back into /etc you won't need to make temporary working copy (SVN workspace) into /var/tmp/conf2svn

Script can't add new files. It only commits changes in existing ones.

MySQL schemas -> SVN


#!/bin/bash

export LC_ALL=C
export EDITOR=/usr/bin/vim
tmpconf=/var/tmp/schema2svn

rm -rf $tmpconf > /dev/null 2>&1
mkdir $tmpconf > /dev/null 2>&1

cd $tmpconf
svn checkout --quiet svn://svn.mydomain.com/repos/schemas .

for srv in dbA dbB
do
  mysqldump --host=$srv --password=nkjn39c134 --no-data --all-databases > ${tmpconf}/${srv}_schema_dump.txt
done

for file in *txt
do
  grep -v -E -- '(^--|ENGINE=MyISAM AUTO_INCREMENT)' $file > ${file}.new
  mv ${file}.new $file
done

svn diff | grep "" > /dev/null
if [[ $? -eq 0 ]]
then
  svn commit --quiet -m "Committing schema changes on live servers."
fi

rm -rf $tmpconf > /dev/null 2>&1


___
If you implemented your own schemas of backup/archiving, please let me know - send me an e-mail, drop a comment or meet me on IRC.