Incremental backup of Zimbra on LVM volume with duplicity

by admin — published on January 22nd, 2007

The best way to do a reliable backup of Zimbra Community Edition (ZCS) and minimize downtime is to put /opt/zimbra on a dedicate LVM volume.

The reason why you have to satisfy this requirement is that you cannot simply backup /opt/zimbra while Zimbra is running.

At the same time, creating a backup of a full Zimbra installation can take from several minutes to a few hours, depending on the size of your mailboxes and the resources of your server. This means that you cannot stop Zimbra, backup it and restart it as it would cause a huge downtime.

Welcome LVM.

Making a snapshot of a LVM volume takes only a few seconds. All we have to do is:

  • stop zimbra
  • make a snapshot of the LVM volume mounted under /opt/zimbra
  • start zimbra
  • mount the snapshot on a temporary dir
  • backup the temporary dir
  • unmount the snapshot
  • destroy the snapshot

You can find many examples of this procedure on Zimbra Wiki, but I wasn’t satisfied with any of them because they didn’t support incremental remote backups.

Welcome duplicity.

Duplicity is a nifty piece of software which can do either a full or incremental backup of a directory and store it on a local disk or on a remote storage, via ftp or scp.
duplicity is smart enough to detect when the last full backup was performed more than 1 month ago and will automatically switch from “incremental backup” to “full backup” in that case.
As a bonus it can also purge old backups automatically, without the need for additional scripting.

In my example, I use duplicity to do a remote backup via ftp.
I choose to use symmetrical encryption because I’m lazy. Asymmetrical encryption is supported too, though.

The script shall be run by cron everyday, as root. E.g.:
0 3 * * * /root/bin/backup-zimbra.sh

You need a little more than 2 GB of free disk space inside $TEMPDIR ! duplicity is smart enough to write one backup volume at a time (the size can be set in $VOLSIZE – in my example it’s 2GB), upload it, delete it, and then start writing the next backup volume. At any given time, your TEMPDIR will contain only a single 2GB backup volume.

SECURITY WARNING: in this example script, I put all passwords inside the script itself. Make sure to chmod 700 and chown root:root the script or any user will be able to download and decrypt your backups! Alternatively, you can move the configuration vars to an external file, chmod 600 it and source it inside the script.

#!/bin/bash
LOGDIR="/var/log"
TEMPDIR=$(mktemp -dp /var/tmp zimbra-backup-XXXXX)
FTP_USER='myuser'
FTP_PASSWORD='mypass'
FTP_HOST='ftpserver.tld'
PASSPHRASE='changeme'
DUPLICITY='/usr/bin/duplicity'
VERBOSE=5
RETENTION=3
VOLSIZE=2048
export FTP_PASSWORD
export PASSPHRASE
# what to backup, in my case: /dev/vgdata/mail
VG=vgdata
LV=mail
# snapshot name
LV_SNAP=ZimbraBackup
# lvcreate and lvremove commands path
lvcreate_cmd="/sbin/lvcreate"
lvremove_cmd="/sbin/lvremove"
echo Backup started at `date`
# Stop the Zimbra services
echo "Stopping the Zimbra services..."
/etc/init.d/zimbra stop || exit
echo "Creating a snapshot called $LV_SNAP"
# depending on the time it takes to create the backup and the traffic of your server, you may need to increase the 2G value
$lvcreate_cmd -L2G -s -n $LV_SNAP /dev/$VG/$LV
# Create a mountpoint to mount the logical volume to
echo "Creating a mountpoint for the LV..."
mkdir -p $TEMPDIR/$LV_SNAP
# Mount the logical volume to the mountpoint
echo "Mounting the snapshot..."
# WARNING: if you use xfs you MUST add nouuid option here!
mount -o ro /dev/$VG/$LV_SNAP $TEMPDIR/$LV_SNAP/
# Start the Zimbra services
echo "Restarting the Zimbra services..."
/etc/init.d/zimbra start || echo "ERROR restarting zimbra"
# Create the current backup
FTP_PATH=zimbra
DUPOPTS="-v$VERBOSE --full-if-older-than 1M --tempdir $TEMPDIR --exclude $TEMPDIR/$LV_SNAP/store/delme/ --allow-source-mismatch --volsize $VOLSIZE"
# Clean up incomplete backup archive files
$DUPLICITY cleanup --force -v$VERBOSE \
ftp://$FTP_USER@$FTP_HOST/$FTP_PATH/$i >>/$LOGDIR/$FTP_PATH.log
# remove old backup sets: keep only the last N full backups, where N is the value of $RETENTION
$DUPLICITY remove-all-but-n-full $RETENTION --force -v$VERBOSE \
 ftp://$FTP_USER@$FTP_HOST/$FTP_PATH >>/$LOGDIR/$FTP_PATH
# create new backup
$DUPLICITY $DUPOPTS \
 $TEMPDIR/$LV_SNAP/ \
 ftp://$FTP_USER@$FTP_HOST/$FTP_PATH >>$LOGDIR/$FTP_PATH.log
# Unmount /tmp/$lv_zimbra and remove the logical volume
echo "Unmounting and removing the snapshot."
umount /dev/$VG/$LV_SNAP
$lvremove_cmd --force /dev/$VG/$LV_SNAP
echo $lvremove_cmd /dev/$VG/$LV_SNAP
# Done!
echo Zimbra backed up to $FTP_PATH on $FTP_HOST
echo Backup ended at `date`

Note: when you cut & paste the script, make sure the resulting file is in UNIX format.

Last update: 9 Aug 2010

Share
 
  1. andynz says:
    August 6th, 2010 at 5:24

    Hi. Having some trouble getting this to work and hoping you can help. The first error I get is this:
    Restarting the Zimbra services…
    No old backup sets found, nothing deleted.
    Command line error: Expected 2 args, got 3
    Enter ‘duplicity –help’ for help screen.
    Traceback (most recent call last):
    File “/usr/bin/duplicity”, line 463, in
    And more after that…

    I run Ubuntu 8.04 with Zimbra 6.0.5 I did notice that after using apt-get update then install of duplicity I get a version 0.4.10 of duplcity.

    Andy ideas on what to try would be great thanks.
    Cheers, Andy.

  2. admin says:
    August 6th, 2010 at 9:11

    Can you post the output of: sh -x ./nameofscript ?

  3. andynz says:
    August 7th, 2010 at 1:24

    Hi Thanks for the reply. here it is:
    andy@ZIMBRA1:/backup$ ./trial.sh ?
    mktemp: cannot make temp dir /var/tmp/zimbra-backup-XXXXX/tmp.DExYG29548: No such file or directory
    Backup started at Sat Aug 7 11:19:07 NZST 2010
    Creating a snapshot called ZimbraBackup
    /dev/mapper/control: open failed: Permission denied
    Failure to communicate with kernel device-mapper driver.
    /dev/mapper/control: open failed: Permission denied
    Failure to communicate with kernel device-mapper driver.
    Incompatible libdevmapper 1.02.20 (2007-06-15)(compat) and kernel driver
    snapshot: Required device-mapper target(s) not detected in your kernel
    lvcreate: Create a logical volume

    lvcreate
    [-A|--autobackup {y|n}]
    [--addtag Tag]
    [--alloc AllocationPolicy]
    [-C|--contiguous {y|n}]
    [-d|--debug]
    [-h|-?|--help]
    [-i|--stripes Stripes [-I|--stripesize StripeSize]]
    {-l|–extents LogicalExtentsNumber |
    -L|–size LogicalVolumeSize[kKmMgGtTpPeE]}
    [-M|--persistent {y|n}] [--major major] [--minor minor]
    [-m|--mirrors Mirrors [--nosync] [--corelog]]
    [-n|--name LogicalVolumeName]
    [-p|--permission {r|rw}]
    [-r|--readahead ReadAheadSectors]
    [-R|--regionsize MirrorLogRegionSize]
    [-t|--test]
    [--type VolumeType]
    [-v|--verbose]
    [-Z|--zero {y|n}]
    [--version]
    VolumeGroupName [PhysicalVolumePath...]

    lvcreate -s|–snapshot
    [-c|--chunksize]
    [-A|--autobackup {y|n}]
    [--addtag Tag]
    [--alloc AllocationPolicy]
    [-C|--contiguous {y|n}]
    [-d|--debug]
    [-h|-?|--help]
    [-i|--stripes Stripes [-I|--stripesize StripeSize]]
    {-l|–extents LogicalExtentsNumber[%{VG|LV|FREE}] |
    -L|–size LogicalVolumeSize[kKmMgGtTpPeE]}
    [-M|--persistent {y|n}] [--major major] [--minor minor]
    [-n|--name LogicalVolumeName]
    [-p|--permission {r|rw}]
    [-r|--readahead ReadAheadSectors]
    [-t|--test]
    [-v|--verbose]
    [--version]
    OriginalLogicalVolume[Path] [PhysicalVolumePath...]

    Creating a mountpoint for the LV…
    Mounting the snapshot…
    mount: only root can do that
    ./trial.sh: line 37: //var/log/duplicity/zimbra: Permission denied
    ./trial.sh: line 39: /var/log/duplicity/zimbra.log: Permission denied
    Unmounting and removing the snapshot.
    umount: /dev/vg1/ZimbraBackup is not mounted (according to mtab)
    /var/lock/lvm/V_vg1: open failed: Permission denied
    Can’t lock vg1: skipping
    /sbin/lvremove /dev/vg1/ZimbraBackup
    Zimbra backed up to zimbra on 192.168.1.5
    Backup ended at Sat Aug 7 11:19:07 NZST 2010

    So obviously permission issues here and the mktemp is failing at the start of the script. Script runs as ‘root’ in /backup directory with ‘root’ permission.

    Cheers Andy.

  4. admin says:
    August 7th, 2010 at 11:04

    It looks like you are running it as “andy” not root, also are you sure that zimbra is on a LVM volume? It looks like it is not.

  5. andynz says:
    August 8th, 2010 at 0:55

    Looking at the LVM issue I get this with df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/md3 139G 33G 100G 25% /
    varrun 502M 2.4M 500M 1% /var/run
    varlock 502M 0 502M 0% /var/lock
    udev 502M 88K 502M 1% /dev
    devshm 502M 0 502M 0% /dev/shm
    /dev/mapper/vg1-backup
    50G 1.8G 46G 4% /backup
    /dev/md2 92M 25M 62M 29% /boot
    /dev/mapper/vg1-zimbra
    149G 2.9G 139G 3% /opt/zimbra

    So looks like the LVM “zimbra” is mounted under /opt/zimbra

    Sorry yes should have run it as root. Here is what I get running as root:

    root@ZIMBRA1:/backup# sh -x ./trial.sh ?
    + LOGDIR=/var/log/duplicity
    + mktemp -dp /var/tmp/zimbra-backup-XXXXX
    mktemp: cannot make temp dir /var/tmp/zimbra-backup-XXXXX/tmp.gALRoW7325: No such file or directory
    + TEMPDIR=
    + FTP_USER=***
    + FTP_PASSWORD=***
    + FTP_HOST=192.168.1.5
    + PASSPHRASE=********
    + DUPLICITY=/usr/bin/duplicity
    + VERBOSE=5
    + RETENTION=3
    + export FTP_PASSWORD
    + export PASSPHRASE
    + date +%d
    + DATE=08
    + VG=vg1
    + LV=zimbra
    + LV_SNAP=ZimbraBackup
    + which lvcreate
    + lvcreate_cmd=/sbin/lvcreate
    + which lvremove
    + lvremove_cmd=/sbin/lvremove
    + date
    + echo Backup started at Sun Aug 8 10:26:02 NZST 2010
    Backup started at Sun Aug 8 10:26:02 NZST 2010
    + echo Creating a snapshot called ZimbraBackup
    Creating a snapshot called ZimbraBackup
    + /sbin/lvcreate -L2G -s -n ZimbraBackup /dev/vg1/zimbra
    Logical volume “ZimbraBackup” created
    + echo Creating a mountpoint for the LV…
    Creating a mountpoint for the LV…
    + mkdir -p /ZimbraBackup
    + echo Mounting the snapshot…
    Mounting the snapshot…
    + mount -o ro /dev/vg1/ZimbraBackup /ZimbraBackup/
    + FTP_PATH=zimbra
    + DUPOPTS=-v5 –full-if-older-than 1M –tempdir –exclude /ZimbraBackup/store/delme/
    + /usr/bin/duplicity remove-all-but-n-full 3 –force -v5 ftp://www1@192.168.1.5/zimbra
    No old backup sets found, nothing deleted.
    + /usr/bin/duplicity -v5 –full-if-older-than 1M –tempdir –exclude /ZimbraBackup/store/delme/ /ZimbraBackup/ ftp://www1@192.168.1.5/zimbra
    Command line error: Expected 2 args, got 3
    Enter ‘duplicity –help’ for help screen.
    Traceback (most recent call last):
    File “/usr/bin/duplicity”, line 463, in
    with_tempdir(main)
    File “/usr/bin/duplicity”, line 460, in with_tempdir
    tempdir.default().cleanup()
    File “/usr/lib/python2.5/site-packages/duplicity/tempdir.py”, line 54, in default
    _defaultInstance = TemporaryDirectory(temproot = globals.temproot)
    File “/usr/lib/python2.5/site-packages/duplicity/tempdir.py”, line 113, in __init__
    self.__dir = tempfile.mkdtemp(“-tempdir”, “duplicity-”, temproot)
    File “/usr/lib/python2.5/tempfile.py”, line 328, in mkdtemp
    _os.mkdir(file, 0700)
    OSError: [Errno 2] No such file or directory: ‘–exclude/duplicity-VtPtMH-tempdir’
    Exception exceptions.AttributeError: “TemporaryDirectory instance has no attribute ‘_TemporaryDirectory__lock’” in <bound method TemporaryDirectory.__del__ of > ignored
    + echo Unmounting and removing the snapshot.
    Unmounting and removing the snapshot.
    + umount /dev/vg1/ZimbraBackup
    + /sbin/lvremove –force /dev/vg1/ZimbraBackup
    Logical volume “ZimbraBackup” successfully removed
    + echo /sbin/lvremove /dev/vg1/ZimbraBackup
    /sbin/lvremove /dev/vg1/ZimbraBackup
    + echo Zimbra backed up to zimbra on 192.168.1.5
    Zimbra backed up to zimbra on 192.168.1.5
    + date
    + echo Backup ended at Sun Aug 8 10:26:05 NZST 2010
    Backup ended at Sun Aug 8 10:26:05 NZST 2010

    Cheers Andy.

  6. admin says:
    August 8th, 2010 at 1:47

    Thanks for your report. Indeed there is an error in the script!
    It should be:
    mktemp -dp /var/tmp zimbra-backup-XXXXX
    instead of:
    mktemp -dp /var/tmp/zimbra-backup-XXXXX
    I just corrected the script in my post.

    Thanks again!

  7. andynz says:
    August 8th, 2010 at 6:23

    Good to know it wasn’t just me! :)

    I still get another error about source directory mismatch after restarting zimbra services:

    Restarting the Zimbra services…
    No old backup sets found, nothing deleted.
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    Fatal Error: Backup source directory has changed.
    Current directory: /var/tmp/zimbra-backup-15998/ZimbraBackup
    Previous directory: /var/tmp/zimbra-backup-g2599/ZimbraBackup

    Aborting because you may have accidentally tried to backup two
    different data sets to the same remote location, or using the same
    archive directory. If this is not a mistake, use the
    –allow-source-mismatch switch to avoid seeing this message
    Unmounting and removing the snapshot.
    Logical volume “ZimbraBackup” successfully removed
    /sbin/lvremove /dev/vg1/ZimbraBackup
    Zimbra backed up to zimbra on 192.168.1.5
    Backup ended at Sun Aug 8 16:13:47 NZST 2010

    I tried one backup which had an error due to me not entering all my settings in your new script. So I deleted all the files off the ftp directory and the /var/tmp directory. Is this a duplicity config or log I need to reset?
    Cheers Andy.

  8. admin says:
    August 8th, 2010 at 12:10

    You found another glitch :) I updated the script on my server, but forgot to update my post accordingly. You should add the following options:
    --allow-source-mismatch --volsize 2048
    to the var DUPOPTS. I already updated the post on my blog accordingly.
    We need the --allow-source-mismatch option because we mount the Zimbra volume under a temporary directory created with mktemp. The name of this directory changes everytime we run the script and duplicity doesn’t like it, hence we have to force it to accept the new path.
    I also added the --volsize 2048 option because with some FTP daemons + filesystems combinations the max file size is 2GB. It’s safe to let duplicity split the backup archives in 2GB slices.

    Thanks again for your feedback! Keep it coming in :)

  9. andynz says:
    August 9th, 2010 at 3:49

    Great, that fixed it and it is working fine now. Thanks for your help on this. Might be worth mentioning in your intro about using dos2unix when you copy and past the script. That caught me out early on.

    Thanks again
    Cheers Andy

  10. admin says:
    August 9th, 2010 at 10:21

    glad it worked! Cheers :)

  11. hiatsu says:
    November 9th, 2010 at 18:02

    Hi,

    can you please help me with this: I’m not sure do I understand the part “make a snapshot of the LVM volume mounted under /opt/zimbra”. Do I have to mount /opt/zimbra somehow, before I can use the script?

  12. admin says:
    November 9th, 2010 at 18:16

    This backup script assumes that you have a LVM volume mounted under /opt/zimbra and that zimbra is installed on it.
    The script will create a snapshot of that volume, mount it under a temporary path, do a backup, destroy the snapshot.
    If you you don’t already use LVM then this script is not for you.

  13. hiatsu says:
    November 9th, 2010 at 20:59

    OK, thanks for the information. It seems that full backup is my only solution since I can’t use LVM + Duplicity?

  14. admin says:
    November 9th, 2010 at 21:22

    Let me try to explain you what is the main difference between the “classic” backup approach, and the LVM snapshot approach.
    If your system does NOT uses LVM, you have to stop zimbra, do a backup of /opt/zimbra (with whatever tool you like), restart zimbra.
    If your system uses LVM, you can stop zimbra, create a LVM snapshot, restart zimbra, do a backup of the LVM snapshot (with whatever tool you like), destroy the LVM snapshot.
    The obvious advantage is that with the LVM snapshot approach you must stop zimbra only for the time needed to create the LVM snapshot.
    Without LVM, you have to stop zimbra for the time needed to perform the backup.
    The time needed to create the LVM snapshot is always a few seconds. The time needed to perform the backup is variable: from a few minutes to a few hours, depending on how big your mail spool is.
    Hope it makes sense.

  15. hiatsu says:
    November 9th, 2010 at 21:49

    This makes perfect sense, thank you. It seems that only usable way to back up Zimbra is this. I should have realized that before installation, there is no way to enable LVM afterwards?

  16. admin says:
    November 9th, 2010 at 22:51

    well it depends on your skills. If you have some experience with LVM and you can schedule a few hours of downtime, you can do it even now.

  17. andynz says:
    January 13th, 2011 at 3:21

    Hi Again :)

    I have noticed that I get some “command not found” errors when running as a cron job but it works fine when run manually through console. Been searching and it looks like it might be a path issue but not quite sure how to fix it. Any suggestions would be great. Here is the cron job output:

    Backup started at Tue Dec 21 02:00:01 NZDT 2010
    Stopping the Zimbra services…
    Creating a snapshot called ZimbraBackup
    /backup/./zmbklvm.sh: line 28: -L2G: command not found
    Creating a mountpoint for the LV…
    Mounting the snapshot…
    mount: special device /dev/vg1/ZimbraBackup does not exist
    Restarting the Zimbra services…
    No old backup sets found, nothing deleted.
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    Unmounting and removing the snapshot.
    umount: /dev/vg1/ZimbraBackup: not found
    /backup/./zmbklvm.sh: line 52: –force: command not found
    /dev/vg1/ZimbraBackup
    Zimbra backed up to zimbra on 192.168.1.3
    Backup ended at Tue Dec 21 02:11:11 NZDT 2010

  18. admin says:
    January 13th, 2011 at 13:44

    These errors show that $lvcreate_cmd and $lvremove_cmd were not properly set. Please replace these lines:


    lvcreate_cmd=`which lvcreate`
    lvremove_cmd=`which lvremove`

    with something like:


    lvcreate_cmd="/sbin/lvcreate"
    lvremove_cmd="/sbin/lvremove"

    or wherever you have lvcreate and lvremove installed.

  19. andynz says:
    January 14th, 2011 at 1:41

    Hmm…should have found that shouldn’t I! Thanks for the help.

    Cheers
    Andy

  20. luma says:
    March 27th, 2011 at 7:04

    Hello,

    Excellent script.

    How can I use this script with a server that does not use default port of 22? My backup servers use other ports such as 9291 for example.

    Will this work with Cron?

    Thank you.

    Luma

  21. admin says:
    March 27th, 2011 at 11:37

    My script by default uses FTP. If you are using scp as the transfer method and the destination host is listening on port 9291, you should add the following to each $DUPLICITY command:
    –ssh-options=”-oPort=9291″.

    I see no reason why this script would not run under cron.

  22. andynz says:
    August 7th, 2011 at 4:20

    Hi. I have had a couple of occasions recently ( 1 a month ago and 1 last night) where during the backup process Zimbra is stopped and never restarts. The script has been running flawlessly for months prior to that. I am trying to find out why. The duplicity files are not written to the ftp backup so something is stopping the process but it seems to be random and rare.

    I ma just wondering if it is to do with” depending on the time it takes to create the backup and the traffic of your server, you may need to increase the 2G value”. Can you let me know what problems the 2G value could have?

    Thanks
    Andy

  23. admin says:
    August 7th, 2011 at 10:46

    Strange. The 2G value means that after generating the LVM snapshot, zimbra can write up to 2GB of data to the “live” LVM volume. In other words: during the time that it takes to generate the backup, zimbra can receive up to ~2GB of mail (including logs, metadata, …)

Leave a comment

You must be logged in to post a comment.