Incremental backup of Zimbra on LVM volume with duplicity

The best way to do a reliable backup of Zimbra Community Edition (ZCS) and minimize downtime is to put /opt/zimbra on a dedicate LVM volume.

The reason why you have to satisfy this requirement is that you cannot simply backup /opt/zimbra while Zimbra is running.

At the same time, creating a backup of a full Zimbra installation can take from several minutes to a few hours, depending on the size of your mailboxes and the resources of your server. This means that you cannot stop Zimbra, backup it and restart it as it would cause a huge downtime.

Welcome LVM.

Making a snapshot of a LVM volume takes only a few seconds. All we have to do is:

  • stop zimbra
  • make a snapshot of the LVM volume mounted under /opt/zimbra
  • start zimbra
  • mount the snapshot on a temporary dir
  • backup the temporary dir
  • unmount the snapshot
  • destroy the snapshot

You can find many examples of this procedure on Zimbra Wiki, but I wasn’t satisfied with any of them because they didn’t support incremental remote backups.

Welcome duplicity.

Duplicity is a nifty piece of software which can do either a full or incremental backup of a directory and store it on a local disk or on a remote storage, via ftp or scp.
duplicity is smart enough to detect when the last full backup was performed more than 1 month ago and will automatically switch from “incremental backup” to “full backup” in that case.
As a bonus it can also purge old backups automatically, without the need for additional scripting.

In my example, I use duplicity to do a remote backup via ftp.
I choose to use symmetrical encryption because I’m lazy. Asymmetrical encryption is supported too, though.

The script shall be run by cron everyday, as root. E.g.:
0 3 * * * /root/bin/backup-zimbra.sh

You need a little more than 2 GB of free disk space inside $TEMPDIR ! duplicity is smart enough to write one backup volume at a time (the size can be set in $VOLSIZE – in my example it’s 2GB), upload it, delete it, and then start writing the next backup volume. At any given time, your TEMPDIR will contain only a single 2GB backup volume.

SECURITY WARNING: in this example script, I put all passwords inside the script itself. Make sure to chmod 700 and chown root:root the script or any user will be able to download and decrypt your backups! Alternatively, you can move the configuration vars to an external file, chmod 600 it and source it inside the script.

Note: when you cut & paste the script, make sure the resulting file is in UNIX format.

Last update: 9 Aug 2010

Share

28 thoughts on “Incremental backup of Zimbra on LVM volume with duplicity

  1. Hi. Having some trouble getting this to work and hoping you can help. The first error I get is this:
    Restarting the Zimbra services…
    No old backup sets found, nothing deleted.
    Command line error: Expected 2 args, got 3
    Enter ‘duplicity –help’ for help screen.
    Traceback (most recent call last):
    File “/usr/bin/duplicity”, line 463, in
    And more after that…

    I run Ubuntu 8.04 with Zimbra 6.0.5 I did notice that after using apt-get update then install of duplicity I get a version 0.4.10 of duplcity.

    Andy ideas on what to try would be great thanks.
    Cheers, Andy.

  2. Hi Thanks for the reply. here it is:
    andy@ZIMBRA1:/backup$ ./trial.sh ?
    mktemp: cannot make temp dir /var/tmp/zimbra-backup-XXXXX/tmp.DExYG29548: No such file or directory
    Backup started at Sat Aug 7 11:19:07 NZST 2010
    Creating a snapshot called ZimbraBackup
    /dev/mapper/control: open failed: Permission denied
    Failure to communicate with kernel device-mapper driver.
    /dev/mapper/control: open failed: Permission denied
    Failure to communicate with kernel device-mapper driver.
    Incompatible libdevmapper 1.02.20 (2007-06-15)(compat) and kernel driver
    snapshot: Required device-mapper target(s) not detected in your kernel
    lvcreate: Create a logical volume

    lvcreate
    [-A|–autobackup {y|n}]
    [–addtag Tag]
    [–alloc AllocationPolicy]
    [-C|–contiguous {y|n}]
    [-d|–debug]
    [-h|-?|–help]
    [-i|–stripes Stripes [-I|–stripesize StripeSize]]
    {-l|–extents LogicalExtentsNumber |
    -L|–size LogicalVolumeSize[kKmMgGtTpPeE]}
    [-M|–persistent {y|n}] [–major major] [–minor minor]
    [-m|–mirrors Mirrors [–nosync] [–corelog]]
    [-n|–name LogicalVolumeName]
    [-p|–permission {r|rw}]
    [-r|–readahead ReadAheadSectors]
    [-R|–regionsize MirrorLogRegionSize]
    [-t|–test]
    [–type VolumeType]
    [-v|–verbose]
    [-Z|–zero {y|n}]
    [–version]
    VolumeGroupName [PhysicalVolumePath…]

    lvcreate -s|–snapshot
    [-c|–chunksize]
    [-A|–autobackup {y|n}]
    [–addtag Tag]
    [–alloc AllocationPolicy]
    [-C|–contiguous {y|n}]
    [-d|–debug]
    [-h|-?|–help]
    [-i|–stripes Stripes [-I|–stripesize StripeSize]]
    {-l|–extents LogicalExtentsNumber[%{VG|LV|FREE}] |
    -L|–size LogicalVolumeSize[kKmMgGtTpPeE]}
    [-M|–persistent {y|n}] [–major major] [–minor minor]
    [-n|–name LogicalVolumeName]
    [-p|–permission {r|rw}]
    [-r|–readahead ReadAheadSectors]
    [-t|–test]
    [-v|–verbose]
    [–version]
    OriginalLogicalVolume[Path] [PhysicalVolumePath…]

    Creating a mountpoint for the LV…
    Mounting the snapshot…
    mount: only root can do that
    ./trial.sh: line 37: //var/log/duplicity/zimbra: Permission denied
    ./trial.sh: line 39: /var/log/duplicity/zimbra.log: Permission denied
    Unmounting and removing the snapshot.
    umount: /dev/vg1/ZimbraBackup is not mounted (according to mtab)
    /var/lock/lvm/V_vg1: open failed: Permission denied
    Can’t lock vg1: skipping
    /sbin/lvremove /dev/vg1/ZimbraBackup
    Zimbra backed up to zimbra on 192.168.1.5
    Backup ended at Sat Aug 7 11:19:07 NZST 2010

    So obviously permission issues here and the mktemp is failing at the start of the script. Script runs as ‘root’ in /backup directory with ‘root’ permission.

    Cheers Andy.

  3. Looking at the LVM issue I get this with df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/md3 139G 33G 100G 25% /
    varrun 502M 2.4M 500M 1% /var/run
    varlock 502M 0 502M 0% /var/lock
    udev 502M 88K 502M 1% /dev
    devshm 502M 0 502M 0% /dev/shm
    /dev/mapper/vg1-backup
    50G 1.8G 46G 4% /backup
    /dev/md2 92M 25M 62M 29% /boot
    /dev/mapper/vg1-zimbra
    149G 2.9G 139G 3% /opt/zimbra

    So looks like the LVM “zimbra” is mounted under /opt/zimbra

    Sorry yes should have run it as root. Here is what I get running as root:

    root@ZIMBRA1:/backup# sh -x ./trial.sh ?
    + LOGDIR=/var/log/duplicity
    + mktemp -dp /var/tmp/zimbra-backup-XXXXX
    mktemp: cannot make temp dir /var/tmp/zimbra-backup-XXXXX/tmp.gALRoW7325: No such file or directory
    + TEMPDIR=
    + FTP_USER=***
    + FTP_PASSWORD=***
    + FTP_HOST=192.168.1.5
    + PASSPHRASE=********
    + DUPLICITY=/usr/bin/duplicity
    + VERBOSE=5
    + RETENTION=3
    + export FTP_PASSWORD
    + export PASSPHRASE
    + date +%d
    + DATE=08
    + VG=vg1
    + LV=zimbra
    + LV_SNAP=ZimbraBackup
    + which lvcreate
    + lvcreate_cmd=/sbin/lvcreate
    + which lvremove
    + lvremove_cmd=/sbin/lvremove
    + date
    + echo Backup started at Sun Aug 8 10:26:02 NZST 2010
    Backup started at Sun Aug 8 10:26:02 NZST 2010
    + echo Creating a snapshot called ZimbraBackup
    Creating a snapshot called ZimbraBackup
    + /sbin/lvcreate -L2G -s -n ZimbraBackup /dev/vg1/zimbra
    Logical volume “ZimbraBackup” created
    + echo Creating a mountpoint for the LV…
    Creating a mountpoint for the LV…
    + mkdir -p /ZimbraBackup
    + echo Mounting the snapshot…
    Mounting the snapshot…
    + mount -o ro /dev/vg1/ZimbraBackup /ZimbraBackup/
    + FTP_PATH=zimbra
    + DUPOPTS=-v5 –full-if-older-than 1M –tempdir –exclude /ZimbraBackup/store/delme/
    + /usr/bin/duplicity remove-all-but-n-full 3 –force -v5 ftp://www1@192.168.1.5/zimbra
    No old backup sets found, nothing deleted.
    + /usr/bin/duplicity -v5 –full-if-older-than 1M –tempdir –exclude /ZimbraBackup/store/delme/ /ZimbraBackup/ ftp://www1@192.168.1.5/zimbra
    Command line error: Expected 2 args, got 3
    Enter ‘duplicity –help’ for help screen.
    Traceback (most recent call last):
    File “/usr/bin/duplicity”, line 463, in
    with_tempdir(main)
    File “/usr/bin/duplicity”, line 460, in with_tempdir
    tempdir.default().cleanup()
    File “/usr/lib/python2.5/site-packages/duplicity/tempdir.py”, line 54, in default
    _defaultInstance = TemporaryDirectory(temproot = globals.temproot)
    File “/usr/lib/python2.5/site-packages/duplicity/tempdir.py”, line 113, in __init__
    self.__dir = tempfile.mkdtemp(“-tempdir”, “duplicity-“, temproot)
    File “/usr/lib/python2.5/tempfile.py”, line 328, in mkdtemp
    _os.mkdir(file, 0700)
    OSError: [Errno 2] No such file or directory: ‘–exclude/duplicity-VtPtMH-tempdir’
    Exception exceptions.AttributeError: “TemporaryDirectory instance has no attribute ‘_TemporaryDirectory__lock'” in <bound method TemporaryDirectory.__del__ of > ignored
    + echo Unmounting and removing the snapshot.
    Unmounting and removing the snapshot.
    + umount /dev/vg1/ZimbraBackup
    + /sbin/lvremove –force /dev/vg1/ZimbraBackup
    Logical volume “ZimbraBackup” successfully removed
    + echo /sbin/lvremove /dev/vg1/ZimbraBackup
    /sbin/lvremove /dev/vg1/ZimbraBackup
    + echo Zimbra backed up to zimbra on 192.168.1.5
    Zimbra backed up to zimbra on 192.168.1.5
    + date
    + echo Backup ended at Sun Aug 8 10:26:05 NZST 2010
    Backup ended at Sun Aug 8 10:26:05 NZST 2010

    Cheers Andy.

  4. Thanks for your report. Indeed there is an error in the script!
    It should be:
    mktemp -dp /var/tmp zimbra-backup-XXXXX
    instead of:
    mktemp -dp /var/tmp/zimbra-backup-XXXXX
    I just corrected the script in my post.

    Thanks again!

  5. Good to know it wasn’t just me! :)

    I still get another error about source directory mismatch after restarting zimbra services:

    Restarting the Zimbra services…
    No old backup sets found, nothing deleted.
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    Fatal Error: Backup source directory has changed.
    Current directory: /var/tmp/zimbra-backup-15998/ZimbraBackup
    Previous directory: /var/tmp/zimbra-backup-g2599/ZimbraBackup

    Aborting because you may have accidentally tried to backup two
    different data sets to the same remote location, or using the same
    archive directory. If this is not a mistake, use the
    –allow-source-mismatch switch to avoid seeing this message
    Unmounting and removing the snapshot.
    Logical volume “ZimbraBackup” successfully removed
    /sbin/lvremove /dev/vg1/ZimbraBackup
    Zimbra backed up to zimbra on 192.168.1.5
    Backup ended at Sun Aug 8 16:13:47 NZST 2010

    I tried one backup which had an error due to me not entering all my settings in your new script. So I deleted all the files off the ftp directory and the /var/tmp directory. Is this a duplicity config or log I need to reset?
    Cheers Andy.

  6. You found another glitch :) I updated the script on my server, but forgot to update my post accordingly. You should add the following options:
    --allow-source-mismatch --volsize 2048
    to the var DUPOPTS. I already updated the post on my blog accordingly.
    We need the --allow-source-mismatch option because we mount the Zimbra volume under a temporary directory created with mktemp. The name of this directory changes everytime we run the script and duplicity doesn’t like it, hence we have to force it to accept the new path.
    I also added the --volsize 2048 option because with some FTP daemons + filesystems combinations the max file size is 2GB. It’s safe to let duplicity split the backup archives in 2GB slices.

    Thanks again for your feedback! Keep it coming in :)

  7. Great, that fixed it and it is working fine now. Thanks for your help on this. Might be worth mentioning in your intro about using dos2unix when you copy and past the script. That caught me out early on.

    Thanks again
    Cheers Andy

  8. Hi,

    can you please help me with this: I’m not sure do I understand the part “make a snapshot of the LVM volume mounted under /opt/zimbra”. Do I have to mount /opt/zimbra somehow, before I can use the script?

  9. This backup script assumes that you have a LVM volume mounted under /opt/zimbra and that zimbra is installed on it.
    The script will create a snapshot of that volume, mount it under a temporary path, do a backup, destroy the snapshot.
    If you you don’t already use LVM then this script is not for you.

  10. Let me try to explain you what is the main difference between the “classic” backup approach, and the LVM snapshot approach.
    If your system does NOT uses LVM, you have to stop zimbra, do a backup of /opt/zimbra (with whatever tool you like), restart zimbra.
    If your system uses LVM, you can stop zimbra, create a LVM snapshot, restart zimbra, do a backup of the LVM snapshot (with whatever tool you like), destroy the LVM snapshot.
    The obvious advantage is that with the LVM snapshot approach you must stop zimbra only for the time needed to create the LVM snapshot.
    Without LVM, you have to stop zimbra for the time needed to perform the backup.
    The time needed to create the LVM snapshot is always a few seconds. The time needed to perform the backup is variable: from a few minutes to a few hours, depending on how big your mail spool is.
    Hope it makes sense.

  11. This makes perfect sense, thank you. It seems that only usable way to back up Zimbra is this. I should have realized that before installation, there is no way to enable LVM afterwards?

  12. Hi Again :)

    I have noticed that I get some “command not found” errors when running as a cron job but it works fine when run manually through console. Been searching and it looks like it might be a path issue but not quite sure how to fix it. Any suggestions would be great. Here is the cron job output:

    Backup started at Tue Dec 21 02:00:01 NZDT 2010
    Stopping the Zimbra services…
    Creating a snapshot called ZimbraBackup
    /backup/./zmbklvm.sh: line 28: -L2G: command not found
    Creating a mountpoint for the LV…
    Mounting the snapshot…
    mount: special device /dev/vg1/ZimbraBackup does not exist
    Restarting the Zimbra services…
    No old backup sets found, nothing deleted.
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    gpg: CAST5 encrypted data
    gpg: encrypted with 1 passphrase
    Unmounting and removing the snapshot.
    umount: /dev/vg1/ZimbraBackup: not found
    /backup/./zmbklvm.sh: line 52: –force: command not found
    /dev/vg1/ZimbraBackup
    Zimbra backed up to zimbra on 192.168.1.3
    Backup ended at Tue Dec 21 02:11:11 NZDT 2010

  13. These errors show that $lvcreate_cmd and $lvremove_cmd were not properly set. Please replace these lines:


    lvcreate_cmd=
    which lvcreate
    lvremove_cmd=which lvremove

    with something like:


    lvcreate_cmd="/sbin/lvcreate"
    lvremove_cmd="/sbin/lvremove"

    or wherever you have lvcreate and lvremove installed.

  14. Hello,

    Excellent script.

    How can I use this script with a server that does not use default port of 22? My backup servers use other ports such as 9291 for example.

    Will this work with Cron?

    Thank you.

    Luma

  15. My script by default uses FTP. If you are using scp as the transfer method and the destination host is listening on port 9291, you should add the following to each $DUPLICITY command:
    –ssh-options=”-oPort=9291″.

    I see no reason why this script would not run under cron.

  16. Hi. I have had a couple of occasions recently ( 1 a month ago and 1 last night) where during the backup process Zimbra is stopped and never restarts. The script has been running flawlessly for months prior to that. I am trying to find out why. The duplicity files are not written to the ftp backup so something is stopping the process but it seems to be random and rare.

    I ma just wondering if it is to do with” depending on the time it takes to create the backup and the traffic of your server, you may need to increase the 2G value”. Can you let me know what problems the 2G value could have?

    Thanks
    Andy

  17. Strange. The 2G value means that after generating the LVM snapshot, zimbra can write up to 2GB of data to the “live” LVM volume. In other words: during the time that it takes to generate the backup, zimbra can receive up to ~2GB of mail (including logs, metadata, …)

  18. Hey Luca:

    I know this thread is old, but I have tried to get your script working without success. The mount works and temp directories are created but Duplicity fails every time with the following (from the logfile):

    ******************************************************************************************************************
    Using archive dir: /root/.cache/duplicity/339d1f0b74e9d7a2165a3d08a9e124da
    Using backup name: 339d1f0b74e9d7a2165a3d08a9e124da
    Import of duplicity.backends.ftpbackend Succeeded
    Import of duplicity.backends.localbackend Succeeded
    Import of duplicity.backends.ftpsbackend Succeeded
    Import of duplicity.backends.sshbackend Failed: No module named paramiko
    Import of duplicity.backends.webdavbackend Succeeded
    Import of duplicity.backends.rsyncbackend Succeeded
    Import of duplicity.backends.u1backend Succeeded
    Import of duplicity.backends.botobackend Succeeded
    Import of duplicity.backends.tahoebackend Succeeded
    Import of duplicity.backends.giobackend Succeeded
    Import of duplicity.backends.imapbackend Succeeded
    Import of duplicity.backends.cloudfilesbackend Succeeded
    Import of duplicity.backends.gdocsbackend Succeeded
    Import of duplicity.backends.hsibackend Succeeded
    NcFTP version is 3.2.5
    Using temporary directory /tmp/duplicity-0kxU1b-tempdir
    Main action: cleanup
    ================================================================================
    duplicity 0.6.18 (February 29, 2012)
    Args: /usr/bin/duplicity cleanup –force -v5 ftp://zimbra_Ftp@192.138.1.3/zimbra7/
    Linux zmail 3.2.0-33-generic #52-Ubuntu SMP Thu Oct 18 16:29:15 UTC 2012 x86_64 x86_64
    /usr/bin/python 2.7.3 (default, Aug 1 2012, 05:14:39)
    [GCC 4.6.3]
    ================================================================================
    Reading results of ‘ncftpls -f /tmp/duplicity-0kxU1b-tempdir/mkstemp-RWBesm-1 -F -t 30 -o useCLNT=0,useHELP_SITE=0 -l ‘ftp://192.138.1.3/zimbra7/”
    Running ‘ncftpls -f /tmp/duplicity-0kxU1b-tempdir/mkstemp-RWBesm-1 -F -t 30 -o useCLNT=0,useHELP_SITE=0 -l ‘ftp://192.138.1.3/zimbra7/” failed with code 1 (attempt #1)
    Error is:
    Could not connect to 192.138.1.3 — try again later: Connection timed out.
    Could not connect to 192.138.1.3 — try again later: Connection timed out.
    Could not connect to 192.138.1.3 — try again later: Connection timed out.
    ncftpls: cannot open 192.138.1.3: could not connect to remote host, but can try again.

    Reading results of ‘ncftpls -f /tmp/duplicity-0kxU1b-tempdir/mkstemp-RWBesm-1 -F -t 30 -o useCLNT=0,useHELP_SITE=0 -l ‘ftp://192.138.1.3/zimbra7/”
    ***********************************************************************************************************************

    Now I can connect with a test duplicity script to the same FTP location using the same PW & user so I know Duplicity is working. It is something in the script but I just don’t know where to look. Any ideas??

    Rick

  19. Ciao Luca,

    Thanks for the script. Question: what about hard links? Apparently Zimbra uses lot of them and Duplicity doesn’t support hard links. Is this correct?

  20. Indeed, that means you’ll be wasting some disk space. Not much you can do about it, you might want to use another backup software or create a tarball of /opt/zimbra and backup the tarball instead (tar preserves hardlinks by default).

Leave a Reply