How to Backup Using Duplicity on Ubuntu 20.04
Abstract: you will need to create a directory on the backup server to store the backup. Login to the backup server and create a backup directory with the follow
If you are a system administrator and responsible for managing servers than backing up data is one of the most important tasks of you. So you must have enough knowledge of backup tools that makes your task easier.
Duplicity is a free, open-source and an advanced command-line backup utility built on top of librsync and GnuPG. It produces digitally signed, versioned and encrypted tar volumes for storage on a local or remote computer. Duplicity supports many protocols for connecting to a file server including, ssh/scp, rsync, ftp, DropBox, Amazon S3, Google Docs, Google Drive, local filesystem, OneDrive, WebDAV and many more.
In this tutorial, we will show you how to install and configure Duplicity to perform manually and automate backup on Ubuntu 20.04 server.
SQL Server Back Up and Restore usin...To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video
SQL Server Back Up and Restore using Script Prerequisites• Two servers running Ubuntu 20.04.
• A static IP address 104.245.32.161 is configured on the Duplicity server and 104.245.32.194 is configured on the backup server.
• A root password is configured in your server.
Before starting, you will need to update your package to the latest version. You can update all of them with the following command:
apt-get update -yAfter updating all packages, you will also need to install some required dependencies on your server. You can install all of them by running the following command:
apt-get install ncftp python3-paramiko python-pycryptopp lftp python3-boto python3-dev librsync-dev -yOnce all the packages are installed, you can proceed to install Duplicity.
Install DuplicityBy default, Duplicity is available in the Ubuntu 20.04 default repository. You can install it by just running the following command:
apt-get install duplicity -yOnce installed, verify the installed version of Duplicity with the following command:
duplicity --versionYou should see the following output:
duplicity 0.8.12Next, you will need to generate an RSA 2048 bit encrypted SSH key and use it to securely authenticate with the remote system without providing a password.
First, generate an RSA key with the following command:
ssh-keygen -t rsa -m PEMYou will be asked to specify the location of the key and passphrase as shown below:
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:BwKSLtXT+WHhPqhu3MIH07QEb0c5Zh/f0XolMRyQaQE root@ubuntu2004
The key's randomart image is:
+---[RSA 3072]----+
| .o.. ..oE.o*++ |
| o.+.o.O . + +.o|
| o +.*o+ + . +.|
|. . =+o.. . o .|
| . =.oS . . |
| o.o o |
| o.+ |
| .= o |
| ..o |
+----[SHA256]-----+
Next, copy the generated RSA key to the remote backup server with the following command:
ssh-copy-id [email protected]You will be asked to provide remote server password as shown below:
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.Next, you will also need to generate GPG keys to encrypt the data before we transfer it to the remote backup server. GPG keys are used to provide an extra layer of security and encryption of the data transferred across the servers.
You can GPG key with the following command on the Duplicity server:
gpg --gen-keyYou will be asked to provide the password of the GPG key during the Key generation process as shown below:
gpg (GnuPG) 2.2.19; Copyright (C) 2019 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
Note: Use "gpg --full-generate-key" for a full featured key generation dialog.
GnuPG needs to construct a user ID to identify your key.
Real name: Hitesh Jethva
Email address: [email protected]
You selected this USER-ID:
"Hitesh Jethva <[email protected]>"
Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key E8A334EB747BAD6A marked as ultimately trusted
gpg: directory '/root/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A.rev'
public and secret key created and signed.
pub rsa3072 2020-07-30 [SC] [expires: 2022-07-30]
6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A
uid Hitesh Jethva <[email protected]>
sub rsa3072 2020-07-30 [E] [expires: 2022-07-30]The above command will generate the GPG keys and save it under /root/.gnupg/ directory.
If you forget your GPG key, you can find it with the following command:
gpg --list-keysYou should get the following output:
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2022-07-30
/root/.gnupg/pubring.kbx
------------------------
pub rsa3072 2020-07-30 [SC] [expires: 2022-07-30]
6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A
uid [ultimate] Hitesh Jethva <[email protected]>
sub rsa3072 2020-07-30 [E] [expires: 2022-07-30]Next, you will need to create a directory on the backup server to store the backup.
Login to the backup server and create a backup directory with the following command:
mkdir -p /remotebackup/Duplicity/Once you are finished, you can proceed to the next step.
Perform Manual BackupAt this point, Duplicity is installed and configured. Now, let's perform a manual backup on the Duplicity server.
Run the following command to backup a directory named /etc to the remote backup server(104.245.32.194) inside /remotebackup/Duplicity/ directory:
duplicity /etc/ sftp://[email protected]//remotebackup/Duplicity/You will need to provide your GPG key password to perform the backup operation as shown below:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase for decryption:
Retype passphrase for decryption to confirm:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1596110093.56 (Thu Jul 30 11:54:53 2020)
EndTime 1596110095.24 (Thu Jul 30 11:54:55 2020)
ElapsedTime 1.67 (1.67 seconds)
SourceFiles 1413
SourceFileSize 2698964 (2.57 MB)
NewFiles 1413
NewFileSize 2698964 (2.57 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1413
RawDeltaSize 1857098 (1.77 MB)
TotalDestinationSizeChange 493579 (482 KB)
Errors 0
-------------------------------------------------
Now, login to the remote backup server and verify the backup with the following command:
ls /remotebackup/Duplicity/
You should see the following output:
duplicity-full.20200730T115821Z.manifest.gpg duplicity-full-signatures.20200730T115821Z.sigtar.gpg
duplicity-full.20200730T115821Z.vol1.difftar.gpgYou can also schedule your backup daily or weekly by setting Cron job to run it automatically at a specified interval of time.
First, you will need to create a passphrase file under /root directory to pass the passphrase automatically without an interactive prompt for paraphrase.
You can create it with the following command:
nano /root/.passphraseAdd your GPG key password as shown below:
PASSPHRASE="admin"Save and close the file then secure the file with the following command:
chmod 600 /root/.passphraseNow, create a daily backup script inside /etc/cron.daily/ directory that will run daily:
nano /etc/cron.daily/duplicity.incAdd the following lines:
#!/bin/sh
test -x $(which duplicity) || exit 0
. /root/.passphrase
export PASSPHRASE
$(which duplicity) --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A --exclude /var --exclude /home --exclude /swapfile --exclude /proc --exclude /sys --exclude /tmp --exclude /usr / sftp://[email protected]//remotebackup/Duplicity/Save and close the file when you are finished. Then, give executable permission to the script with the following command:
chmod +x /etc/cron.daily/duplicity.incThe above script will back up an entire root server excluding some of the folders.
Now test the script to make sure everything works properly.
/etc/cron.daily/duplicity.incYou should see the following output:
Local and Remote metadata are synchronized, no sync needed.
Last full backup left a partial set, restarting.
Last full backup date: Thu Jul 30 12:06:52 2020
RESTART: The first volume failed to upload before termination.
Restart is impossible...starting backup from beginning.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1596110992.57 (Thu Jul 30 12:09:52 2020)
EndTime 1596111062.63 (Thu Jul 30 12:11:02 2020)
ElapsedTime 70.06 (1 minute 10.06 seconds)
SourceFiles 13313
SourceFileSize 777994890 (742 MB)
NewFiles 13313
NewFileSize 777994890 (742 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 13313
RawDeltaSize 772067520 (736 MB)
TotalDestinationSizeChange 318263295 (304 MB)
Errors 0
-------------------------------------------------
You can now check the status of your backup with the following command:
PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A collection-status sftp://[email protected]//remotebackup/Duplicity/You should get the following output:
Last full backup date: Thu Jul 30 12:14:17 2020
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/b2af509c9fab82d874aa492a933aaf70
Found 0 secondary backup chains.
Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Thu Jul 30 12:14:17 2020
Chain end time: Thu Jul 30 12:14:17 2020
Number of contained backup sets: 1
Total number of contained volumes: 2
Type of backup set: Time: Num volumes:
Full Thu Jul 30 12:14:17 2020 2
-------------------------
No orphaned or incomplete backup sets found.If you want to take a full backup, run the following command:
PASSPHRASE="admin" duplicity full --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A remove-all-but-n-full 3 --force sftp://[email protected]//remotebackup/Duplicity/You can also clean the old backup files using the option remove-all-but-n-full. For example, if you want to remove all unwanted old backup files and retain only 2 backups and their corresponding incremental backups, run the following command:
PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A remove-all-but-n-full 2 --force sftp://[email protected]//remotebackup/Duplicity/You can also restore your files and directories from the remote backup server easily. You can use the --file-to-restore option to restore files and directories.
Before performing the restoration process, you can list all available files and directories on the backup server with the following command:
duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A list-current-files sftp://[email protected]//remotebackup/Duplicity/ | lessYou should see the following output:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Jul 30 12:14:17 2020
Thu Jul 30 12:04:04 2020 .
Thu May 14 03:28:54 2020 bin
Thu May 14 03:35:45 2020 boot
Mon Apr 20 16:33:49 2020 boot/System.map-5.4.0-26-generic
Wed Apr 29 13:34:16 2020 boot/System.map-5.4.0-29-generic
Mon Apr 20 16:33:49 2020 boot/config-5.4.0-26-generic
Wed Apr 29 13:34:16 2020 boot/config-5.4.0-29-generic
Thu May 14 03:37:26 2020 boot/grub
Thu May 14 03:31:36 2020 boot/grub/fonts
Thu May 14 03:31:36 2020 boot/grub/fonts/unicode.pf2
Thu May 14 03:31:29 2020 boot/grub/gfxblacklist.txt
Thu May 14 03:37:26 2020 boot/grub/grub.cfg
Wed Jul 29 10:05:41 2020 boot/grub/grubenv
Thu May 14 03:31:37 2020 boot/grub/i386-pc
Thu May 14 03:31:36 2020 boot/grub/i386-pc/915resolution.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/acpi.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/adler32.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/affs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/afs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/ahci.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/all_video.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/aout.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/archelp.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/at_keyboard.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/ata.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/backtrace.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bfs.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/biosdisk.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bitmap.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bitmap_scale.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/blocklist.mod
Thu May 14 03:31:37 2020 boot/grub/i386-pc/boot.img
Thu May 14 03:31:36 2020 boot/grub/i386-pc/boot.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/bsd.mod
Thu May 14 03:31:35 2020 boot/grub/i386-pc/bswap_test.mod
Thu May 14 03:31:36 2020 boot/grub/i386-pc/btrfs.modNow, restore the directory named "opt" from the backup server to the /opt directory by running the following command:
PASSPHRASE="admin" duplicity --encrypt-key 6683A9D4720C5895BF7A7AE9E8A334EB747BAD6A --file-to-restore opt sftp://[email protected]//remotebackup/Duplicity/ /optYou the restoration process has been completed successfully, you should see the following output:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Jul 30 12:14:17 2020In the above tutorial, we learned how to backup and restore files and directories with Duplicity on Ubuntu 20.04 server. We also learned how to perform the backup process automatically by scheduling a backup process with Cron. I hope this article will help you to save lots of time.
