Backups

Blunix uses borgbackup to backup all instances. Blunix backs up all files in all mounted filesystems, including the / partition.

For databases and alike, hooks to run scripts like mysqldump are triggered before the backup is taken.

ansible-roles

• git.blunix.com/ansible-roles/role-borgbackup-server
• git.blunix.com/ansible-roles/role-borgbackup-client

Principle of operation

Simplified, borgbackup works as follows:

client: the Debian installation that has to be backed up server: the Debian installation that stores the backup files

• the server runs the command borg serve, which allows backups to be saved to its disk
• the client connects to the server using SSH and delivers files in encrypted form to be backed up

Files and directories on the backup server

File	Purpose
/home/borgbackup/archives/	Directory where all client backups are stored
/home/borgbackup/logs	Directory for borgbackup logfiles
/home/borgbackup/.ssh	SSH private key directory for accessing the clients
/home/borgbackup/prune.sh	BASH script for cronjob to remove outdated backups

Files and directories on the backup clients

File	Purpose
/root/.ssh/authorized_keys	File where the borgbackup servers SSH public key is stored (restricted to the specific backup command)
/root/.bashrc.d/borgbackup.sh	BASH script that is sourced when starting /bin/bash as interactive shell, sourcing variables so the client can use the borg command
/usr/local/sbin/borgbackup-create.sh	BASH script that creates the backup
/etc/borgbackup/hooks	Directory to store bash scripts which are run before a backup
/var/log/borgbackup	Directory where backup hook logs are stored

Common commands

Gathering information

Show borg help

root@client ~ # borg help
root@client ~ # borg help create
root@client ~ # borg help mount

List all backups of a server

root@client ~ # borg list
04_01_22-01_08_01                    Tue, 2022-01-04 01:08:02 [a62k61f0732803841zaca6326b813o986a61323a89r719817ed1by33df9550c0]
05_01_22-01_08_01                    Wed, 2022-01-05 01:08:02 [1f9a3789c9a2950l4bab3id37b22585838v7708791ac4f01d376c3kcc1265f02]
06_01_22-01_08_01                    Thu, 2022-01-06 01:08:02 [a2q5e901bn7aa2cam681240p2dc45v3b24a09ee02faf27f4949w98923e978f0a]
[...]

List all backups of all servers

In ansible-cake, run:

cake master * cake -i pub -f debug_borg_backups -nc

Listing files in a backup

Note that !!! THERE IS NO LEADING SLASH BEFORE THE DIRECTORY NAME !!!

root@client ~ # borg list $BORG_REPO::04_01_22-01_08_01 home/
drwxr-xr-x root   root          0 Mon, 2022-01-03 16:24:26 home
drwxr-x--- borgbackup borgbackup        0 Sat, 2022-01-22 12:55:15 home/borgbackup
-rw------- borgbackup users        13 Mon, 2022-01-03 16:40:21 home/borgbackup/.bash_history
-r-x------ borgbackup borgbackup    13509 Sat, 2022-01-22 12:55:13 home/borgbackup/prune.sh
[...]

Show differences between two backups

Read the syntax carefully! The $BORG_REPO variable comes from /root/.bashrc.d/borgbackup.sh, which is automatically loaded when you ssh onto a server and open an interactive /bin/bash as root.
NOTE that the $BORG_REPO variable is only stated before the first archive name!

root@client ~ # borg diff $BORG_REPO::04_01_22-01_08_01 05_01_22-01_08_01
    +26 B     -26 B root/.config/borg/security/a59016e5bee18c3e825b352166c0d69c087ed139a08ad8ed679e14136ce2047b/manifest-timestamp
    +16 B     -16 B root/.config/borg/security/a59016e5bee18c3e825b352166c0d69c087ed139a08ad8ed679e14136ce2047b/nonce
changed link        run/systemd/units/invocation:atop.service
changed link        run/systemd/units/invocation:user-runtime-dir@998.service
changed link        run/systemd/units/invocation:user@998.service
[...]

Restoring backups

Mounting a backup archive

Useful for restoring a few files. Not very performant. Do not forget to borg umount /mnt after you are done!

root@client ~ # borg mount $BORG_REPO:05_01_22-01_08_01 /mnt

root@client ~ # ls /mnt
bin  boot  dev  etc  home  initrd.img  initrd.img.old  lib  lib32  lib64  libx32  lost+found  opt  root  run  sbin  srv  usr  vmlinuz  vmlinuz.old

root@client ~ # borg umount /mnt
root@client ~ # ls /mnt
[empty]

Restoring a complete backup

You most likely only want to restore a specific path, see below at "Restoring a directory".

root@client ~ # mkdir restore
root@client ~ # cd restore

root@client ~/restore # borg extract $BORG_REPO::05_01_22-01_08_01

root@client ~/restore # ls
bin  boot  dev  etc  home  initrd.img  initrd.img.old  lib  lib32  lib64  libx32  lost+found  opt  root  run  sbin  srv  usr  vmlinuz  vmlinuz.old

Restoring a directory

root@client ~ # mkdir restore
root@client ~ # cd restore

# For small restores add --progress
root@client ~/restore # borg extract --progress $BORG_REPO::05_01_22-01_08_01 home
# For large restores omit --progress
root@client ~/restore # borg extract $BORG_REPO::05_01_22-01_08_01 home

root@client ~/restore # ls
home

Restoring a directory to the exact state of the backup

Borgbackup sadly does not have a rsync --delete like functionality. Hence, we have to manually diff the files and directories and restore accordingly.

TODO

Creating backups

While working on a server, you may want to backup a specific state before applying changes.

Automatic backups

Backups are created daily by default. To view the cronjob, run:

root@client ~ # crontab -e

Creating a full backup

The script will:

• first trigger backup hooks
• then create a backup excluding common paths like /proc, /var/lib/mysql and alike

If in doubt, make sure to read the script first.

root@client ~ # /usr/local/sbin/borgbackup-create.sh

Creating a backup of a specific path

root@client ~ # borg create $BORG_REPO::my_backup_name /home

Deleting backups

Deleting a specific backup (client)

Clients can remove backups from the index of present backups, but can not actually delete the backup files on the server. This means that if you run borg remove on a server that is not the backup server, you will remove the backup from the index (borg list output), but the files will still be present on the backup server.

This quite frankly sucks about borgbackup - clients should not be able to mess with the index.

Deleting a specific backup (server)

Only the borgbackup server has the permission to actually delete files. This is done daily with borg prune using server:/home/borgbackup/prune.sh to delete old backups.

In general, you should not manually delete backups at all. However you may want to delete temporary backups. In this example, we will create a temporary backup and then delete it.

# Creating a temporary backup
root@client ~ # borg create $BORG_REPO::my_backup_name /home
root@client ~ # borg list $BORG_REPO my_backup_name

# Removing the temporary backup on the server
root@cus-util-prod-backup-1 ~ # TODO

Restore a backup that was deleted from the index

When a client deleted a backup from its index, you can restore it as follows.

TODO

=================================

Backups

For borg backup, the backup server has to prepared first to create storage directories for the new servers.

For this, edit plays/util/server/borgbackup.yml and add the new servers to the variable borgbackup_server_clients.

In addition borg backup encryption passwords have to be generated for the new servers like so:

cake pwgen borgbackup_passphrase_cus-www-prod-web-1

Then run the borgbackup server function:

cake -f util_server_borgbackup -nc

After that, the new servers can be setup with the borgbackup clients:

cake -f util_client_borgbackup -l cus_www_prod -nc

Note that the cronjobs for the backups are running at random times - this means that if you run the util_client_borgbackup function twice, the times for the cronjobs will be changed. That i s expected behavior and you do not have to worry about it.