Borgmatic#
Let’s say we want to backup the root (/
) partition. By using Borgmatic and these
scripts you get:
compressed and differential backups
optional notifications before, after and during the backup
possibility browse through all versions of a backup and to have optional notifications about the mount action
Vedi anche
A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks [1]
borgmatic [2]
How to monitor your backups - borgmatic [3]
How to deal with very large backups - borgmatic [4]
Low power consumption home backup server project - Where is it? [5]
Borgmatic Systemd unit service file [6]
Borgmatic Systemd unit timer file [7]
GitHub - caronc/apprise: Apprise - Push Notifications that work with just about every platform! [11]
Setup#
install the dependencies
apt-get install borgmatic python3-yaml python3-venv
create the jobs directories. See reference
mkdir -p /home/jobs/{scripts,services}/by-user/root chmod 700 -R /home/jobs/{scripts,services}/by-user/root
create a new virtual environment
cd /home/jobs/scripts/by-user/root python3 -m venv .venv_borgmatic_hooks . .venv_borgmatic_hooks/bin/activate
install fpyutils. See reference
create the
requirements_notify_unit_status.txt
fileapprise PyYAML
install the dependencies
pip3 install -r requirements_borgmatic_hooks.txt deactivate
create the
hooks script
1#!/usr/bin/env python3 2# 3# borgmatic_hooks.py 4# 5# Copyright (C) 2021-2024 Franco Masotti 6# 7# This program is free software: you can redistribute it and/or modify 8# it under the terms of the GNU General Public License as published by 9# the Free Software Foundation, either version 3 of the License, or 10# (at your option) any later version. 11# 12# This program is distributed in the hope that it will be useful, 13# but WITHOUT ANY WARRANTY; without even the implied warranty of 14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15# GNU General Public License for more details. 16# 17# You should have received a copy of the GNU General Public License 18# along with this program. If not, see <https://www.gnu.org/licenses/>. 19r"""Send a notification after the backup operation.""" 20 21import shlex 22import sys 23 24import apprise 25import yaml 26 27if __name__ == '__main__': 28 configuration_file = shlex.quote(sys.argv[1]) 29 config = yaml.load(open(configuration_file), Loader=yaml.SafeLoader) 30 action = sys.argv[2] 31 borgmatic_config_file = shlex.quote(sys.argv[3]) 32 repository = sys.argv[4] 33 output = sys.argv[5] 34 error = sys.argv[6] 35 36 message = action + ' on ' + repository + ' using ' + borgmatic_config_file + ': ' + '\n' + 'output: ' + output + '\n' + 'error: ' + error 37 38 # Create an Apprise instance. 39 apobj = apprise.Apprise() 40 41 # Add all of the notification services by their server url. 42 for uri in config['apprise_notifiers']['dest']: 43 apobj.add(uri) 44 45 apobj.notify( 46 body=message, 47 title=config['apprise_notifiers']['title'], 48 )
create the
mount script
1#!/usr/bin/env python3 2# 3# borgmatic_mount.py 4# 5# Copyright (C) 2020-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 6# 7# This program is free software: you can redistribute it and/or modify 8# it under the terms of the GNU General Public License as published by 9# the Free Software Foundation, either version 3 of the License, or 10# (at your option) any later version. 11# 12# This program is distributed in the hope that it will be useful, 13# but WITHOUT ANY WARRANTY; without even the implied warranty of 14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15# GNU General Public License for more details. 16# 17# You should have received a copy of the GNU General Public License 18# along with this program. If not, see <https://www.gnu.org/licenses/>. 19r"""Mount a borg backup.""" 20 21import pathlib 22import shlex 23import sys 24 25import fpyutils 26import yaml 27 28if __name__ == '__main__': 29 configuration_file = shlex.quote(sys.argv[1]) 30 config = yaml.load(open(configuration_file), Loader=yaml.SafeLoader) 31 # Action may only be "mount" or "umount". 32 action = shlex.quote(sys.argv[2]) 33 34 pathlib.Path(shlex.quote(config['files']['mountpoint'])).mkdir( 35 0o700, exist_ok=True, parents=True) 36 fpyutils.shell.execute_command_live_output( 37 shlex.quote(config['files']['borgmatic_binary']) + ' --config ' + 38 shlex.quote(config['files']['borgmatic_config']) + ' ' + action + 39 ' --mount-point ' + shlex.quote(config['files']['mountpoint'])) 40 41 message = 'borgmatic_mount using ' + shlex.quote( 42 config['files']['borgmatic_config'] 43 ) + '\n' + 'action: ' + action + '\n' + 'on: ' + shlex.quote( 44 config['files']['mountpoint']) 45 if config['notify']['gotify']['enabled']: 46 m = config['notify']['gotify']['message'] + '\n' + message 47 fpyutils.notify.send_gotify_message( 48 config['notify']['gotify']['url'], 49 config['notify']['gotify']['token'], m, 50 config['notify']['gotify']['title'], 51 config['notify']['gotify']['priority']) 52 if config['notify']['email']['enabled']: 53 fpyutils.notify.send_email(message, 54 config['notify']['email']['smtp_server'], 55 config['notify']['email']['port'], 56 config['notify']['email']['sender'], 57 config['notify']['email']['user'], 58 config['notify']['email']['password'], 59 config['notify']['email']['receiver'], 60 config['notify']['email']['subject'])
create the
borgmatic service
1# 2# borgmatic.myhostname_root.service 3# 4# Copyright (C) 2016-2020 Dan Helfman <https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.service> 5# 2020-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 6# 7# This program is free software: you can redistribute it and/or modify 8# it under the terms of the GNU General Public License as published by 9# the Free Software Foundation, either version 3 of the License, or 10# (at your option) any later version. 11# 12# This program is distributed in the hope that it will be useful, 13# but WITHOUT ANY WARRANTY; without even the implied warranty of 14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15# GNU General Public License for more details. 16# 17# You should have received a copy of the GNU General Public License 18# along with this program. If not, see <https://www.gnu.org/licenses/>. 19 20[Unit] 21Description=borgmatic myhostname_root backup 22Wants=network-online.target 23After=network-online.target 24ConditionACPower=true 25Requires=backed-up-mountpoint.mount 26Requires=mnt-backups-myhostname_root.mount 27After=backed-up-mountpoint.mount 28After=mnt-backups-myhostname_root.mount 29 30[Service] 31Type=oneshot 32 33# Lower CPU and I/O priority. 34Nice=19 35CPUSchedulingPolicy=batch 36IOSchedulingClass=best-effort 37IOSchedulingPriority=7 38IOWeight=100 39 40# Do not retry. 41Restart=no 42 43# Prevent rate limiting of borgmatic log events. If you are using an older version of systemd that 44# doesn't support this (pre-240 or so), you may have to remove this option. 45LogRateLimitIntervalSec=0 46 47# Delay start to prevent backups running during boot. 48ExecStartPre=/usr/bin/sleep 1m 49ExecStart=/usr/bin/systemd-inhibit --who="borgmatic" --why="Prevent interrupting scheduled backup" /usr/bin/borgmatic --config /home/jobs/scripts/by-user/root/borgmatic.myhostname_root.yaml --syslog-verbosity 1 50 51User=root 52Group=root 53 54[Install] 55WantedBy=multi-user.target
create the
borgmatic timer
1# 2# borgmatic.myhostname_root.timer 3# 4# Copyright (C) 2020-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 5# 6# This program is free software: you can redistribute it and/or modify 7# it under the terms of the GNU General Public License as published by 8# the Free Software Foundation, either version 3 of the License, or 9# (at your option) any later version. 10# 11# This program is distributed in the hope that it will be useful, 12# but WITHOUT ANY WARRANTY; without even the implied warranty of 13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14# GNU General Public License for more details. 15# 16# You should have received a copy of the GNU General Public License 17# along with this program. If not, see <https://www.gnu.org/licenses/>. 18 19[Unit] 20Description=Once a day borgmatic myhostname_root backup 21 22[Timer] 23OnCalendar=*-*-* 3:00:00 24Persistent=true 25 26[Install] 27WantedBy=timers.target
create the
borgmatic mount service
1# 2# borgmatic-mount.myhostname_root.service 3# 4# Copyright (C) 2020-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 5# 6# This program is free software: you can redistribute it and/or modify 7# it under the terms of the GNU General Public License as published by 8# the Free Software Foundation, either version 3 of the License, or 9# (at your option) any later version. 10# 11# This program is distributed in the hope that it will be useful, 12# but WITHOUT ANY WARRANTY; without even the implied warranty of 13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14# GNU General Public License for more details. 15# 16# You should have received a copy of the GNU General Public License 17# along with this program. If not, see <https://www.gnu.org/licenses/>. 18 19[Unit] 20Description=borgmatic myhostname_root mount and umount 21 22[Service] 23Type=simple 24RemainAfterExit=yes 25ExecStart=/home/jobs/scripts/by-user/root/borgmatic_mount.py /home/jobs/scripts/by-user/root/borgmatic_mount.myhostname_root.yaml 'mount' 26ExecStop=/home/jobs/scripts/by-user/root/borgmatic_mount.py /home/jobs/scripts/by-user/root/borgmatic_mount.myhostname_root.yaml 'umount' 27User=root 28Group=root
create a
borgmatic hooks configuration file
1# 2# borgmatic_hooks.myhostname_your_source_mountpoint.conf 3# 4# Copyright (C) 2020-2024 Franco Masotti 5# 6# This program is free software: you can redistribute it and/or modify 7# it under the terms of the GNU General Public License as published by 8# the Free Software Foundation, either version 3 of the License, or 9# (at your option) any later version. 10# 11# This program is distributed in the hope that it will be useful, 12# but WITHOUT ANY WARRANTY; without even the implied warranty of 13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14# GNU General Public License for more details. 15# 16# You should have received a copy of the GNU General Public License 17# along with this program. If not, see <https://www.gnu.org/licenses/>. 18 19apprise_notifiers: 20 dest: 21 - 'nctalks://<string>/' 22 - 'mailtos://<string>' 23 title: 'borgmatic backup'
create a
borgmatic mount configuration file
1# 2# borgmatic_mount.myhostname_root.yaml 3# 4# Copyright (C) 2020-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 5# 6# This program is free software: you can redistribute it and/or modify 7# it under the terms of the GNU General Public License as published by 8# the Free Software Foundation, either version 3 of the License, or 9# (at your option) any later version. 10# 11# This program is distributed in the hope that it will be useful, 12# but WITHOUT ANY WARRANTY; without even the implied warranty of 13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14# GNU General Public License for more details. 15# 16# You should have received a copy of the GNU General Public License 17# along with this program. If not, see <https://www.gnu.org/licenses/>. 18 19files: 20 borgmatic_config: '/home/jobs/scripts/by-user/root/borgmatic.myhostname_root.yaml' 21 mountpoint: '/tmp/borgmatic.myhostname_root' 22 borgmatic_binary: '/usr/bin/borgmatic' 23 24notify: 25 email: 26 enabled: true 27 smtp_server: 'smtp.gmail.com' 28 port: 465 29 sender: 'myusername@gmail.com' 30 user: 'myusername' 31 password: 'my awesome password' 32 receiver: 'myusername@gmail.com' 33 subject: 'borgmatic mount' 34 gotify: 35 enabled: true 36 url: '<gotify url>' 37 token: '<app token>' 38 title: 'borgmatic mount' 39 message: 'borgmatic mount' 40 priority: 5
New repository#
Before enabling the backup you need to decide exactly where to backup the directory, what to exclude, etc…
In this example we are not using Borg’s encryption because:
it works with older versions of borg
data is simpler to recover in case things go wrong
these are not to be considered offsite backups
By using Borgmatic you can either decide to backup on a local directory or on a remote machine via SSH.
Vedi anche
borg init — Borg - Deduplicating Archiver 1.2.0 documentation [8]
Types of backup#
Run as user |
Instruction number |
|
* |
If you are backing up on a local directory you simply have to create a new repository according to the configuration.
create a new
configuration file
for the local backup1# 2# borgmatic.myhostname_root.yaml 3# 4# Copyright (C) 2014-2020 Dan Helfman <https://torsion.org/borgmatic/docs/reference/config.yaml> 5# 2020-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 6# 7# This program is free software: you can redistribute it and/or modify 8# it under the terms of the GNU General Public License as published by 9# the Free Software Foundation, either version 3 of the License, or 10# (at your option) any later version. 11# 12# This program is distributed in the hope that it will be useful, 13# but WITHOUT ANY WARRANTY; without even the implied warranty of 14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15# GNU General Public License for more details. 16# 17# You should have received a copy of the GNU General Public License 18# along with this program. If not, see <https://www.gnu.org/licenses/>. 19 20location: 21 source_directories: 22 - /backed/up/mountpoint 23 repositories: 24 - /mnt/backups/myhostname_root.borg 25 exclude_patterns: 26 - /data 27 - /proc 28 - /sys 29 - /tmp 30 - /run 31 - /mnt 32 - /media 33 - /lost+found 34storage: 35 checkpoint_interval: 900 36 lock_wait: 120 37retention: 38 keep_within: 1w 39 keep_monthly: 1 40consistency: 41 checks: 42 - disabled 43output: 44 color: false 45hooks: 46 before_backup: 47 - /bin/sh -c '. /home/jobs/scripts/by-user/root/.venv_borgmatic_hooks/bin/activate && /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.myhostname_root.yaml "start" "{configuration_filename}" "{repository}" "{output}" "{error}"; deactivate' 48 after_backup: 49 - /bin/sh -c '. /home/jobs/scripts/by-user/root/.venv_borgmatic_hooks/bin/activate && /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.myhostname_root.yaml "finish" "{configuration_filename}" "{repository}" "{output}" "{error}"; deactivate' 50 on_error: 51 - /bin/sh -c '. /home/jobs/scripts/by-user/root/.venv_borgmatic_hooks/bin/activate && /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.myhostname_root.yaml "error" "{configuration_filename}" "{repository}" "{output}" "{error}"; deactivate'
create the new repository
borg init -e none /mnt/backups/myhostname_root.borg
Run as user |
Instruction number |
|
2,4-5 |
|
1 |
|
3 |
connect to the remote machine via SSH and create a new user called
borgmatic
useradd -m -s /bin/bash -U borgmatic passwd borgmatic
quit the remote machine and create SSH key pair: do not add a passphrase when prompted
ssh-keygen -t rsa -b 16384 -C "root@myhostname-$(date "+%F")"
add the public key to the authorized keys file on remote machine
cat mykey.pub >> ~/.ssh/authorized_keys
create the new repository
borg init -e none user@remote-host:/mnt/backups/myhostname_root.borg
create a new
configuration file
for the backup1# 2# borgmatic.myhostname_root.yaml 3# 4# Copyright (C) 2014-2020 Dan Helfman <https://torsion.org/borgmatic/docs/reference/config.yaml> 5# 2020-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) 6# 7# This program is free software: you can redistribute it and/or modify 8# it under the terms of the GNU General Public License as published by 9# the Free Software Foundation, either version 3 of the License, or 10# (at your option) any later version. 11# 12# This program is distributed in the hope that it will be useful, 13# but WITHOUT ANY WARRANTY; without even the implied warranty of 14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 15# GNU General Public License for more details. 16# 17# You should have received a copy of the GNU General Public License 18# along with this program. If not, see <https://www.gnu.org/licenses/>. 19 20location: 21 source_directories: 22 - /backed/up/mountpoint 23 repositories: 24 - borgmatic@remote-host:/mnt/backups/myhostname_root.borg 25 exclude_patterns: 26 - /data 27 - /proc 28 - /sys 29 - /tmp 30 - /run 31 - /mnt 32 - /media 33 - /lost+found 34storage: 35 checkpoint_interval: 900 36 lock_wait: 120 37retention: 38 keep_within: 1w 39 keep_monthly: 1 40consistency: 41 checks: 42 - disabled 43output: 44 color: false 45hooks: 46 before_backup: 47 - /bin/sh -c '. /home/jobs/scripts/by-user/root/.venv_borgmatic_hooks/bin/activate && /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.myhostname_root.yaml "start" "{configuration_filename}" "{repository}" "{output}" "{error}"; deactivate' 48 after_backup: 49 - /bin/sh -c '. /home/jobs/scripts/by-user/root/.venv_borgmatic_hooks/bin/activate && /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.myhostname_root.yaml "finish" "{configuration_filename}" "{repository}" "{output}" "{error}"; deactivate' 50 on_error: 51 - /bin/sh -c '. /home/jobs/scripts/by-user/root/.venv_borgmatic_hooks/bin/activate && /home/jobs/scripts/by-user/root/borgmatic_hooks.py /home/jobs/scripts/by-user/root/borgmatic_hooks.myhostname_root.yaml "error" "{configuration_filename}" "{repository}" "{output}" "{error}"; deactivate'
Deploy#
run the deploy script
check if the backup service works
Nota
If you get an unknown repo error try adding this key:
storage.unknown_unencrypted_repo_access_is_ok: true
Mount#
To mount the backup you just have to run the Systemd service
systemctl start borgmatic-mount.myhostname_root.service
To unmount, stop the service
systemctl stop borgmatic-mount.myhostname_root.service
Other#
Databases#
Borgmatic also support backing up databases. Just put []
as location.source_directories
and add a database object in the hooks
section.
Vedi anche
How to backup your databases - borgmatic [9]
Running the service before computer shutdown#
Vedi anche
linux - How do I run a script before everything else on shutdown with systemd? - Super User [10]
change some of the original borgmatic service Systemd unit file lines with these:
ExecStart=/bin/true RemainAfterExit=yes TimeoutStopSec=infinity ExecStop=/usr/bin/borgmatic --config /home/jobs/scripts/by-user/root/borgmatic.myhostname_root.yaml --syslog-verbosity 1
Although this service remains active all the time, the syncronization action
runs when the system is halted using an ExecStop
directive. Since we don’t
know how much time the syncronization takes, a TimeoutStopSec=infinity
directive is present.
Automatic backup on a removable USB drive on plug in#
create a partition on a block device
create a filesystem. In this example the filsystem’s UUID is
ABCD-0123
and is obtained with this commandudevadm info --name=${partition} | grep "ID_FS_UUID="
add its entry in the
fstab file
, for example1UUID=ABCD-0123 /mnt/backups/myhostname_root.borg btrfs nofail,auto,rw,relatime 0 0
create a new
udev rule
to enable the auto mounting1ACTION=="add", SUBSYSTEMS=="usb", ENV{ID_FS_UUID}=="ABCD-0123", SUBSYSTEM=="block", RUN{program}+="/usr/bin/bash -c '(/usr/bin/systemctl start root.mount && systemctl start borgmatic.myhostname_root.service && /usr/bin/systemctl start udev-umount.myhostname_root.service) &'"
reboot to apply the rules
plug in the USB device and check if the service starts
Mounting backups on other computers#
Let’s say computer A backups on computer B, but A is offline.
You can access A’s backups via B on a third computer, C. C just needs to have SSH access to B and a copy of A’s original borgmatic files.
Footnotes