Maintenance#

Software#

Database#

PostgreSQL#

Important

All these instructions have been tested with PostgreSQL version 13 only!

Changing collate#

Sometimes you might create a new database without specifying the collate information. PosgreSQL will use the default collate setting and there is no way to change it once the database is created. The only solution is to dump the database, to create a new one with the correct collate and finally to import the dump and drop the original database.

See also

What is Collation in Databases? 1

backup old database

pg_dump -U myuser olddb > olddb.bak.sql

create new database with correct collate

CREATE DATABASE newdb WITH OWNER myuser TEMPLATE template0 ENCODING UTF8 LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8';

import dump into new database

psql -U myuser -d newdb < olddb.bak.sql

rename old database

ALTER DATABASE olddb RENAME TO olddb_bak;

rename the new database to the original database name
```
ALTER DATABASE newdb RENAME TO olddb;
```
restart services and check that everything works
drop old database
```
DROP DATABASE olddb_bak;
```

Moving data directory#

If the data directory in the root partition is getting too large you can create a new partition (mounted on /postgresql in this example) and let PostgreSQL point to that one instead.

install the dependencies
```
apt-get install rsync
```
stop PostgreSQL
```
systemctl stop postgresql
```

copy the database directory

rsync -avAX /var/lib/postgresql/13/main /postgresql/13

change the data_directory setting in /etc/postgresql/postgresql.conf

data_directory = '/postgresql/13/main'      # use data in another directory

restart PostgreSQL
```
systemctl start postgresql
```

High availability#

In this example we configure two nodes. When the master node goes offline the backup node takes over the floating IP address. We replicate services on the backup node such as Apache and Unbound.

Keepalived is a tool which handles network replication for layers 3 and 4. Here you can find the configuration file for the master node (the backup node just needs minor edits).

A script you find below helps you copy the content of webservers, DNS server, etc on the backup node as well as restarting those services automatically.

See also

Keepalived for Linux 2
Is it possible to add a static mac address for a vrrp ip? · Issue #34 · osixia/docker-keepalived · GitHub 3
Building Layer 3 High Availability | Documentation 4

Key#

Node	IP	Hostname	Network interface name
MASTER	192.168.0.10	mst	eno1
BACKUP	192.168.0.11	bak	eno1
floating IP	192.168.0.100	-	-

Basic setup#

install the dependencies. Keepalived must be installed on all nodes
```
apt-get install keepalived rsync
```

create the configuration for the master node

/etc/keepalived/keepalived.conf#

global_defs {
    max_auto_priority -1
}

########################
## VRRP configuration ##
########################

# Identify the VRRP instance as, in this case, "failover_link".
vrrp_instance failover_link {

    # Initial state of the keepalived VRRP instance on this host
    # (MASTER or BACKUP). Once started, only priority matters.
    state MASTER

    # interface this VRRP instance is bound to.
    interface eno1

    # Arbitrary value between 1 and 255 to distinguish this VRRP
    # instance from others running on the same device. It must match
    # with other peering devices.
    virtual_router_id 1

    # Highest priority value takes the MASTER role and the
    # virtual IP (default value is 100).
    priority 110

    # Time, in seconds, between VRRP advertisements. The default is 1,
    # but in some cases you can achieve more reliable results by
    # increasing this value.
    advert_int 2

    use_vmac
    vmac_xmit_base

    # Authentication method: AH indicates ipsec Authentication Header.
    # It offers more security than PASS, which transmits the
    # authentication password in plaintext. Some implementations
    # have complained of problems with AH, so it may be necessary
    # to use PASS to get keepalived's VRRP working.
    #
    # The auth_pass will only use the first 8 characters entered.
    authentication {
        auth_type AH
        auth_pass f5K.*0Bq
    }

    # VRRP advertisements ordinarily go out over multicast. This
    # configuration paramter causes keepalived to send them
    # as unicasts. This specification can be useful in environments
    # where multicast isn't supported or in instances where you want
    # to limit which devices see your VRRP announcements. The IP
    # address(es) can be IPv4 or IPv6, and indicate the real IP of
    # other members.
    unicast_peer {
        192.168.0.11
    }
    # Virtual IP address(es) that will be shared among VRRP
    # members. "Dev" indicates the interface the virtual IP will
    # be assigned to. And "label" allows for clearer description of the
    # virtual IP.
    virtual_ipaddress {
        192.168.0.100 dev eno1 label eno1:vip
    }
}

Note

Copy this file in the backup node as well and change:

state MASTER to state BACKUP
unicast_peer { 192.168.0.11 } to unicast_peer { 192.168.0.10 }
priority 110 to priority 100

restart keepalived on both nodes
```
systemctl restart keepalived
```
ping the floating IP address
```
ping -c1 192.168.0.100
```
test replication by stopping Keepalived on the master node only and pinging the floating IP address. Finally, restart keepalived
```
systemctl stop keepalived
ping -c1 192.168.0.100
systemctl start keepalived
```

Service replication#

Make sure to be in a trusted network because we allow root login via SSH to simplify operations. In this example we copy files from

Apache
Unbound
dnscrypt-proxy
Certbot (Let’s encrypt)

The enabled_files directory in the master node contains files with lists of files or directories which will be copied by rsync to the backup server.

create the script

/home/jobs/scripts/by-user/root/keepalived/keepalived_deploy.sh#

#!/usr/bin/env bash
#
# keepalived_deploy.sh
#
# Copyright (C) 2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

set -euo pipefail

. "${1}"

SRC='/'
DST='/'
ENABLED_FILES=$(find enabled_files/* -type f)
SYSTEMD_DEPLOY_SERVICES=$(cat systemd_deploy_services.txt)

# Sync files.
for f in ${ENABLED_FILES}; do
    printf "%s\n" "${RSYNC_BASE} --files-from="${f}" "${SRC}" \
        "${USER}"@"${HOST}":"${DST}""
    ${RSYNC_BASE} --files-from="${f}" "${SRC}" "${USER}"@"${HOST}":"${DST}"
done

# Restart systemd services.
ssh "${USER}"@"${HOST}" "\
    systemctl daemon-reload \
    && systemctl reenable --all "${SYSTEMD_DEPLOY_SERVICES}" \
    && systemctl restart --all "${SYSTEMD_DEPLOY_SERVICES}" \
    && systemctl status --all --no-pager "${SYSTEMD_DEPLOY_SERVICES}""

create a configuration file

/home/jobs/scripts/by-user/root/keepalived/keepalived_deploy.conf#

# o      The  --archive  (-a) option's behavior does not imply --recursive
#        (-r), so specify it explicitly, if you want it.
# Use --dry-run to simulate
RSYNC_BASE='rsync -avAX -r --delete'
USER='root'
HOST='192.168.0.11'

create an SSH key. Do not set a password for it

ssh-keygen -t rsa -b 16384 -C "$(whoami)@$(hostname)-$(date +%F)"

Add the following to the SSH configuration

/root/.ssh/config#

Match host 192.168.0.11 user root
  IdentityFile=/root/.ssh/bak_root

go to the backup node and copy the newly created public key in /root/.ssh/authorized_keys

edit the SSH server configuration

/etc/ssh/sshd_config#

# [ ... ]

PermitRootLogin yes
AllowUsers root    # [ ... ]
Match user root
    PasswordAuthentication no

# [ ... ]

restart the SSH service in the backup node
```
systemctl restart ssh
```
go back to the master node and test if the key is working
```
ssh root@192.168.0.11
```

create a Systemd service unit file

/home/jobs/services/by-user/root/keepalived-deploy.service#

[Unit]
Description=Copy files for keepalived
Requires=network-online.target
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/home/jobs/scripts/by-user/root/keepalived
ExecStart=/home/jobs/scripts/by-user/root/keepalived/keepalived_deploy.sh /home/jobs/scripts/by-user/root/keepalived/keepalived_deploy.conf
User=root
Group=root

create a Systemd timer unit file

/home/jobs/services/by-user/root/keepalived-deploy.timer#

[Unit]
Description=Once every day copy files for keepalived

[Timer]
OnCalendar=*-*-* 5:30:00
Persistent=true

[Install]
WantedBy=timers.target

Apache replication#

See also

How to redirect all pages to one page? 5

in your master node, separate replicatable service from non-replicatable ones. You can do this by separating the configuration in multiple files and then including those configuration in the main file (/etc/apache2/apache2.conf).
add these files in the master node.

The first one copies the Apache configuration
/home/jobs/scripts/by-user/root/keepalived/enabled_files/apache2.txt#
```
/etc/apache2/apache2.conf
/etc/apache2/replicated-servers.conf
```
Important

you must change the virtual host directive to use the floating IP like this: <VirtualHost 192.168.0.100:443>

The second file copies the server data. You can replicate static data (Jekyll website, HTML, etc…) but not programs that rely on databases without extra work
/home/jobs/scripts/by-user/root/keepalived/enabled_files/replicated_webservers_data.txt#
```
/var/www/franco.net.eu.org
/var/www/assets.franco.net.eu.org
/var/www/blog.franco.net.eu.org
/var/www/docs.franco.net.eu.org
/var/www/keepachangelog.franco.net.eu.org
```
The third file copies all the HTTPS certificates
/home/jobs/scripts/by-user/root/keepalived/enabled_files/letsencrypt.txt#
```
/etc/letsencrypt
```

in the backup node you must “patch” the non-replicatable service. You can setup an error message for each server like this:

/etc/apache2/standard-servers.conf#

<IfModule mod_ssl.c>
<VirtualHost 192.168.0.100:443>
    UseCanonicalName on
    Keepalive On
    SSLCompression      off
    ServerName software.franco.net.eu.org
    RewriteEngine On

    Include /etc/apache2/standard-servers-outage-text.conf

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/software.franco.net.eu.org/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/software.franco.net.eu.org/privkey.pem
</VirtualHost>
</IfModule>

/etc/apache2/standard-servers-outage-text.conf#

DocumentRoot "/var/www/standard-servers"
<Directory "/var/www/standard-servers">
    Options -ExecCGI -Includes -FollowSymLinks -Indexes
    AllowOverride None
    Require all granted
</Directory>

# Redirect all requests to the root directory of the virtual server.
RewriteEngine On
RewriteRule \/.+ / [L,R]

Create a file in /var/www/standard-servers/index.html with your outage message

DNS replication#

I use dnscrypt-proxy as DNS server and Unbound as caching server. The systemd socket file is useful to set the listening port.

See also

Unbound DNS server behind a VIP - solving reply from unexpected source 6

use these configurations to replicate the two services.
/home/jobs/scripts/by-user/root/keepalived/enabled_files/dnscrypt-proxy.txt#
```
/etc/dnscrypt-proxy/dnscrypt-proxy.toml
/etc/systemd/system/dnscrypt-proxy.socket
```
/home/jobs/scripts/by-user/root/keepalived/enabled_files/unbound.txt#
```
/etc/unbound/unbound.conf
```

Important

Add interface-automatic: yes to the unbound configuration in the server section.

Final steps#

run the deploy script

Kernel#

See also

filesystem - Where does update-initramfs look for kernel versions? - Ask Ubuntu 7

RAID#

Run periodical RAID data scrubs on hard drives and SSDs.

See also

ubuntu - How to wipe md raid meta? - Unix & Linux Stack Exchange 8
RAID data scrubbing 9

install the dependencies

apt-get install mdadm python3-yaml python3-requests

install fpyutils. See reference

create the jobs directories. See reference

mkdir -p /home/jobs/{scripts,services}/by-user/root

create the script

/home/jobs/scripts/by-user/root/mdadm_check.py#

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# Copyright (C) 2014-2017 Neil Brown <neilb@suse.de>
#
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    Author: Neil Brown
#    Email: <neilb@suse.com>
#
# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
r"""Run RAID tests."""

import collections
import multiprocessing
import os
import pathlib
import sys
import time

import fpyutils
import yaml

# Constants.
STATUS_CLEAN = 'clean'
STATUS_ACTIVE = 'active'
STATUS_IDLE = 'idle'


class UserNotRoot(Exception):
    """The user running the script is not root."""


class NoAvailableArrays(Exception):
    """No available arrays."""


class NoSelectedArraysPresent(Exception):
    """None of the arrays in the configuration file exists."""


def get_active_arrays():
    active_arrays = list()
    with open('/proc/mdstat', 'r') as f:
        line = f.readline()
        while line:
            if STATUS_ACTIVE in line:
                active_arrays.append(line.split()[0])
            line = f.readline()

    return active_arrays


def get_array_state(array: str):
    return open('/sys/block/' + array + '/md/array_state', 'r').read().rstrip()


def get_sync_action(array: str):
    return open('/sys/block/' + array + '/md/sync_action', 'r').read().rstrip()


def run_action(array: str, action: str):
    with open('/sys/block/' + array + '/md/sync_action', 'w') as f:
        f.write(action)


def main_action(array: str, config: dict):
    action = devices[array]
    go = True
    while go:
        if get_sync_action(array) == STATUS_IDLE:
            message = 'running ' + action + ' on /dev/' + array + '. pid: ' + str(
                os.getpid())
            run_action(array, action)
            message += '\n\n'
            message += 'finished pid: ' + str(os.getpid())
            print(message)

            if config['notify']['gotify']['enabled']:
                m = config['notify']['gotify']['message'] + ' ' + '\n' + message
                fpyutils.notify.send_gotify_message(
                    config['notify']['gotify']['url'],
                    config['notify']['gotify']['token'], m,
                    config['notify']['gotify']['title'],
                    config['notify']['gotify']['priority'])
            if config['notify']['email']['enabled']:
                fpyutils.notify.send_email(
                    message, config['notify']['email']['smtp_server'],
                    config['notify']['email']['port'],
                    config['notify']['email']['sender'],
                    config['notify']['email']['user'],
                    config['notify']['email']['password'],
                    config['notify']['email']['receiver'],
                    config['notify']['email']['subject'])

            go = False
        if go:
            print('waiting ' + array + ' to be idle...')
            time.sleep(config['generic']['timeout_idle_check'])


if __name__ == '__main__':
    if os.getuid() != 0:
        raise UserNotRoot

    configuration_file = sys.argv[1]
    config = yaml.load(open(configuration_file, 'r'), Loader=yaml.SafeLoader)
    devices = dict()
    for dev_element in config['devices']:
        key = dev_element.keys()
        device = list(key)[0]
        devices[device] = dev_element[device]

    active_arrays = get_active_arrays()
    dev_queue = collections.deque()
    if len(active_arrays) > 0:
        for dev in active_arrays:
            if pathlib.Path('/sys/block/' + dev + '/md/sync_action').is_file():
                state = get_array_state(dev)
                if state == STATUS_CLEAN or state == STATUS_ACTIVE or state == STATUS_IDLE:
                    try:
                        if devices[dev] != 'ignore' and dev in devices:
                            dev_queue.append(dev)
                    except KeyError:
                        pass

    if len(active_arrays) == 0:
        raise NoAvailableArrays
    if len(dev_queue) == 0:
        raise NoSelectedArraysPresent

    while len(dev_queue) > 0:
        for i in range(0, config['generic']['max_concurrent_checks']):
            if len(dev_queue) > 0:
                ready = dev_queue.popleft()
                p = multiprocessing.Process(target=main_action,
                                            args=(
                                                ready,
                                                config,
                                            ))
                p.start()
        p.join()

create a configuration file

/home/jobs/scripts/by-user/root/mdadm_check.yaml#

#
# mdadm_check.yaml
#
# Copyright (C) 2014-2017 Neil Brown <neilb@suse.de>
#
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    Author: Neil Brown
#    Email: <neilb@suse.com>
#
# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)

generic:
    # The maximum number of concurrent processes.
    max_concurrent_checks: 2

    # In seconds.
    timeout_idle_check: 10

# key:      RAID array name without '/dev/'.
# value:    action.
devices:
    md1: 'check'
    md2: 'ignore'
    md3: 'check'

notify:
    email:
        enabled: true
        smtp_server: 'smtp.gmail.com'
        port: 465
        sender: 'myusername@gmail.com'
        user: 'myusername'
        password: 'my awesome password'
        receiver: 'myusername@gmail.com'
        subject: 'mdadm operation'
    gotify:
        enabled: true
        url: '<gotify url>'
        token: '<app token>'
        title: 'mdadm operation'
        message: 'starting mdadm operation'
        priority: 5

Important

do not prepend /dev to RAID device names
possible values: check, repair, idle, ignore
- ignore will make the script skip the device
- use repair at your own risk
absent devices are ignored
run these commands to get the names of RAID arrays
```
lsblk
cat /proc/mdstat
```

create a Systemd service unit file

/home/jobs/services/by-user/root/mdadm-check.service#

[Unit]
Description=mdadm check
Requires=sys-devices-virtual-block-md1.device
Requires=sys-devices-virtual-block-md2.device
Requires=sys-devices-virtual-block-md3.device
After=sys-devices-virtual-block-md1.device
After=sys-devices-virtual-block-md2.device
After=sys-devices-virtual-block-md3.device

[Service]
Type=simple
ExecStart=/home/jobs/scripts/by-user/root/mdadm_check.py /home/jobs/scripts/by-user/root/mdadm_check.yaml
User=root
Group=root

[Install]
WantedBy=multi-user.target

create a Systemd timer unit file

/home/jobs/services/by-user/root/mdadm-check.timer#

[Unit]
Description=Once a month check mdadm arrays

[Timer]
OnCalendar=Monthly
Persistent=true

[Install]
WantedBy=timers.target

fix the permissions

chmod 700 /home/jobs/{scripts,services}/by-user/root

run the deploy script

S.M.A.R.T.#

Run periodical S.M.A.R.T. tests on hard drives and SSDs. The provided script supports only /dev/disk/by-id names.

See also

A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks 10

install the dependencies

apt-get install hdparm smartmontools python3-yaml python3-requests

install fpyutils. See reference
identify the drives you want to check S.M.A.R.T. values
```
ls /dev/disk/by-id
```
See also the udev rule file /lib/udev/rules.d/60-persistent-storage.rules. You can also use this command to have more details of specific drives
```
hdparm -I /dev/disk/by-id/${drive_name}
# or
hdparm -I /dev/sd${letter}
```

create the jobs directories. See reference

mkdir -p /home/jobs/{scripts,services}/by-user/root
chmod 700 -R /home/jobs/{scripts,services}/by-user/root

create the script

/home/jobs/scripts/by-user/root/smartd_test.py#

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# smartd_test.py
#
# Copyright (C) 2019-2021 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
r"""Run S.M.A.R.T tests on hard drives."""

import json
import os
import pathlib
import re
import shlex
import subprocess
import sys

import fpyutils
import yaml


class UserNotRoot(Exception):
    """The user running the script is not root."""


def get_disks() -> list:
    r"""Scan all the disks."""
    disks = list()
    for d in pathlib.Path('/dev/disk/by-id').iterdir():
        # Ignore disks ending with part-${integer} to avoid duplicates (names
        # corresponding to partitions of the same disk).
        disk = str(d)
        if re.match('.+-part[0-9]+$', disk) is None:
            try:
                ddict = json.loads(
                    subprocess.run(
                        shlex.split('smartctl --capabilities --json ' +
                                    shlex.quote(disk)),
                        capture_output=True,
                        check=False,
                        shell=False,
                        timeout=30).stdout)
                try:
                    # Check for smart test support.
                    if ddict['ata_smart_data']['capabilities'][
                            'self_tests_supported']:
                        disks.append(disk)
                except KeyError:
                    pass
            except subprocess.TimeoutExpired:
                print('timeout for ' + disk)
            except subprocess.CalledProcessError:
                print('device ' + disk +
                      ' does not support S.M.A.R.T. commands, skipping...')

    return disks


def disk_ready(disk: str, busy_status: int = 249) -> bool:
    r"""Check if the disk is ready."""
    # Raises a KeyError if disk has not S.M.A.R.T. status capabilities.
    ddict = json.loads(
        subprocess.run(shlex.split('smartctl --capabilities --json ' +
                                   shlex.quote(disk)),
                       capture_output=True,
                       check=True,
                       shell=False,
                       timeout=30).stdout)
    if ddict['ata_smart_data']['self_test']['status']['value'] != busy_status:
        return True
    else:
        return False


def run_test(disk: str, test_length: str = 'long') -> str:
    r"""Run the smartd test."""
    return subprocess.run(
        shlex.split('smartctl --test=' + shlex.quote(test_length) + ' ' +
                    shlex.quote(disk)),
        capture_output=True,
        check=True,
        shell=False,
        timeout=30).stdout


if __name__ == '__main__':
    if os.getuid() != 0:
        raise UserNotRoot

    configuration_file = shlex.quote(sys.argv[1])
    config = yaml.load(open(configuration_file, 'r'), Loader=yaml.SafeLoader)

    # Do not prepend '/dev/disk/by-id/'.
    disks_to_check = shlex.quote(sys.argv[2])
    disks_available = get_disks()

    for d in config['devices']:
        dev = '/dev/disk/by-id/' + d
        if config['devices'][d]['enabled'] and dev in disks_available:
            if disks_to_check == 'all' or disks_to_check == d:
                if disk_ready(dev, config['devices'][d]['busy_status']):
                    print('attempting ' + d + ' ...')
                    message = run_test(
                        dev, config['devices'][d]['test']).decode('utf-8')
                    print(message)
                    if config['devices'][d]['log']:
                        if config['notify']['gotify']['enabled']:
                            m = config['notify']['gotify'][
                                'message'] + ' ' + d + '\n' + message
                            fpyutils.notify.send_gotify_message(
                                config['notify']['gotify']['url'],
                                config['notify']['gotify']['token'], m,
                                config['notify']['gotify']['title'],
                                config['notify']['gotify']['priority'])
                        if config['notify']['email']['enabled']:
                            fpyutils.notify.send_email(
                                message,
                                config['notify']['email']['smtp_server'],
                                config['notify']['email']['port'],
                                config['notify']['email']['sender'],
                                config['notify']['email']['user'],
                                config['notify']['email']['password'],
                                config['notify']['email']['receiver'],
                                config['notify']['email']['subject'])
                else:
                    # Drop test requests if a disk is running a test in a particular moment.
                    # This avoid putting the disks under too much stress.
                    print('disk ' + d + ' not ready, checking the next...')

create a configuration file

includes/home/jobs/scripts/by-user/root/smartd_test.yaml#

#
# smartd_test.yaml
#
# Copyright (C) 2019-2020 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

devices:
    ata-disk1:
        enabled: true
        test: 'long'
        log: true
        busy_status: 249
    ata-disk2:
        enabled: true
        test: 'long'
        log: false
        busy_status: 249
    ata-diskn:
        enabled: true
        test: 'long'
        log: true
        busy_status: 249

notify:
    gotify:
        enabled: true
        url: '<gotify url>'
        token: '<app token>'
        title: 'smart test'
        message: 'starting smart test on'
        priority: 5
    email:
        enabled: true
        smtp_server: 'smtp.gmail.com'
        port: 465
        sender: 'myusername@gmail.com'
        user: 'myusername'
        password: 'my awesome password'
        receiver: 'myusername@gmail.com'
        subject: 'smartd test'

Important

absent devices are ignored
devices must be explicitly enabled
do not prepend /dev/disk/by-id/ to drive names
run a short test to get the busy_status value.
```
smartctl -t short /dev/disk/by-id/${drive_name}
```
You should be able to capture the value while the test is running by looking at the Self-test execution status: line. In my case it is always 249, but this value is not hardcoded in smartmontools’ source code
```
smartctl --all /dev/disk/by-id/${drive_name}
```

use this Systemd service unit file

/home/jobs/services/by-user/root/smartd-test.ata_disk1.service#

[Unit]
Description=execute smartd on ata-disk1

[Service]
Type=simple
ExecStart=/home/jobs/scripts/by-user/root/smartd_test.py /home/jobs/scripts/by-user/root/smartd_test.yaml ata-disk1
User=root
Group=root

use this Systemd timer unit file

/home/jobs/services/by-user/root/smartd-test.ata_disk1.timer#

[Unit]
Description=Once every two months smart test ata-disk1

[Timer]
OnCalendar=*-01,03,05,07,09,11-01 00:00:00
Persistent=true

[Install]
WantedBy=timers.target

fix the permissions

chmod 700 -R /home/jobs/scripts/by-user/smartd_test.*
chmod 700 -R /home/jobs/services/by-user/root

run the deploy script

Important

To avoid tests being interrupted you must avoid putting the disks to sleep, therefore, programs like hd-idle must be stopped before running the tests.

Services#

Notify unit status#

This script is useful to notfiy about failed Systemd service.

Some time ago my Gitea instance could not start after an update. If I used this script I would have known immediately about the problem instead of several days later.

See also

linux - get notification when systemd-monitored service enters failed state - Server Fault 11

install fpyutils. See reference

create the script

/home/jobs/scripts/by-user/root/notify_unit_status.py#

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# notify_unit_status.py
#
# Copyright (C) 2015 Pablo Martinez @ Stack Exchange (https://serverfault.com/a/701100)
# Copyright (C) 2018 Davy Landman @ Stack Exchange (https://serverfault.com/a/701100)
# Copyright (C) 2020-2021 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This script is licensed under a
# Creative Commons Attribution-ShareAlike 4.0 International License.
#
# You should have received a copy of the license along with this
# work. If not, see <http://creativecommons.org/licenses/by-sa/4.0/>.
r"""Send a notification when a Systemd unit fails."""

import shlex
import sys

import fpyutils
import yaml

if __name__ == '__main__':
    configuration_file = shlex.quote(sys.argv[1])
    config = yaml.load(open(configuration_file, 'r'), Loader=yaml.SafeLoader)

    failed_unit = shlex.quote(sys.argv[2])

    message = 'service failure: ' + failed_unit
    if config['notify']['gotify']['enabled']:
        m = config['notify']['gotify']['message'] + '\n' + message
        fpyutils.notify.send_gotify_message(
            config['notify']['gotify']['url'],
            config['notify']['gotify']['token'], m,
            config['notify']['gotify']['title'],
            config['notify']['gotify']['priority'])
    if config['notify']['email']['enabled']:
        fpyutils.notify.send_email(message,
                                   config['notify']['email']['smtp server'],
                                   config['notify']['email']['port'],
                                   config['notify']['email']['sender'],
                                   config['notify']['email']['user'],
                                   config['notify']['email']['password'],
                                   config['notify']['email']['receiver'],
                                   config['notify']['email']['subject'])

create a configuration file

includes/home/jobs/scripts/by-user/root/notify_unit_status.yaml#

#
# notify_unit_status.yaml
#
# Copyright (C) 2015 Pablo Martinez @ Stack Exchange (https://serverfault.com/a/701100)
# Copyright (C) 2018 Davy Landman @ Stack Exchange (https://serverfault.com/a/701100)
# Copyright (C) 2020 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This script is licensed under a
# Creative Commons Attribution-ShareAlike 4.0 International License.
#
# You should have received a copy of the license along with this
# work. If not, see <http://creativecommons.org/licenses/by-sa/4.0/>.

notify:
    gotify:
        enabled: true
        url: '<gotify url>'
        token: '<app token>'
        title: 'service failure'
        message: 'service failure'
        priority: 5
    email:
        enabled: true
        smtp server: 'smtp.gmail.com'
        port: 465
        sender: 'myusername@gmail.com'
        user: 'myusername'
        password: 'my awesome password'
        receiver: 'myusername@gmail.com'
        subject: 'service failure'

use this Systemd service unit file

includes/home/jobs/services/by-user/root/notify-unit-status@.service#

#
# notify-unit-status@.service
#
# Copyright (C) 2015 Pablo Martinez @ Stack Exchange (https://serverfault.com/a/701100)
# Copyright (C) 2018 Davy Landman @ Stack Exchange (https://serverfault.com/a/701100)
# Copyright (C) 2020 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This script is licensed under a
# Creative Commons Attribution-ShareAlike 4.0 International License.
#
# You should have received a copy of the license along with this
# work. If not, see <http://creativecommons.org/licenses/by-sa/4.0/>.

[Unit]
Description=Unit Status Mailer Service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/home/jobs/scripts/by-user/root/notify_unit_status.py /home/jobs/scripts/by-user/root/notify_unit_status.yaml %I

edit the Systemd service you want to monitor. In this example the service to be monitored is Gitea
```
systemctl edit gitea.service
```

add this content

# [ ... ]

[Unit]

# [ ... ]

OnFailure=notify-unit-status@%n.service

# [ ... ]

Updates#

Update action#

This script can be used to update software not supported by the package manager, for example Docker images.

Important

Any arbitrary command can be configured.

See also

A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks 10

install the dependencies

apt-get install python3-yaml python3-requests

install fpyutils. See reference

create the script

/home/jobs/scripts/by-user/root/update_action.py#

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# update_action.py
#
# Copyright (C) 2021-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
r"""update_action.py."""

import shlex
import sys

import fpyutils
import yaml


def send_notification(message: str, notify: dict):
    m = notify['gotify']['message'] + '\n' + message
    if notify['gotify']['enabled']:
        fpyutils.notify.send_gotify_message(notify['gotify']['url'],
                                            notify['gotify']['token'], m,
                                            notify['gotify']['title'],
                                            notify['gotify']['priority'])
    if notify['email']['enabled']:
        fpyutils.notify.send_email(
            message, notify['email']['smtp_server'], notify['email']['port'],
            notify['email']['sender'], notify['email']['user'],
            notify['email']['password'], notify['email']['receiver'],
            notify['email']['subject'])


if __name__ == '__main__':

    def main():
        configuration_file = shlex.quote(sys.argv[1])
        config = yaml.load(open(configuration_file, 'r'),
                           Loader=yaml.SafeLoader)

        # Action types. Preserve this order.
        types = ['pre', 'update', 'post']
        services = config['services']

        for service in services:
            for type in types:
                for cmd in services[service]['commands'][type]:
                    for name in cmd:
                        retval = fpyutils.shell.execute_command_live_output(
                            cmd[name]['command'], dry_run=False)
                        if cmd[name]['notify']['success'] and retval == cmd[
                                name]['expected_retval']:
                            send_notification(
                                'command "' + name + '" of service "' +
                                service + '": OK', config['notify'])
                        elif cmd[name]['notify']['error'] and retval != cmd[
                                name]['expected_retval']:
                            send_notification(
                                'command "' + name + '" of service "' +
                                service + '": ERROR', config['notify'])

    main()

create a configuration file

includes/home/jobs/scripts/by-user/root/update_action.mypurpose.yaml#

#
# update_action.mypurpose.yaml
#
# Copyright (C) 2021-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

notify:
    email:
        enabled: true
        smtp_server: 'smtp.gmail.com'
        port: 465
        sender: 'myusername@gmail.com'
        user: 'myusername'
        password: 'my awesome password'
        receiver: 'myusername@gmail.com'
        subject: 'update action'
    gotify:
        enabled: true
        url: '<gotify url>'
        token: '<app token>'
        title: 'update action'
        message: 'update action'
        priority: 5

services:
    hello:
        commands:
            pre:
                - stop_service:
                    # string
                    command: 'systemctl stop docker-compose.hello.service'
                    # integer
                    expected_retval: 0
                    # boolean: {true,false}
                    notify:
                        success: true
                        error: true
            update:
                - pull:
                    command: 'pushd /home/jobs/scripts/by-user/root/docker/hello && docker-compose pull'
                    expected_retval: 0
                    notify:
                        success: true
                        error: true
                - build:
                    command: 'pushd /home/jobs/scripts/by-user/root/docker/hello && docker-compose build --pull'
                    expected_retval: 0
                    notify:
                        success: true
                        error: true
            post:
                - start_service:
                    command: 'systemctl start docker-compose.hello.service'
                    expected_retval: 0
                    notify:
                        success: true
                        error: true
    goodbye:
        commands:
            pre:
                - stop_service:
                    command: 'systemctl stop docker-compose.goodbye.service'
                    expected_retval: 0
                    notify:
                        success: true
                        error: true
            update:
                - pull_only:
                    command: 'pushd /home/jobs/scripts/by-user/root/docker/goodbye && docker-compose pull'
                    expected_retval: 0
                    notify:
                        success: true
                        error: true
            post:
                - start_service:
                    command: 'systemctl start docker-compose.goodbye.service'
                    expected_retval: 0
                    notify:
                        success: true
                        error: true

use this Systemd service unit file

/home/jobs/services/by-user/root/update-action.mypurpose.service#

[Unit]
Description=Update action mypurpose
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/home/jobs/scripts/by-user/root/update_action.py /home/jobs/scripts/by-user/root/update_action.mypurpose.yaml
User=root
Group=root

use this Systemd timer unit file

/home/jobs/services/by-user/root/update-action.mypurpose.timer#

[Unit]
Description=Update action mypurpose monthly

[Timer]
OnCalendar=monthly
Persistent=true

[Install]
WantedBy=timers.target

fix the permissions

chmod 700 -R /home/jobs/scripts/by-user/update_action.*
chmod 700 -R /home/jobs/services/by-user/root

run the deploy script

Footnotes

1: https://database.guide/what-is-collation-in-databases/ unknown license
2: https://www.keepalived.org/index.html unknown license
3: https://github.com/osixia/docker-keepalived/issues/34 unknown license
4: https://docs.syseleven.de/syseleven-stack/en/howtos/l3-high-availability unknown license
5: https://forums.digitalpoint.com/threads/how-to-redirect-all-pages-to-one-page.25353/ unknown license
6: https://www.claudiokuenzler.com/blog/695/unbound-behind-a-virtual-ip-vip-reply-from-unexpected-source unknown license
7: https://askubuntu.com/questions/759802/where-does-update-initramfs-look-for-kernel-versions CC BY-SA 3.0, copyright (c) 2016-2017, askubuntu contributors
8: https://unix.stackexchange.com/questions/411206/how-to-wipe-md-raid-meta CC BY-SA 3.0, copyright (c) 2017, stackexchange contributors
9: https://blog.franco.net.eu.org/notes/raid-data-scrubbing.html CC-BY-SA 4.0, copyright (c) 2019-2021, Franco Masotti
10(1,2): https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti
11: https://serverfault.com/questions/694818/get-notification-when-systemd-monitored-service-enters-failed-state CC BY-SA 3.0, copyright (c) 2015-2018, serverfault contributors

ftutorials documentation

Maintenance

Contents

Maintenance#

Software#

Database#

PostgreSQL#

Changing collate#

Moving data directory#

High availability#

Key#

Basic setup#

Service replication#

Apache replication#

DNS replication#

Final steps#

Kernel#

RAID#

S.M.A.R.T.#

Services#

Notify unit status#

Updates#

Update action#