Maintenance#

Software#

Database#

PostgreSQL#

Important

All these instructions have been tested with PostgreSQL version 13 only!

Changing collate#

Sometimes you might create a new database without specifying the collate information. PosgreSQL will use the default collate setting and there is no way to change it once the database is created. The only solution is to dump the database, to create a new one with the correct collate and finally to import the dump and drop the original database.

See also

  • What is Collation in Databases? [1]

  1. backup old database

    pg_dump -U myuser olddb > olddb.bak.sql
    
  2. create new database with correct collate

    CREATE DATABASE newdb WITH OWNER myuser TEMPLATE template0 ENCODING UTF8 LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8';
    
  3. import dump into new database

    psql -U myuser -d newdb < olddb.bak.sql
    
  4. rename old database

    ALTER DATABASE olddb RENAME TO olddb_bak;
    
  5. rename the new database to the original database name

    ALTER DATABASE newdb RENAME TO olddb;
    
  6. restart services and check that everything works

  7. drop old database

    DROP DATABASE olddb_bak;
    
Moving data directory#

If the data directory in the root partition is getting too large you can create a new partition (mounted on /postgresql in this example) and let PostgreSQL point to that one instead.

  1. install the dependencies

    apt-get install rsync
    
  2. stop PostgreSQL

    systemctl stop postgresql
    
  3. copy the database directory

    rsync -avAX /var/lib/postgresql/13/main /postgresql/13
    
  4. change the data_directory setting in /etc/postgresql/postgresql.conf

    data_directory = '/postgresql/13/main'      # use data in another directory
    
  5. restart PostgreSQL

    systemctl start postgresql
    

High availability#

In this example we configure two nodes. When the master node goes offline the backup node takes over the floating IP address. We replicate services on the backup node such as Apache and Unbound.

Keepalived is a tool which handles network replication for layers 3 and 4. Here you can find the configuration file for the master node (the backup node just needs minor edits).

A script you find below helps you copy the content of webservers, DNS server, etc on the backup node as well as restarting those services automatically.

See also

  • Keepalived for Linux [2]

  • Is it possible to add a static mac address for a vrrp ip? · Issue #34 · osixia/docker-keepalived · GitHub [3]

  • Building Layer 3 High Availability | Documentation [4]

Key#

Node

IP

Hostname

Network interface name

MASTER

192.168.0.10

mst

eno1

BACKUP

192.168.0.11

bak

eno1

floating IP

192.168.0.100

-

-

Basic setup#

  1. install the dependencies. Keepalived must be installed on all nodes

    apt-get install keepalived rsync
    
  2. create the configuration for the master node

    /etc/keepalived/keepalived.conf#
     1global_defs {
     2    max_auto_priority -1
     3}
     4
     5########################
     6## VRRP configuration ##
     7########################
     8
     9# Identify the VRRP instance as, in this case, "failover_link".
    10vrrp_instance failover_link {
    11
    12    # Initial state of the keepalived VRRP instance on this host
    13    # (MASTER or BACKUP). Once started, only priority matters.
    14    state MASTER
    15
    16    # interface this VRRP instance is bound to.
    17    interface eno1
    18
    19    # Arbitrary value between 1 and 255 to distinguish this VRRP
    20    # instance from others running on the same device. It must match
    21    # with other peering devices.
    22    virtual_router_id 1
    23
    24    # Highest priority value takes the MASTER role and the
    25    # virtual IP (default value is 100).
    26    priority 110
    27
    28    # Time, in seconds, between VRRP advertisements. The default is 1,
    29    # but in some cases you can achieve more reliable results by
    30    # increasing this value.
    31    advert_int 2
    32
    33    use_vmac
    34    vmac_xmit_base
    35
    36    # Authentication method: AH indicates ipsec Authentication Header.
    37    # It offers more security than PASS, which transmits the
    38    # authentication password in plaintext. Some implementations
    39    # have complained of problems with AH, so it may be necessary
    40    # to use PASS to get keepalived's VRRP working.
    41    #
    42    # The auth_pass will only use the first 8 characters entered.
    43    authentication {
    44        auth_type AH
    45        auth_pass f5K.*0Bq
    46    }
    47
    48    # VRRP advertisements ordinarily go out over multicast. This
    49    # configuration paramter causes keepalived to send them
    50    # as unicasts. This specification can be useful in environments
    51    # where multicast isn't supported or in instances where you want
    52    # to limit which devices see your VRRP announcements. The IP
    53    # address(es) can be IPv4 or IPv6, and indicate the real IP of
    54    # other members.
    55    unicast_peer {
    56        192.168.0.11
    57    }
    58    # Virtual IP address(es) that will be shared among VRRP
    59    # members. "Dev" indicates the interface the virtual IP will
    60    # be assigned to. And "label" allows for clearer description of the
    61    # virtual IP.
    62    virtual_ipaddress {
    63        192.168.0.100 dev eno1 label eno1:vip
    64    }
    65}
    

    Note

    Copy this file in the backup node as well and change:

    • state MASTER to state BACKUP

    • unicast_peer { 192.168.0.11 } to unicast_peer { 192.168.0.10 }

    • priority 110 to priority 100

  3. restart keepalived on both nodes

    systemctl restart keepalived
    
  4. ping the floating IP address

    ping -c1 192.168.0.100
    
  5. test replication by stopping Keepalived on the master node only and pinging the floating IP address. Finally, restart keepalived

    systemctl stop keepalived
    ping -c1 192.168.0.100
    systemctl start keepalived
    

Service replication#

Make sure to be in a trusted network because we allow root login via SSH to simplify operations. In this example we copy files from

  • Apache

  • Unbound

  • dnscrypt-proxy

  • Certbot (Let’s encrypt)

The enabled_files directory in the master node contains files with lists of files or directories which will be copied by rsync to the backup server.

  1. create the script

    /home/jobs/scripts/by-user/root/keepalived/keepalived_deploy.sh#
     1#!/usr/bin/env bash
     2#
     3# keepalived_deploy.sh
     4#
     5# Copyright (C) 2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     6#
     7# This program is free software: you can redistribute it and/or modify
     8# it under the terms of the GNU General Public License as published by
     9# the Free Software Foundation, either version 3 of the License, or
    10# (at your option) any later version.
    11#
    12# This program is distributed in the hope that it will be useful,
    13# but WITHOUT ANY WARRANTY; without even the implied warranty of
    14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    15# GNU General Public License for more details.
    16#
    17# You should have received a copy of the GNU General Public License
    18# along with this program.  If not, see <http://www.gnu.org/licenses/>.
    19
    20set -euo pipefail
    21
    22. "${1}"
    23
    24SRC='/'
    25DST='/'
    26ENABLED_FILES=$(find enabled_files/* -type f)
    27SYSTEMD_DEPLOY_SERVICES=$(cat systemd_deploy_services.txt)
    28
    29# Sync files.
    30for f in ${ENABLED_FILES}; do
    31    printf "%s\n" "${RSYNC_BASE} --files-from="${f}" "${SRC}" \
    32        "${USER}"@"${HOST}":"${DST}""
    33    ${RSYNC_BASE} --files-from="${f}" "${SRC}" "${USER}"@"${HOST}":"${DST}"
    34done
    35
    36# Restart systemd services.
    37ssh "${USER}"@"${HOST}" "\
    38    systemctl daemon-reload \
    39    && systemctl reenable --all "${SYSTEMD_DEPLOY_SERVICES}" \
    40    && systemctl restart --all "${SYSTEMD_DEPLOY_SERVICES}" \
    41    && systemctl status --all --no-pager "${SYSTEMD_DEPLOY_SERVICES}""
    
  2. create a configuration file

    /home/jobs/scripts/by-user/root/keepalived/keepalived_deploy.conf#
    1# o      The  --archive  (-a) option's behavior does not imply --recursive
    2#        (-r), so specify it explicitly, if you want it.
    3# Use --dry-run to simulate
    4RSYNC_BASE='rsync -avAX -r --delete'
    5USER='root'
    6HOST='192.168.0.11'
    
  3. create an SSH key. Do not set a password for it

    ssh-keygen -t rsa -b 16384 -C "$(whoami)@$(hostname)-$(date +%F)"
    
  4. Add the following to the SSH configuration

    /root/.ssh/config#
    1Match host 192.168.0.11 user root
    2  IdentityFile=/root/.ssh/bak_root
    
  5. go to the backup node and copy the newly created public key in /root/.ssh/authorized_keys

  6. edit the SSH server configuration

    /etc/ssh/sshd_config#
    1# [ ... ]
    2
    3PermitRootLogin yes
    4AllowUsers root    # [ ... ]
    5Match user root
    6    PasswordAuthentication no
    7
    8# [ ... ]
    
  7. restart the SSH service in the backup node

    systemctl restart ssh
    
  8. go back to the master node and test if the key is working

    ssh root@192.168.0.11
    
  9. create a Systemd service unit file

    /home/jobs/services/by-user/root/keepalived-deploy.service#
     1[Unit]
     2Description=Copy files for keepalived
     3Requires=network-online.target
     4After=network-online.target
     5
     6[Service]
     7Type=simple
     8WorkingDirectory=/home/jobs/scripts/by-user/root/keepalived
     9ExecStart=/home/jobs/scripts/by-user/root/keepalived/keepalived_deploy.sh /home/jobs/scripts/by-user/root/keepalived/keepalived_deploy.conf
    10User=root
    11Group=root
    
  10. create a Systemd timer unit file

    /home/jobs/services/by-user/root/keepalived-deploy.timer#
    1[Unit]
    2Description=Once every day copy files for keepalived
    3
    4[Timer]
    5OnCalendar=*-*-* 5:30:00
    6Persistent=true
    7
    8[Install]
    9WantedBy=timers.target
    

Apache replication#

See also

  • How to redirect all pages to one page? [5]

  1. in your master node, separate replicatable service from non-replicatable ones. You can do this by separating the configuration in multiple files and then including those configuration in the main file (/etc/apache2/apache2.conf).

  2. add these files in the master node.

    The first one copies the Apache configuration

    /home/jobs/scripts/by-user/root/keepalived/enabled_files/apache2.txt#
    /etc/apache2/apache2.conf
    /etc/apache2/replicated-servers.conf
    

    Important

    you must change the virtual host directive to use the floating IP like this: <VirtualHost 192.168.0.100:443>

    The second file copies the server data. You can replicate static data (Jekyll website, HTML, etc…) but not programs that rely on databases without extra work

    /home/jobs/scripts/by-user/root/keepalived/enabled_files/replicated_webservers_data.txt#
    /var/www/franco.net.eu.org
    /var/www/assets.franco.net.eu.org
    /var/www/blog.franco.net.eu.org
    /var/www/docs.franco.net.eu.org
    /var/www/keepachangelog.franco.net.eu.org
    

    The third file copies all the HTTPS certificates

    /home/jobs/scripts/by-user/root/keepalived/enabled_files/letsencrypt.txt#
    /etc/letsencrypt
    
  3. in the backup node you must “patch” the non-replicatable service. You can setup an error message for each server like this:

    /etc/apache2/standard-servers.conf#
     1<IfModule mod_ssl.c>
     2<VirtualHost 192.168.0.100:443>
     3    UseCanonicalName on
     4    Keepalive On
     5    SSLCompression      off
     6    ServerName software.franco.net.eu.org
     7    RewriteEngine On
     8
     9    Include /etc/apache2/standard-servers-outage-text.conf
    10
    11    Include /etc/letsencrypt/options-ssl-apache.conf
    12    SSLCertificateFile /etc/letsencrypt/live/software.franco.net.eu.org/fullchain.pem
    13    SSLCertificateKeyFile /etc/letsencrypt/live/software.franco.net.eu.org/privkey.pem
    14</VirtualHost>
    15</IfModule>
    
    /etc/apache2/standard-servers-outage-text.conf#
     1DocumentRoot "/var/www/standard-servers"
     2<Directory "/var/www/standard-servers">
     3    Options -ExecCGI -Includes -FollowSymLinks -Indexes
     4    AllowOverride None
     5    Require all granted
     6</Directory>
     7
     8# Redirect all requests to the root directory of the virtual server.
     9RewriteEngine On
    10RewriteRule \/.+ / [L,R]
    

    Create a file in /var/www/standard-servers/index.html with your outage message

DNS replication#

I use dnscrypt-proxy as DNS server and Unbound as caching server. The systemd socket file is useful to set the listening port.

See also

  • Unbound DNS server behind a VIP - solving reply from unexpected source [6]

  1. use these configurations to replicate the two services.

    /home/jobs/scripts/by-user/root/keepalived/enabled_files/dnscrypt-proxy.txt#
    /etc/dnscrypt-proxy/dnscrypt-proxy.toml
    /etc/systemd/system/dnscrypt-proxy.socket
    
    /home/jobs/scripts/by-user/root/keepalived/enabled_files/unbound.txt#
    /etc/unbound/unbound.conf
    

Important

Add interface-automatic: yes to the unbound configuration in the server section.

Final steps#

  1. run the deploy script

Kernel#

See also

  • filesystem - Where does update-initramfs look for kernel versions? - Ask Ubuntu [7]

RAID#

Run periodical RAID data scrubs on hard drives and SSDs.

See also

  • ubuntu - How to wipe md raid meta? - Unix & Linux Stack Exchange [8]

  • RAID data scrubbing [9]

  1. install the dependencies

    apt-get install mdadm python3-yaml python3-requests
    
  2. install fpyutils. See reference

  3. create the jobs directories. See reference

    mkdir -p /home/jobs/{scripts,services}/by-user/root
    
  4. create the script

    /home/jobs/scripts/by-user/root/mdadm_check.py#
      1#!/usr/bin/env python3
      2#
      3# Copyright (C) 2014-2017 Neil Brown <neilb@suse.de>
      4#
      5#
      6#    This program is free software; you can redistribute it and/or modify
      7#    it under the terms of the GNU General Public License as published by
      8#    the Free Software Foundation; either version 2 of the License, or
      9#    (at your option) any later version.
     10#
     11#    This program is distributed in the hope that it will be useful,
     12#    but WITHOUT ANY WARRANTY; without even the implied warranty of
     13#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     14#    GNU General Public License for more details.
     15#
     16#    Author: Neil Brown
     17#    Email: <neilb@suse.com>
     18#
     19# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     20r"""Run RAID tests."""
     21
     22import collections
     23import multiprocessing
     24import os
     25import pathlib
     26import sys
     27import time
     28
     29import fpyutils
     30import yaml
     31
     32# Constants.
     33STATUS_CLEAN = 'clean'
     34STATUS_ACTIVE = 'active'
     35STATUS_IDLE = 'idle'
     36
     37
     38class UserNotRoot(Exception):
     39    """The user running the script is not root."""
     40
     41
     42class NoAvailableArrays(Exception):
     43    """No available arrays."""
     44
     45
     46class NoSelectedArraysPresent(Exception):
     47    """None of the arrays in the configuration file exists."""
     48
     49
     50def get_active_arrays():
     51    active_arrays = list()
     52    with open('/proc/mdstat') as f:
     53        line = f.readline()
     54        while line:
     55            if STATUS_ACTIVE in line:
     56                active_arrays.append(line.split()[0])
     57            line = f.readline()
     58
     59    return active_arrays
     60
     61
     62def get_array_state(array: str):
     63    return open('/sys/block/' + array + '/md/array_state').read().rstrip()
     64
     65
     66def get_sync_action(array: str):
     67    return open('/sys/block/' + array + '/md/sync_action').read().rstrip()
     68
     69
     70def run_action(array: str, action: str):
     71    with open('/sys/block/' + array + '/md/sync_action', 'w') as f:
     72        f.write(action)
     73
     74
     75def main_action(array: str, config: dict):
     76    action = devices[array]
     77    go = True
     78    while go:
     79        if get_sync_action(array) == STATUS_IDLE:
     80            message = 'running ' + action + ' on /dev/' + array + '. pid: ' + str(
     81                os.getpid())
     82            run_action(array, action)
     83            message += '\n\n'
     84            message += 'finished pid: ' + str(os.getpid())
     85            print(message)
     86
     87            if config['notify']['gotify']['enabled']:
     88                m = config['notify']['gotify']['message'] + ' ' + '\n' + message
     89                fpyutils.notify.send_gotify_message(
     90                    config['notify']['gotify']['url'],
     91                    config['notify']['gotify']['token'], m,
     92                    config['notify']['gotify']['title'],
     93                    config['notify']['gotify']['priority'])
     94            if config['notify']['email']['enabled']:
     95                fpyutils.notify.send_email(
     96                    message, config['notify']['email']['smtp_server'],
     97                    config['notify']['email']['port'],
     98                    config['notify']['email']['sender'],
     99                    config['notify']['email']['user'],
    100                    config['notify']['email']['password'],
    101                    config['notify']['email']['receiver'],
    102                    config['notify']['email']['subject'])
    103
    104            go = False
    105        if go:
    106            print('waiting ' + array + ' to be idle...')
    107            time.sleep(config['generic']['timeout_idle_check'])
    108
    109
    110if __name__ == '__main__':
    111    if os.getuid() != 0:
    112        raise UserNotRoot
    113
    114    configuration_file = sys.argv[1]
    115    config = yaml.load(open(configuration_file), Loader=yaml.SafeLoader)
    116    devices = dict()
    117    for dev_element in config['devices']:
    118        key = dev_element.keys()
    119        device = list(key)[0]
    120        devices[device] = dev_element[device]
    121
    122    active_arrays = get_active_arrays()
    123    dev_queue = collections.deque()
    124    if len(active_arrays) > 0:
    125        for dev in active_arrays:
    126            if pathlib.Path('/sys/block/' + dev + '/md/sync_action').is_file():
    127                state = get_array_state(dev)
    128                if state == STATUS_CLEAN or state == STATUS_ACTIVE or state == STATUS_IDLE:
    129                    try:
    130                        if devices[dev] != 'ignore' and dev in devices:
    131                            dev_queue.append(dev)
    132                    except KeyError:
    133                        pass
    134
    135    if len(active_arrays) == 0:
    136        raise NoAvailableArrays
    137    if len(dev_queue) == 0:
    138        raise NoSelectedArraysPresent
    139
    140    while len(dev_queue) > 0:
    141        for i in range(0, config['generic']['max_concurrent_checks']):
    142            if len(dev_queue) > 0:
    143                ready = dev_queue.popleft()
    144                p = multiprocessing.Process(target=main_action,
    145                                            args=(
    146                                                ready,
    147                                                config,
    148                                            ))
    149                p.start()
    150        p.join()
    
  5. create a configuration file

    /home/jobs/scripts/by-user/root/mdadm_check.yaml#
     1#
     2# mdadm_check.yaml
     3#
     4# Copyright (C) 2014-2017 Neil Brown <neilb@suse.de>
     5#
     6#
     7#    This program is free software; you can redistribute it and/or modify
     8#    it under the terms of the GNU General Public License as published by
     9#    the Free Software Foundation; either version 2 of the License, or
    10#    (at your option) any later version.
    11#
    12#    This program is distributed in the hope that it will be useful,
    13#    but WITHOUT ANY WARRANTY; without even the implied warranty of
    14#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    15#    GNU General Public License for more details.
    16#
    17#    Author: Neil Brown
    18#    Email: <neilb@suse.com>
    19#
    20# Copyright (C) 2019-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
    21
    22generic:
    23    # The maximum number of concurrent processes.
    24    max_concurrent_checks: 2
    25
    26    # In seconds.
    27    timeout_idle_check: 10
    28
    29# key:      RAID array name without '/dev/'.
    30# value:    action.
    31devices:
    32    md1: 'check'
    33    md2: 'ignore'
    34    md3: 'check'
    35
    36notify:
    37    email:
    38        enabled: true
    39        smtp_server: 'smtp.gmail.com'
    40        port: 465
    41        sender: 'myusername@gmail.com'
    42        user: 'myusername'
    43        password: 'my awesome password'
    44        receiver: 'myusername@gmail.com'
    45        subject: 'mdadm operation'
    46    gotify:
    47        enabled: true
    48        url: '<gotify url>'
    49        token: '<app token>'
    50        title: 'mdadm operation'
    51        message: 'starting mdadm operation'
    52        priority: 5
    

    Important

    • do not prepend /dev to RAID device names

    • possible values: check, repair, idle, ignore

      • ignore will make the script skip the device

      • use repair at your own risk

    • absent devices are ignored

    • run these commands to get the names of RAID arrays

      lsblk
      cat /proc/mdstat
      
  6. create a Systemd service unit file

    /home/jobs/services/by-user/root/mdadm-check.service#
     1[Unit]
     2Description=mdadm check
     3Requires=sys-devices-virtual-block-md1.device
     4Requires=sys-devices-virtual-block-md2.device
     5Requires=sys-devices-virtual-block-md3.device
     6After=sys-devices-virtual-block-md1.device
     7After=sys-devices-virtual-block-md2.device
     8After=sys-devices-virtual-block-md3.device
     9
    10[Service]
    11Type=simple
    12ExecStart=/home/jobs/scripts/by-user/root/mdadm_check.py /home/jobs/scripts/by-user/root/mdadm_check.yaml
    13User=root
    14Group=root
    15
    16[Install]
    17WantedBy=multi-user.target
    
  7. create a Systemd timer unit file

    /home/jobs/services/by-user/root/mdadm-check.timer#
    1[Unit]
    2Description=Once a month check mdadm arrays
    3
    4[Timer]
    5OnCalendar=Monthly
    6Persistent=true
    7
    8[Install]
    9WantedBy=timers.target
    
  8. fix the permissions

    chmod 700 /home/jobs/{scripts,services}/by-user/root
    
  9. run the deploy script

S.M.A.R.T.#

Run periodical S.M.A.R.T. tests on hard drives and SSDs. The provided script supports only /dev/disk/by-id names.

See also

  • A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks [10]

  1. install the dependencies

    apt-get install hdparm smartmontools python3-yaml python3-requests
    
  2. install fpyutils. See reference

  3. identify the drives you want to check S.M.A.R.T. values

    ls /dev/disk/by-id
    

    See also the udev rule file /lib/udev/rules.d/60-persistent-storage.rules. You can also use this command to have more details of specific drives

    hdparm -I /dev/disk/by-id/${drive_name}
    # or
    hdparm -I /dev/sd${letter}
    
  4. create the jobs directories. See reference

    mkdir -p /home/jobs/{scripts,services}/by-user/root
    chmod 700 -R /home/jobs/{scripts,services}/by-user/root
    
  5. create the script

    /home/jobs/scripts/by-user/root/smartd_test.py#
      1#!/usr/bin/env python3
      2#
      3# smartd_test.py
      4#
      5# Copyright (C) 2019-2021 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
      6#
      7# This program is free software: you can redistribute it and/or modify
      8# it under the terms of the GNU General Public License as published by
      9# the Free Software Foundation, either version 3 of the License, or
     10# (at your option) any later version.
     11#
     12# This program is distributed in the hope that it will be useful,
     13# but WITHOUT ANY WARRANTY; without even the implied warranty of
     14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     15# GNU General Public License for more details.
     16#
     17# You should have received a copy of the GNU General Public License
     18# along with this program.  If not, see <http://www.gnu.org/licenses/>.
     19r"""Run S.M.A.R.T tests on hard drives."""
     20
     21import json
     22import os
     23import pathlib
     24import re
     25import shlex
     26import subprocess
     27import sys
     28
     29import fpyutils
     30import yaml
     31
     32
     33class UserNotRoot(Exception):
     34    """The user running the script is not root."""
     35
     36
     37def get_disks() -> list:
     38    r"""Scan all the disks."""
     39    disks = list()
     40    for d in pathlib.Path('/dev/disk/by-id').iterdir():
     41        # Ignore disks ending with part-${integer} to avoid duplicates (names
     42        # corresponding to partitions of the same disk).
     43        disk = str(d)
     44        if re.match('.+-part[0-9]+$', disk) is None:
     45            try:
     46                ddict = json.loads(
     47                    subprocess.run(
     48                        shlex.split('smartctl --capabilities --json ' +
     49                                    shlex.quote(disk)),
     50                        capture_output=True,
     51                        check=False,
     52                        shell=False,
     53                        timeout=30).stdout)
     54                try:
     55                    # Check for smart test support.
     56                    if ddict['ata_smart_data']['capabilities'][
     57                            'self_tests_supported']:
     58                        disks.append(disk)
     59                except KeyError:
     60                    pass
     61            except subprocess.TimeoutExpired:
     62                print('timeout for ' + disk)
     63            except subprocess.CalledProcessError:
     64                print('device ' + disk +
     65                      ' does not support S.M.A.R.T. commands, skipping...')
     66
     67    return disks
     68
     69
     70def disk_ready(disk: str, busy_status: int = 249) -> bool:
     71    r"""Check if the disk is ready."""
     72    # Raises a KeyError if disk has not S.M.A.R.T. status capabilities.
     73    ddict = json.loads(
     74        subprocess.run(shlex.split('smartctl --capabilities --json ' +
     75                                   shlex.quote(disk)),
     76                       capture_output=True,
     77                       check=True,
     78                       shell=False,
     79                       timeout=30).stdout)
     80    if ddict['ata_smart_data']['self_test']['status']['value'] != busy_status:
     81        return True
     82    else:
     83        return False
     84
     85
     86def run_test(disk: str, test_length: str = 'long') -> str:
     87    r"""Run the smartd test."""
     88    return subprocess.run(
     89        shlex.split('smartctl --test=' + shlex.quote(test_length) + ' ' +
     90                    shlex.quote(disk)),
     91        capture_output=True,
     92        check=True,
     93        shell=False,
     94        timeout=30).stdout
     95
     96
     97if __name__ == '__main__':
     98    if os.getuid() != 0:
     99        raise UserNotRoot
    100
    101    configuration_file = shlex.quote(sys.argv[1])
    102    config = yaml.load(open(configuration_file), Loader=yaml.SafeLoader)
    103
    104    # Do not prepend '/dev/disk/by-id/'.
    105    disks_to_check = shlex.quote(sys.argv[2])
    106    disks_available = get_disks()
    107
    108    for d in config['devices']:
    109        dev = '/dev/disk/by-id/' + d
    110        if config['devices'][d]['enabled'] and dev in disks_available:
    111            if disks_to_check == 'all' or disks_to_check == d:
    112                if disk_ready(dev, config['devices'][d]['busy_status']):
    113                    print('attempting ' + d + ' ...')
    114                    message = run_test(
    115                        dev, config['devices'][d]['test']).decode('utf-8')
    116                    print(message)
    117                    if config['devices'][d]['log']:
    118                        if config['notify']['gotify']['enabled']:
    119                            m = config['notify']['gotify'][
    120                                'message'] + ' ' + d + '\n' + message
    121                            fpyutils.notify.send_gotify_message(
    122                                config['notify']['gotify']['url'],
    123                                config['notify']['gotify']['token'], m,
    124                                config['notify']['gotify']['title'],
    125                                config['notify']['gotify']['priority'])
    126                        if config['notify']['email']['enabled']:
    127                            fpyutils.notify.send_email(
    128                                message,
    129                                config['notify']['email']['smtp_server'],
    130                                config['notify']['email']['port'],
    131                                config['notify']['email']['sender'],
    132                                config['notify']['email']['user'],
    133                                config['notify']['email']['password'],
    134                                config['notify']['email']['receiver'],
    135                                config['notify']['email']['subject'])
    136                else:
    137                    # Drop test requests if a disk is running a test in a particular moment.
    138                    # This avoid putting the disks under too much stress.
    139                    print('disk ' + d + ' not ready, checking the next...')
    
  6. create a configuration file

    includes/home/jobs/scripts/by-user/root/smartd_test.yaml#
     1#
     2# smartd_test.yaml
     3#
     4# Copyright (C) 2019-2020 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     5#
     6# This program is free software: you can redistribute it and/or modify
     7# it under the terms of the GNU General Public License as published by
     8# the Free Software Foundation, either version 3 of the License, or
     9# (at your option) any later version.
    10#
    11# This program is distributed in the hope that it will be useful,
    12# but WITHOUT ANY WARRANTY; without even the implied warranty of
    13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    14# GNU General Public License for more details.
    15#
    16# You should have received a copy of the GNU General Public License
    17# along with this program.  If not, see <http://www.gnu.org/licenses/>.
    18
    19devices:
    20    ata-disk1:
    21        enabled: true
    22        test: 'long'
    23        log: true
    24        busy_status: 249
    25    ata-disk2:
    26        enabled: true
    27        test: 'long'
    28        log: false
    29        busy_status: 249
    30    ata-diskn:
    31        enabled: true
    32        test: 'long'
    33        log: true
    34        busy_status: 249
    35
    36notify:
    37    gotify:
    38        enabled: true
    39        url: '<gotify url>'
    40        token: '<app token>'
    41        title: 'smart test'
    42        message: 'starting smart test on'
    43        priority: 5
    44    email:
    45        enabled: true
    46        smtp_server: 'smtp.gmail.com'
    47        port: 465
    48        sender: 'myusername@gmail.com'
    49        user: 'myusername'
    50        password: 'my awesome password'
    51        receiver: 'myusername@gmail.com'
    52        subject: 'smartd test'
    

    Important

    • absent devices are ignored

    • devices must be explicitly enabled

    • do not prepend /dev/disk/by-id/ to drive names

    • run a short test to get the busy_status value.

      smartctl -t short /dev/disk/by-id/${drive_name}
      

      You should be able to capture the value while the test is running by looking at the Self-test execution status: line. In my case it is always 249, but this value is not hardcoded in smartmontools’ source code

      smartctl --all /dev/disk/by-id/${drive_name}
      
  7. use this Systemd service unit file

    /home/jobs/services/by-user/root/smartd-test.ata_disk1.service#
    1[Unit]
    2Description=execute smartd on ata-disk1
    3
    4[Service]
    5Type=simple
    6ExecStart=/home/jobs/scripts/by-user/root/smartd_test.py /home/jobs/scripts/by-user/root/smartd_test.yaml ata-disk1
    7User=root
    8Group=root
    
  8. use this Systemd timer unit file

    /home/jobs/services/by-user/root/smartd-test.ata_disk1.timer#
    1[Unit]
    2Description=Once every two months smart test ata-disk1
    3
    4[Timer]
    5OnCalendar=*-01,03,05,07,09,11-01 00:00:00
    6Persistent=true
    7
    8[Install]
    9WantedBy=timers.target
    
  9. fix the permissions

    chmod 700 -R /home/jobs/scripts/by-user/smartd_test.*
    chmod 700 -R /home/jobs/services/by-user/root
    
  10. run the deploy script

Important

To avoid tests being interrupted you must avoid putting the disks to sleep, therefore, programs like hd-idle must be stopped before running the tests.

Services#

Notify unit status#

This script is useful to notfiy about failed Systemd service.

Some time ago my Gitea instance could not start after an update. If I used this script I would have known immediately about the problem instead of several days later.

See also

  • linux - get notification when systemd-monitored service enters failed state - Server Fault [11]

  1. install fpyutils. See reference

  2. create the script

    /home/jobs/scripts/by-user/root/notify_unit_status.py#
     1#!/usr/bin/env python3
     2#
     3# notify_unit_status.py
     4#
     5# Copyright (C) 2015 Pablo Martinez @ Stack Exchange (https://serverfault.com/a/701100)
     6# Copyright (C) 2018 Davy Landman @ Stack Exchange (https://serverfault.com/a/701100)
     7# Copyright (C) 2020-2021 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     8#
     9# This script is licensed under a
    10# Creative Commons Attribution-ShareAlike 4.0 International License.
    11#
    12# You should have received a copy of the license along with this
    13# work. If not, see <http://creativecommons.org/licenses/by-sa/4.0/>.
    14r"""Send a notification when a Systemd unit fails."""
    15
    16import shlex
    17import sys
    18
    19import fpyutils
    20import yaml
    21
    22if __name__ == '__main__':
    23    configuration_file = shlex.quote(sys.argv[1])
    24    config = yaml.load(open(configuration_file), Loader=yaml.SafeLoader)
    25
    26    failed_unit = shlex.quote(sys.argv[2])
    27
    28    message = 'service failure: ' + failed_unit
    29    if config['notify']['gotify']['enabled']:
    30        m = config['notify']['gotify']['message'] + '\n' + message
    31        fpyutils.notify.send_gotify_message(
    32            config['notify']['gotify']['url'],
    33            config['notify']['gotify']['token'], m,
    34            config['notify']['gotify']['title'],
    35            config['notify']['gotify']['priority'])
    36    if config['notify']['email']['enabled']:
    37        fpyutils.notify.send_email(message,
    38                                   config['notify']['email']['smtp server'],
    39                                   config['notify']['email']['port'],
    40                                   config['notify']['email']['sender'],
    41                                   config['notify']['email']['user'],
    42                                   config['notify']['email']['password'],
    43                                   config['notify']['email']['receiver'],
    44                                   config['notify']['email']['subject'])
    
  3. create a configuration file

    includes/home/jobs/scripts/by-user/root/notify_unit_status.yaml#
     1#
     2# notify_unit_status.yaml
     3#
     4# Copyright (C) 2015 Pablo Martinez @ Stack Exchange (https://serverfault.com/a/701100)
     5# Copyright (C) 2018 Davy Landman @ Stack Exchange (https://serverfault.com/a/701100)
     6# Copyright (C) 2020 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     7#
     8# This script is licensed under a
     9# Creative Commons Attribution-ShareAlike 4.0 International License.
    10#
    11# You should have received a copy of the license along with this
    12# work. If not, see <http://creativecommons.org/licenses/by-sa/4.0/>.
    13
    14notify:
    15    gotify:
    16        enabled: true
    17        url: '<gotify url>'
    18        token: '<app token>'
    19        title: 'service failure'
    20        message: 'service failure'
    21        priority: 5
    22    email:
    23        enabled: true
    24        smtp server: 'smtp.gmail.com'
    25        port: 465
    26        sender: 'myusername@gmail.com'
    27        user: 'myusername'
    28        password: 'my awesome password'
    29        receiver: 'myusername@gmail.com'
    30        subject: 'service failure'
    
  4. use this Systemd service unit file

    includes/home/jobs/services/by-user/root/notify-unit-status@.service#
     1#
     2# notify-unit-status@.service
     3#
     4# Copyright (C) 2015 Pablo Martinez @ Stack Exchange (https://serverfault.com/a/701100)
     5# Copyright (C) 2018 Davy Landman @ Stack Exchange (https://serverfault.com/a/701100)
     6# Copyright (C) 2020 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     7#
     8# This script is licensed under a
     9# Creative Commons Attribution-ShareAlike 4.0 International License.
    10#
    11# You should have received a copy of the license along with this
    12# work. If not, see <http://creativecommons.org/licenses/by-sa/4.0/>.
    13
    14[Unit]
    15Description=Unit Status Mailer Service
    16After=network-online.target
    17Wants=network-online.target
    18
    19[Service]
    20Type=simple
    21ExecStart=/home/jobs/scripts/by-user/root/notify_unit_status.py /home/jobs/scripts/by-user/root/notify_unit_status.yaml %I
    
  5. edit the Systemd service you want to monitor. In this example the service to be monitored is Gitea

    systemctl edit gitea.service
    
  6. add this content

    # [ ... ]
    
    [Unit]
    
    # [ ... ]
    
    OnFailure=notify-unit-status@%n.service
    
    # [ ... ]
    

Updates#

Update action#

This script can be used to update software not supported by the package manager, for example Docker images.

Important

Any arbitrary command can be configured.

See also

  • A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks [10]

  1. install the dependencies

    apt-get install python3-yaml python3-requests
    
  2. install fpyutils. See reference

  3. create the script

    /home/jobs/scripts/by-user/root/update_action.py#
     1#!/usr/bin/env python3
     2#
     3# update_action.py
     4#
     5# Copyright (C) 2021-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     6#
     7# This program is free software: you can redistribute it and/or modify
     8# it under the terms of the GNU General Public License as published by
     9# the Free Software Foundation, either version 3 of the License, or
    10# (at your option) any later version.
    11#
    12# This program is distributed in the hope that it will be useful,
    13# but WITHOUT ANY WARRANTY; without even the implied warranty of
    14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    15# GNU General Public License for more details.
    16#
    17# You should have received a copy of the GNU General Public License
    18# along with this program.  If not, see <http://www.gnu.org/licenses/>.
    19r"""update_action.py."""
    20
    21import shlex
    22import sys
    23
    24import fpyutils
    25import yaml
    26
    27
    28def send_notification(message: str, notify: dict):
    29    m = notify['gotify']['message'] + '\n' + message
    30    if notify['gotify']['enabled']:
    31        fpyutils.notify.send_gotify_message(notify['gotify']['url'],
    32                                            notify['gotify']['token'], m,
    33                                            notify['gotify']['title'],
    34                                            notify['gotify']['priority'])
    35    if notify['email']['enabled']:
    36        fpyutils.notify.send_email(
    37            message, notify['email']['smtp_server'], notify['email']['port'],
    38            notify['email']['sender'], notify['email']['user'],
    39            notify['email']['password'], notify['email']['receiver'],
    40            notify['email']['subject'])
    41
    42
    43if __name__ == '__main__':
    44
    45    def main():
    46        configuration_file = shlex.quote(sys.argv[1])
    47        config = yaml.load(open(configuration_file), Loader=yaml.SafeLoader)
    48
    49        # Action types. Preserve this order.
    50        types = ['pre', 'update', 'post']
    51        services = config['services']
    52
    53        for service in services:
    54            for type in types:
    55                for cmd in services[service]['commands'][type]:
    56                    for name in cmd:
    57                        retval = fpyutils.shell.execute_command_live_output(
    58                            cmd[name]['command'], dry_run=False)
    59                        if cmd[name]['notify']['success'] and retval == cmd[
    60                                name]['expected_retval']:
    61                            send_notification(
    62                                'command "' + name + '" of service "' +
    63                                service + '": OK', config['notify'])
    64                        elif cmd[name]['notify']['error'] and retval != cmd[
    65                                name]['expected_retval']:
    66                            send_notification(
    67                                'command "' + name + '" of service "' +
    68                                service + '": ERROR', config['notify'])
    69
    70    main()
    
  4. create a configuration file

    includes/home/jobs/scripts/by-user/root/update_action.mypurpose.yaml#
     1#
     2# update_action.mypurpose.yaml
     3#
     4# Copyright (C) 2021-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
     5#
     6# This program is free software: you can redistribute it and/or modify
     7# it under the terms of the GNU General Public License as published by
     8# the Free Software Foundation, either version 3 of the License, or
     9# (at your option) any later version.
    10#
    11# This program is distributed in the hope that it will be useful,
    12# but WITHOUT ANY WARRANTY; without even the implied warranty of
    13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    14# GNU General Public License for more details.
    15#
    16# You should have received a copy of the GNU General Public License
    17# along with this program.  If not, see <http://www.gnu.org/licenses/>.
    18
    19notify:
    20    email:
    21        enabled: true
    22        smtp_server: 'smtp.gmail.com'
    23        port: 465
    24        sender: 'myusername@gmail.com'
    25        user: 'myusername'
    26        password: 'my awesome password'
    27        receiver: 'myusername@gmail.com'
    28        subject: 'update action'
    29    gotify:
    30        enabled: true
    31        url: '<gotify url>'
    32        token: '<app token>'
    33        title: 'update action'
    34        message: 'update action'
    35        priority: 5
    36
    37services:
    38    hello:
    39        commands:
    40            pre:
    41                - stop_service:
    42                    # string
    43                    command: 'systemctl stop docker-compose.hello.service'
    44                    # integer
    45                    expected_retval: 0
    46                    # boolean: {true,false}
    47                    notify:
    48                        success: true
    49                        error: true
    50            update:
    51                - pull:
    52                    command: 'pushd /home/jobs/scripts/by-user/root/docker/hello && docker-compose pull'
    53                    expected_retval: 0
    54                    notify:
    55                        success: true
    56                        error: true
    57                - build:
    58                    command: 'pushd /home/jobs/scripts/by-user/root/docker/hello && docker-compose build --pull'
    59                    expected_retval: 0
    60                    notify:
    61                        success: true
    62                        error: true
    63            post:
    64                - start_service:
    65                    command: 'systemctl start docker-compose.hello.service'
    66                    expected_retval: 0
    67                    notify:
    68                        success: true
    69                        error: true
    70    goodbye:
    71        commands:
    72            pre:
    73                - stop_service:
    74                    command: 'systemctl stop docker-compose.goodbye.service'
    75                    expected_retval: 0
    76                    notify:
    77                        success: true
    78                        error: true
    79            update:
    80                - pull_only:
    81                    command: 'pushd /home/jobs/scripts/by-user/root/docker/goodbye && docker-compose pull'
    82                    expected_retval: 0
    83                    notify:
    84                        success: true
    85                        error: true
    86            post:
    87                - start_service:
    88                    command: 'systemctl start docker-compose.goodbye.service'
    89                    expected_retval: 0
    90                    notify:
    91                        success: true
    92                        error: true
    
  5. use this Systemd service unit file

    /home/jobs/services/by-user/root/update-action.mypurpose.service#
     1[Unit]
     2Description=Update action mypurpose
     3Wants=network-online.target
     4After=network-online.target
     5
     6[Service]
     7Type=simple
     8ExecStart=/home/jobs/scripts/by-user/root/update_action.py /home/jobs/scripts/by-user/root/update_action.mypurpose.yaml
     9User=root
    10Group=root
    
  6. use this Systemd timer unit file

    /home/jobs/services/by-user/root/update-action.mypurpose.timer#
    1[Unit]
    2Description=Update action mypurpose monthly
    3
    4[Timer]
    5OnCalendar=monthly
    6Persistent=true
    7
    8[Install]
    9WantedBy=timers.target
    
  7. fix the permissions

    chmod 700 -R /home/jobs/scripts/by-user/update_action.*
    chmod 700 -R /home/jobs/services/by-user/root
    
  8. run the deploy script

Footnotes