Monitoring
==========

Command assert
--------------

I use this script to check that the result of shell commands correspond to some
expected output. You can execute any arbirtary shell command.

If the resulting output is an unexpected one, a notification is sent.

The script also creates an RSS feed to complement the standard notifications.
The RSS feed file should be accessible by an HTTP server such as Apache.

.. figure:: assets/command_assert_0.png
   :scale: 50 %

   A Gotify notification showing a Gitea server error

Basic setup
```````````

#. install the dependencies

   .. code-block:: shell-session

      apt-get install python3-yaml python3-requests feedgenerator

#. install fpyutils. See :ref:`reference <installation of fpyutils>`
#. create a new user

   .. code-block:: shell-session

      useradd --system -s /bin/bash -U command-assert
      passwd command-assert
      usermod -aG jobs command-assert

#. create the jobs directories. See :ref:`reference <jobs instructions>`

   .. code-block:: shell-session

      mkdir -p /home/jobs/{scripts,services}/by-user/command-assert

#. create the :download:`script <includes/home/jobs/scripts/by-user/command-assert/command_assert.py>`

   .. literalinclude:: includes/home/jobs/scripts/by-user/command-assert/command_assert.py
      :language: python
      :caption: /home/jobs/scripts/by-user/command-assert/command_assert.py

#. create a :download:`configuration file <includes/home/jobs/scripts/by-user/command-assert/command_assert.mypurpose.yaml>`

   .. literalinclude:: includes/home/jobs/scripts/by-user/command-assert/command_assert.mypurpose.yaml
      :language: yaml
      :caption: /home/jobs/scripts/by-user/command-assert/command_assert.mypurpose.yaml

#. create a :download:`Systemd service unit file <includes/home/jobs/services/by-user/command-assert/command-assert.mypurpose.service>`

   .. literalinclude:: includes/home/jobs/services/by-user/command-assert/command-assert.mypurpose.service
      :language: ini
      :caption: /home/jobs/services/by-user/command-assert/command-assert.mypurpose.service

#. create a :download:`Systemd timer unit file <includes/home/jobs/services/by-user/command-assert/command-assert.mypurpose.timer>`

   .. literalinclude:: includes/home/jobs/services/by-user/command-assert/command-assert.mypurpose.timer
      :language: ini
      :caption: /home/jobs/services/by-user/command-assert/command-assert.mypurpose.timer

#. fix owners and permissions

   .. code-block:: shell-session

      chown -R command-assert:command-assert /home/jobs/{scripts,services}/by-user/command-assert
      chmod 700 -R /home/jobs/{scripts,services}/by-user/command-assert

#. run the :ref:`deploy script <deploy script>`

Sharing RSS feeds
`````````````````

We assume that the Apache HTTP webserver is up and running before following
these steps.

#. install the dependencies

   .. code-block:: shell-session

      apt-get install bindfs

#. add this entry to the :download:`fstab <includes/etc/command_assert.fstab>`
   file. In this example the directory is mounted in ``/srv/http/command_assert``

   .. literalinclude:: includes/etc/command_assert.fstab
      :language: ini
      :caption: /etc/fstab

#. create a directory readable be Apache

   .. code-block:: shell-session

      mkdir -p /srv/http/command_assert
      chown www-data:www-data /srv/http/command_assert
      chmod 700 /srv/http/command_assert

#. serve the files via HTTP by creating a new :download:`Apache virtual host <includes/etc/apache2/command_assert.apache.conf>`.
   Replace ``FQDN`` with the appropriate domain and include this file from the Apache configuration

   .. literalinclude:: includes/etc/apache2/command_assert.apache.conf
      :language: apache
      :caption: /etc/apache2/command_assert.apache.conf

#. create an HTML file to be served as an explanation for the RSS feeds

   .. literalinclude:: includes/srv/http/command_assert/footer.html
      :language: html
      :caption: /srv/http/command_assert/footer.html

#. restart the Apache webserver

   .. code-block:: shell-session

      systemctl restart apache2

#. run the bind-mount

   .. code-block:: shell-session

      mount -a