Scripts ======= Archiving --------- extract_gpx_data_from_dashcams.py ````````````````````````````````` Purpose ~~~~~~~ I use this script to delete old dashcam footage and replace it with gpx data extracted from the footage itself. This works for my *AUKEY DR02 D* dashcam (with its separate GPS unit). It should work for other dashcams as well. You can open the generated gpx files with programs like `GPXSee `_ or `GPX-viewer `_. Steps ~~~~~ 1. put the dashcam footage in the appropriate directory 2. edit the ``gpx.fmt`` file if needed .. important:: do not skip step 2. Read comments in file. References ~~~~~~~~~~ - https://www.topografix.com/gpx_manual.asp - http://owl.phy.queensu.ca/~phil/exiftool/geotag.html#Inverse - https://sno.phy.queensu.ca/~phil/exiftool/TagNames/GPS.html - https://sno.phy.queensu.ca/~phil/exiftool/faq.html Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python - perl Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.8.5 | +----------------------+------------+------------------+ | fpyutils | | 1.2.0 | +----------------------+------------+------------------+ | exiftool | - exiftool | 12.00 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> extract_gpx_data_from_dashcams.py: category: archiving running user: myuser configuration files: paths: - gpx.fmt - extract_gpx_data_from_dashcams.myuser.yaml systemd unit files: paths: service: - extract-gpx-data-from-dashcams.myuser.service timer: - extract-gpx-data-from-dashcams.myuser.timer ---- pdftoocr.sh ``````````` Purpose ~~~~~~~ I use this script to transform paper documents in ocr'd PDFs. Examples ~~~~~~~~ This script processes one file per directory. The output filename will be the SHA 1 sum of the directory name. For example, given ``documents/a/out.pdf``, three files will result: ================================================================ ===================================================================================================================================================================================== File name Description ================================================================ ===================================================================================================================================================================================== ``documents/a/86f7e437faa5a7fce15d1ddcb9eaeaea377667b8.pdf`` the compressed, archivable, grayscaled and OCR'd version of ``out.pdf`` ``documents/a/86f7e437faa5a7fce15d1ddcb9eaeaea377667b8.pdf.txt`` a text file of the OCR'd text from ``out.pdf`` ``documents/a/SHA512SUMS`` a checksum file containing the SHA 512 checksums of ``documents/a/86f7e437faa5a7fce15d1ddcb9eaeaea377667b8.pdf`` and ``documents/a/86f7e437faa5a7fce15d1ddcb9eaeaea377667b8.pdf.txt`` ================================================================ ===================================================================================================================================================================================== Infact ``$ echo -n 'a' | sha1sum`` corresponds to ``86f7e437faa5a7fce15d1ddcb9eaeaea377667b8``. Steps ~~~~~ 1. scan documents with ``$ simple-scan`` 2. save the output file as ``${OUTPUT_FILE}`` 3. if you want to keep colors, run ``$ touch "${COLOR_OVERRIDE_FILE}"`` in the directory. This file will be automatically deleted once the script ends. .. important:: Along with installing the listed dependencies you need to install the appropriate Tesseract language data files. References ~~~~~~~~~~ - https://unix.stackexchange.com/a/93971 Programming languages ~~~~~~~~~~~~~~~~~~~~~ - bash Dependencies ~~~~~~~~~~~~ +----------------------+---------------+------------------+ | Name | Binaries | Version | +======================+===============+==================+ | GNU Bash | - bash | 5.0.007 | +----------------------+---------------+------------------+ | Findutils | - find | 4.6.0 | +----------------------+---------------+------------------+ | Gawk | - gawk | 4.2.1 | +----------------------+---------------+------------------+ | GNU Coreutils | - chmod | 8.31 | | | - cp | | | | - date | | | | - env | | | | - rm | | | | - sha1sum | | | | - sha512sum | | | | - sort | | | | - sync | | | | - tr | | +----------------------+---------------+------------------+ | Ghostscript | - gs | 9.27 | +----------------------+---------------+------------------+ | OCRmyPDF | - ocrmypdf | 8.3.0 | +----------------------+---------------+------------------+ | Document Scanner | - simple-scan | 3.36.0 | +----------------------+---------------+------------------+ | Tesseract OCR | | 4.1.1 | +----------------------+---------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ .. important:: It is very important to set the ``OCR_LANG`` variable. Licenses ~~~~~~~~ - CC-BY-SA 3.0 YAML data ~~~~~~~~~ :: <--YAML--> pdftoocr.sh: category: archiving running user: myuser configuration files: paths: - pdftoocr_deploy.sh - pdftoocr_deploy.myuser_documents.conf - pdftoocr.myuser_documents.conf systemd unit files: paths: service: - pdftoocr.myuser_documents.service timer: - pdftoocr.myuser_documents.timer ---- youtube_dl.py ````````````` Purpose ~~~~~~~ I use this script to download and archive videos from various platforms. Steps ~~~~~ 1. get a list of urls and divide them by subject 2. optionally run common command 1 References ~~~~~~~~~~ - https://wiki.archlinux.org/index.php/Youtube-dl - https://gotify.net/ - https://gotify.net/docs/pushmsg Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+----------------+------------------+ | Name | Binaries | Version | +======================+================+==================+ | youtube-dl | - youtube-dl | 2020.06.16.1 | +----------------------+----------------+------------------+ | Python | - python3 | 3.8.4 | +----------------------+----------------+------------------+ | aria2 | | 1.35.0 | +----------------------+----------------+------------------+ | fpyutils | | 1.2.0 | +----------------------+----------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+----------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ Three files must exist for each subject: 1. the ``*.yaml`` file is a generic configuration file 2. the ``*.options`` file contains most of the options used by ``youtube-dl`` 3. the ``*txt`` file contains a list of source URLs Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> youtube_dl.py: category: archiving running user: myuser configuration files: paths: - youtube_dl.some_subject.yaml - youtube_dl.some_subject.options - youtube_dl.some_subject.txt systemd unit files: paths: service: - youtube-dl.some_subject.service timer: - youtube-dl.some_subject.timer ---- archive_invoice_files.py ```````````````````````` Purpose ~~~~~~~ I use this script to archive and print invoice files. Invoice files are downloaded from `PEC accounts (certified mail) `_ as attachments. An HTML file corresponding to the XML invoice file is archived and printed. Finally, a notification is sent to a Gotify instance. During this process, cryptographical signatures and integrity checks are performed. Steps ~~~~~ 1. install the CUPS development headers 2. Create a new virtual environment as explained in `this post `_, and call it ``archive_invoice_files``. Once activated you can run these commands, tested for ``fattura-elettronica-reader`` version ``2.1.0``: :: pip3 install wheel pip3 install requests fpyutils==2.1.0 python-dateutil fattura-elettronica-reader==3.0.0 WeasyPrint==52.1 pycups lxml 3. optionally run common command 1 .. important:: To be able to install pycups and to use WeasyPrint, `CUPS `_ must be already installed. .. warning:: If an error similar to this is raised: ``UserWarning: FontConfig: No fonts configured. Expect ugly output.`` , install a font such as `DejaVu `_. References ~~~~~~~~~~ - https://github.com/markuz/scripts/blob/master/getmail.py Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------------+------------+------------------+ | Name | Binaries | Version | +============================+============+==================+ | Python | - python3 | 3.9.0 | +----------------------------+------------+------------------+ | Requests | | 2.26.0 | +----------------------------+------------+------------------+ | dateutil | | 2.8.1 | +----------------------------+------------+------------------+ | lxml | | 4.6.2 | +----------------------------+------------+------------------+ | pycups | | 2.0.1 | +----------------------------+------------+------------------+ | WeasyPrint | | 52.1 | +----------------------------+------------+------------------+ | fattura-elettronica-reader | | 2.1.0 | +----------------------------+------------+------------------+ | fpyutils | | 2.1.0 | +----------------------------+------------+------------------+ | PyYAML | | 6.0 | +----------------------------+------------+------------------+ Licenses ~~~~~~~~ - GPLv2+ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> archive_invoice_files.py: category: archiving running user: myuser configuration files: paths: - archive_invoice_files.myuser.yaml systemd unit files: paths: service: - archive-invoice-files.myuser.service timer: - archive-invoice-files.myuser.timer ---- archive_media_files.py `````````````````````` Purpose ~~~~~~~ I use this script to archive media files from removable drives such as SD cards. Files are archived using this schema: :: ${device_uuid}/${year}/${month} Udisks2 hanged frequently, so I had to write this new script which uses traditional mount commands. Parallelization of rsync and of metadata extraction was also added. Steps ~~~~~ 1. get a device with media files 2. get the filesystem UUID with: ``$ lsblk -o name,uuid`` 3. Follow the *Automatic backup on a removable USB drive on plug in* example in the borgmatic_hooks.py script 4. get the user id and group id of the user corresponding to the path where the files will be archived 5. References ~~~~~~~~~~ - https://wiki.archlinux.org/index.php?title=Udisks&oldid=575618#udevadm_monitor - https://software.franco.net.eu.org/frnmst/automated-tasks/src/commit/67415cdd7224ff21a2f39bb8180ee36cf6e6e31e/archiving/archive_documents_simple.sh - https://blog.franco.net.eu.org/notes/automatic-removable-media-synchronization.html Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.8.5 | +----------------------+------------+------------------+ | exiftool | - exiftool | 11.16 | +----------------------+------------+------------------+ | fpyutils | | 1.2.2 | +----------------------+------------+------------------+ | rsync | | 3.1.3 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ Add the following to ``/etc/udev/rules.d/99-usb-automount.rules`` and replace: - ``${UUID}``: the UUID as explained above :: ACTION=="add", SUBSYSTEMS=="usb", ENV{ID_FS_UUID}=="${UUID}", SUBSYSTEM=="block", RUN{program}+="/usr/bin/bash -c '(/usr/bin/systemctl start ${MOUNT}.mount && /usr/bin/systemctl start archive-media-files.mypurpose.service; /usr/bin/systemctl start udev-umount.home-myuser-media-auto-backup.service) &'" Then run: :: udevadm control --reload udevadm trigger Systemd unit files ~~~~~~~~~~~~~~~~~~ You can use ``$$(systemd-escape ${mountpoint})`` to escape the strings correctly. Licenses ~~~~~~~~ - GFDLv1.3+ YAML data ~~~~~~~~~ :: <--YAML--> archive_media_files.py: category: archiving running user: root configuration files: paths: - archive_media_files.mypurpose.yaml systemd unit files: paths: service: - archive-media-files.mypurpose.service - udev-umount.home-myuser-media-auto-backup.service ---- archive_emails.py ````````````````` Purpose ~~~~~~~ I use this script to get a local copy of all my emails. References ~~~~~~~~~~ - https://wiki.archlinux.org/index.php/OfflineIMAP - https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap.conf Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+---------------+------------------+ | Name | Binaries | Version | +======================+===============+==================+ | Python | - python3 | 3.8.5 | +----------------------+---------------+------------------+ | fpyutils | | 1.2.0 | +----------------------+---------------+------------------+ | OfflineIMAP | - offlineimap | 7.3.3 | +----------------------+---------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+---------------+------------------+ Licenses ~~~~~~~~ - GFDLv1.3+ YAML data ~~~~~~~~~ :: <--YAML--> archive_emails.py: category: archiving running user: myuser configuration files: paths: - archive_emails.myuser.yaml - archive_emails.myuser.options systemd unit files: paths: service: - archive-emails.myuser.service timer: - archive-emails.myuser.timer ---- archive_media_with_label.py ``````````````````````````` Purpose ~~~~~~~ I use this script to add a label to phjysical supports sucs as tapes, CDs, etc... Steps ~~~~~ 1. run the program with the appriopriate parameters 2. rename the file 3. print or write down the label and stick it on the media 4. once you have filled a box, print or write down all the labels as a single one and stick it on the box Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | python-tabulate | | 0.8.7 | +----------------------+------------+------------------+ | fpyutils | | 1.2.0 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> archive_media_with_label.py: category: archiving running user: myuser configuration files: paths: - archive_media_with_label.yaml ---- Audio ------ set-turntable-loopback-sound.service ```````````````````````````````````` Purpose ~~~~~~~ I use this script to enable the loopback sound of a SONY PS-LX300USB turntable. Steps ~~~~~ 1. connect the turntable via USB 2.0 type B to the computer Programming languages ~~~~~~~~~~~~~~~~~~~~~ - bash Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | GNU Bash | - bash | 5.0.007 | +----------------------+------------+------------------+ | alsa-utils | - arecord | 1.1.9 | | | - aplay | | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ To avoid ``aplay`` bloking the output, configure ALSA with dmix PCMs. Use `aplay -l` to find the device names. In my case I also want to duplicate the analog and HDMI output but there is, however, a slight delay of the HDMI audio. Licenses ~~~~~~~~ - CC-BY-SA 3.0 YAML data ~~~~~~~~~ :: <--YAML--> set-turntable-loopback-sound.service: category: audio running user: mydesktopuser configuration files: paths: - set-turntable-loopback-sound.asoundrc systemd unit files: paths: service: - set-turntable-loopback-sound.service ---- Backups ------- borgmatic_hooks.py `````````````````` Purpose ~~~~~~~ I use this script to send notifications during hard drive backups. A script to mount the backed up archives is also included here. Examples ~~~~~~~~ Automatic backup on a removable USB drive on plug in .................................................... I use a variation of this script to archive important documents on USB flash drives just in case all the backups fail. After creating a filesystem, add its entry in the ``/etc/fstab`` file. See also https://www.freedesktop.org/software/systemd/man/systemd.mount.html#fstab Remove the ``ExecStartPre`` instruction from the provided systemd service unit file. To automatically mount the filesystem create a file called ``/etc/udev/rules.d/99-usb-automount.rule`` and add a udev rule like this: :: ACTION=="add", SUBSYSTEMS=="usb", ENV{ID_FS_UUID}=="${filesystem UUID}", SUBSYSTEM=="block", RUN{program}+="/usr/bin/bash -c '(/usr/bin/systemctl start backed_up_mountpoint.mount && systemctl start borgmatic.myhostname_backed_up_mountpoint.service && /usr/bin/systemctl start udev-umount.myhostname_backed_up_mountpoint.service) &'" where ``${filesystem UUID}`` corresponds to ``# udevadm info --name=${partition} | grep "ID_FS_UUID="`` Finally, use the provided ``udev-umount.backed-up-mountpoint.service`` file. Steps ~~~~~ 1. create a new borg repository .. note:: We want to avoid encryption because: 1. it works with older versions of borg 2. it is simpler 3. these are not offsite backups .. important:: There are two different types of setups: local and remote repositories. .. note:: We will assume that: - our source directory is a mountpoint at ``/backed/up/mountpoint``. This makes sense if we want to backup ``/root`` or ``/home`` for example. - our borg directories will be under ``/mnt/backups`` For example, if we want to backup ``/home`` and our hostname is ``mypc`` we would have: ``/mnt/backups/mypc_home.borg`` To create a local repository run: :: $ borg init -e none /mnt/backups/myhostname_backed_up_mountpoint.borg For remote repositories run common command 1 using ``borgmatic`` as parameter on the destination (backup) server. Create an SSH key pair so that you can connect to the destination server. On the source server run: :: $ borg init -e none user@host:/mnt/backups/myhostname_backed_up_mountpoint.borg 2. edit the Borgmatic YAML configuration file References ~~~~~~~~~~ - https://torsion.org/borgmatic/ - https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/ - https://torsion.org/borgmatic/docs/how-to/deal-with-very-large-backups/ - https://borgbackup.readthedocs.io/en/stable/usage/init.html?highlight=encryption - https://medspx.fr/projects/backup/ - https://borgbackup.readthedocs.io/en/stable/deployment/image-backup.html - https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.service - https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer - https://superuser.com/questions/1016827/how-do-i-run-a-script-before-everything-else-on-shutdown-with-systemd Programming languages ~~~~~~~~~~~~~~~~~~~~~ - bash Dependencies ~~~~~~~~~~~~ +----------------------+-------------+------------------+ | Name | Binaries | Version | +======================+=============+==================+ | GNU Bash | - bash | 5.1.004 | +----------------------+-------------+------------------+ | Python | - python3 | 3.9.1 | +----------------------+-------------+------------------+ | fpyutils | | 1.2.2 | +----------------------+-------------+------------------+ | borgmatic | - borgmatic | 1.5.12 | +----------------------+-------------+------------------+ | Python-LLFUSE | | 1.3.8 | +----------------------+-------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ I use a set of configuration files per mountpoint to back up. Systemd unit files ~~~~~~~~~~~~~~~~~~ I use a set of configuration files per mountpoint to back up. To mount all the archives of a borg backup you simply must run the borgmatic-mount service. To unmount them stop the service. .. tip:: You can use this systemd service unit file to backup when the computer shuts down. When my computer shuts down my home directory gets backed up on the server. What I need are the configuration and *normal* files: I don't care about ``~/.cache``, the shell history nor the browser's history and cache. You should edit the configuration file to reflect that. Although this service remains active all the time, the syncronization action runs when the system is halted using an ``ExecStop`` directive. Since we don't know how much time the syncronization takes a ``TimeoutStopSec=infinity`` directive is present. :: # # borgmatic.myhostname_backed_up_mountpoint.service # # Copyright (C) 2016-2020 Dan Helfman # 2020 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see . # See https://superuser.com/questions/1016827/how-do-i-run-a-script-before-everything-else-on-shutdown-with-systemd # # Copyright (C) 2015 le_me @ Stack Overflow (https://superuser.com/a/1016848) # Copyright (C) 2017 Community @ Stack Overflow (https://superuser.com/a/1016848) # Copyright (C) 2020 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com) # # This script is licensed under a # Creative Commons Attribution-ShareAlike 3.0 International License. # # You should have received a copy of the license along with this # work. If not, see . [Unit] Description=borgmatic myhostname_backed_up_mountpoint backuup Wants=network-online.target After=network-online.target ConditionACPower=true Requires=backed-up-mountpoint.mount Requires=mnt-backups-myhostname_backed_up_mountpoint.mount After=backed-up-mountpoint.mount After=mnt-backups-myhostname_backed_up_mountpoint.mount [Service] Type=oneshot # Lower CPU and I/O priority. Nice=19 CPUSchedulingPolicy=batch IOSchedulingClass=best-effort IOSchedulingPriority=7 IOWeight=100 # Do not Retry. Restart=no # Prevent rate limiting of borgmatic log events. If you are using an older version of systemd that # doesn't support this (pre-240 or so), you may have to remove this option. LogRateLimitIntervalSec=0 ExecStart=/bin/true RemainAfterExit=yes TimeoutStopSec=infinity ExecStop=/usr/bin/borgmatic --config /home/jobs/scripts/by-user/root/borgmatic.myhostname_backed_up_mountpoint.yaml --syslog-verbosity 1 User=root Group=root [Install] WantedBy=multi-user.target Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> borgmatic_hooks.py: category: backups running user: root configuration files: paths: - borgmatic.myhostname_backed_up_mountpoint.yaml - borgmatic_hooks.myhostname_backed_up_mountpoint.yaml - borgmatic_mount.myhostname_backed_up_mountpoint.yaml systemd unit files: paths: service: - borgmatic.myhostname_backed_up_mountpoint.service - borgmatic-mount.myhostname_backed_up_mountpoint.service - udev-umount.backed-up-mountpoint.service timer: - borgmatic.myhostname_backed_up_mountpoint.timer ---- Desktop ------- set_display_gamma.sh ```````````````````` Purpose ~~~~~~~ I use this to automatically set a better gamma for the output on a tv. References ~~~~~~~~~~ - https://askubuntu.com/a/62270 Programming languages ~~~~~~~~~~~~~~~~~~~~~ - bash Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | GNU Bash | - bash | 5.0.007 | +----------------------+------------+------------------+ | Xorg | - xrandr | 1.5.0 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ Make sure that the ``XORG_DISPLAY`` variable is set correctly. To find out the current display variable run ``$ echo ${DISPLAY}`` Licenses ~~~~~~~~ - CC-BY-SA 3.0 YAML data ~~~~~~~~~ :: <--YAML--> set_display_gamma.sh: category: desktop running user: mydesktopuser configuration files: paths: - set_display_gamma.TV_HDMI1.conf systemd unit files: paths: service: - set-display-gamma.service timer: - set-display-gamma.timer ---- Development ----------- build_python_packages.py ```````````````````````` Purpose ~~~~~~~ Build Python packages using git sources and push them to a self-hosted PyPI server. Steps ~~~~~ 1. create a virtual machine with Debian Bullseye (stable) and transform it into Sid (unstable). Using the unstable version will provide more up to date software for development 2. clone the `python-packages-source `_ repository in the ``repository.path``. See the configuration file 3. install and run PyPI server such as `pypiserver `_. You can use a docker compose file like this one: :: version: '3.8' services: pypiserver-authenticated: image: pypiserver/pypiserver:v1.4.2 volumes: # Authentication file. - type: bind source: /home/jobs/scripts/by-user/root/docker/pypiserver/auth target: /data/auth # Python files - type: bind source: /data/pypiserver/packages target: /data/packages ports: - "4000:8080" # I have purposefully removed the # --fallback-url https://pypi.org/simple/ # option to have a fully isolated environment. command: --disable-fallback --passwords /data/auth/.htpasswd --authenticate update /data/packages 4. install these packages in the virtual machine: :: apt-get install build-essential fakeroot devscripts git python3-dev python3-all-dev \ games-python3-dev libgmp-dev libssl-dev libssl1.1=1.1.1k-1 libcurl4-openssl-dev \ python3-pip python3-build twine libffi-dev graphviz libgraphviz-dev pkg-config \ clang-tools libblas-dev astro-all libblas-dev libatlas-base-dev libopenblas-dev \ libgsl-dev libblis-dev liblapack-dev liblapack3 libgslcblas0 libopenblas-base \ libatlas3-base libblas3 clang-9 clang-13 clang-12 clang-11 sphinx-doc \ libbliss-dev libblis-dev libbliss2 libblis64-serial-dev libblis64-pthread-dev \ libblis64-openmp-dev libblis64-3-serial libblis64-dev libblis64-3-pthread \ libblis64-3-openmp libblis64-3 libblis3-serial libblis3-pthread \ libblis-serial-dev libblis-pthread-dev libargon2-dev libargon2-0 libargon2-1 .. note:: This is just a selection. Some Python packages need other dependencies not listed here. 5. you need to manually compile at least these packages and push them to you local PyPI to avoid a chiken and egg problem: - ``setuptools`` - ``setuptools_scm`` - ``wheel`` 1. compile a package like this :: cd python-packages-source python3 -m build --sdist --wheel .. important:: Some packages might need different dependencies. Have a look at the ``setup_requires`` variable in ``setup.py`` or in ``setup.cfg`` or ``requires`` in the ``pyproject.toml`` file. If you cannot compile some, download them directly from `pypi.python.org `_. 2. upload it to your PyPI server :: twine upload --repository-url ${your_pypi_index_url} dist/* 6. change the PyPI index of your programs. See for example https://software.franco.net.eu.org/frnmst/python-packages-source#client-configuration References ~~~~~~~~~~ - https://stackoverflow.com/a/13847807 - https://github.com/pypa/build/issues/328#issuecomment-877028239 Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.9.8 | +----------------------+------------+------------------+ | fpyutils | | 2.0.1 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ | appdirs | | 1.4.4 | +----------------------+------------+------------------+ | Twine | - twine | 3.5.0 | +----------------------+------------+------------------+ | build | | 0.7.0 | +----------------------+------------+------------------+ | Git | - git | 2.33.1 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GPLv3+ - CC-BY-SA 4.0 YAML data ~~~~~~~~~ :: <--YAML--> build_python_packages.py: category: development running user: python-source-packages-updater configuration files: paths: - build_python_packages.yaml systemd unit files: paths: service: - build-python-packages.service timer: - build-python-packages.timer ---- Drives ------ smartd_test.py `````````````` Purpose ~~~~~~~ I use this to run periodical S.M.A.R.T. tests on the hard drives. Steps ~~~~~ 1. run ``# hdparm -I ${drive}`` and compare the results with ``$ ls /dev/disk/by-id`` to know which drive corresponds to the one you want to work on 2. optionally run common command 1 .. important:: To avoid tests being interrupted you must avoid putting the disks to sleep, therefore, programs like `hd-idle `_ must be stopped before running the tests. References ~~~~~~~~~~ Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.7.4 | +----------------------+------------+------------------+ | Smartmontools | - smartctl | 7.0 | +----------------------+------------+------------------+ | fpyutils | | 1.2.3 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ The script supports only ``/dev/disk/by-id`` names. See also the udev rule file ``/lib/udev/rules.d/60-persistent-storage.rules``. Systemd unit files ~~~~~~~~~~~~~~~~~~ I use one file per drive so I can control when a certain drive performs testing, instead of running them all at once. Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> smartd_test.py: category: drives running user: root configuration files: paths: - smartd_test.yaml systemd unit files: paths: service: - smartd-test.ata_disk1.service timer: - smartd-test.ata_disk1.timer ---- mdamd_check.py `````````````` Purpose ~~~~~~~ I use this to run periodical RAID data scrubs on the hard drives. Steps ~~~~~ 1. run ``$ lsblk`` to know the names of the mdadm devices. See also: ``$ cat /proc/mdstat`` 2. optionally run common command 1 References ~~~~~~~~~~ - https://blog.franco.net.eu.org/notes/raid-data-scrubbing.html Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.7.3 | +----------------------+------------+------------------+ | fpyutils | | 1.2.3 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GPLv2+ YAML data ~~~~~~~~~ :: <--YAML--> mdamd_check.py: category: drives running user: root configuration files: paths: - mdadm_check.yaml systemd unit files: paths: service: - mdamd-check.service timer: - mdamd-check.timer ---- File sharing ------------ rtorrent ```````` Purpose ~~~~~~~ I use this to automatically start and manage the torrents. Steps ~~~~~ 1. run common command 0 using ``rtorrent`` as parameter 2. copy the provided configuration file into ``/home/rtorrent/.rtorrent.rc`` References ~~~~~~~~~~ - https://wiki.archlinux.org/index.php/RTorrent#With_screen Programming languages ~~~~~~~~~~~~~~~~~~~~~ - bash Dependencies ~~~~~~~~~~~~ +----------------------+----------------+------------------+ | Name | Binaries | Version | +======================+================+==================+ | RTorrent | - rtorrent | 0.9.8 | +----------------------+----------------+------------------+ | GNU Screen | - screen | 4.8.0 | +----------------------+----------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ .. warning:: The provided configuration file is based on an old version of RTorrent. Some parameters might be deprecated. .. note:: It is assumed that the downloaded files are placed under ``/data/incoming_torrents``. Licenses ~~~~~~~~ - GFDLv1.3+ YAML data ~~~~~~~~~ :: <--YAML--> rtorrent: category: file-sharing running user: rtorrent configuration files: paths: - rtorrent.rc systemd unit files: paths: service: - rtorrent.service ---- kiwix_manage.py ``````````````` .. image:: assets/images/kiwix_manage.py_0.png .. image:: assets/images/kiwix_manage.py_1.png Purpose ~~~~~~~ I use this to download and read Wikipedia as well as other websites offline. Steps ~~~~~ 1. run common command 2 using ``kiwix`` as parameter References ~~~~~~~~~~ - https://www.kiwix.org/en/ - https://download.kiwix.org/zim/ - https://stackoverflow.com/a/39217788 - https://stackoverflow.com/a/53811881 - https://docs.python.org/3/library/shutil.html?highlight=shutil#shutil.rmtree.avoids_symlink_attacks Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+----------------+------------------+ | Name | Binaries | Version | +======================+================+==================+ | Python | - python3 | 3.8.2 | +----------------------+----------------+------------------+ | Requests | | 2.23.0 | +----------------------+----------------+------------------+ | BeautifulSoup | | 4.8.0 | +----------------------+----------------+------------------+ | PyYAML | | 4.8.2 | +----------------------+----------------+------------------+ | aria2 | - aria2c | 1.35.0 | +----------------------+----------------+------------------+ | Kiwix tools | - kiwix-serve | 3.0.1 | +----------------------+----------------+------------------+ | fpyutils | | 1.2.3 | +----------------------+----------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+----------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ It is recommended to use aria2c instead of requests as downloader. aria2c infact supports bandwidth throttling and continuation from interrupted downloads. Systemd unit files ~~~~~~~~~~~~~~~~~~ .. important:: After downloading a new file you must rerun ``kiwix-manage.serve.service``. Licenses ~~~~~~~~ - GPLv3+ - CC-BY-SA 4.0 YAML data ~~~~~~~~~ :: <--YAML--> kiwix_manage.py: category: file-sharing running user: kiwix configuration files: paths: - kiwix-manage.yaml systemd unit files: paths: service: - kiwix-manage.download.service - kiwix-manage.serve.service timer: - kiwix-manage.download.timer ---- Misc ---- monitor_and_notify_git_repo_changes.sh `````````````````````````````````````` Purpose ~~~~~~~ My `Gitea `_ instance is configured to mirror some repositories. Every 30 minutes this script checks for new commits in those bare git repositories. If something new is commited a notification is sent to my `Gotify `_ instance. .. note:: This script also works for non-bare git repositories. Steps ~~~~~ 1. run common command 1 References ~~~~~~~~~~ - https://gitea.io/en-us/ - https://gotify.net/ - https://gotify.net/docs/pushmsg Programming languages ~~~~~~~~~~~~~~~~~~~~~ - bash Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | GNU Bash | - bash | 5.0.007 | +----------------------+------------+------------------+ | curl | - curl | 7.66.0 | +----------------------+------------+------------------+ | Git | - git | 2.23.0 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ To avoid missing or reading duplicate messages, the variable ``CHECK_TIMEOUT_INTERVAL_SECONDS`` should be set to the same value as the one in the systemd timer unit file (``OnCalendar``). Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> monitor_and_notify_git_repo_changes.sh: category: misc running user: gitea configuration files: paths: - monitor_and_notify_git_repo_changes.myrepos.conf systemd unit files: paths: service: - monitor-and-notify-git-repo-changes.myrepos.service timer: - monitor-and-notify-git-repo-changes.myrepos.timer ---- yacy ```` Purpose ~~~~~~~ A personal search engine. Steps ~~~~~ 1. setup `YaCy `_ and run an instance .. note:: To install YaCy you need the `OpenJDK Java 13 headless runtime environment `_ package. 2. run common command 2 using ``yacy`` as parameter 3. clone the YaCy search server repository into ``/home/yacy``: :: $ git clone https://github.com/yacy/yacy_search_serve.git References ~~~~~~~~~~ - https://yacy.net/index.html - https://github.com/yacy/yacy_search_server Programming languages ~~~~~~~~~~~~~~~~~~~~~ - bash - java Dependencies ~~~~~~~~~~~~ +----------------------+----------------+------------------+ | Name | Binaries | Version | +======================+================+==================+ | YaCy | - startYACY.sh | | | | - stopYACY.sh | | +----------------------+----------------+------------------+ Licenses ~~~~~~~~ - LGPLv2+ YAML data ~~~~~~~~~ :: <--YAML--> yacy: category: misc running user: yacy systemd unit files: paths: service: - yacy-search-server.service ---- notify_camera_action.py ``````````````````````` Purpose ~~~~~~~ Notify when a camera connected to a system running `Motion `_ is found or lost (disconnected). .. important:: We will assume that a `Motion `_ instance is configured and running. Steps ~~~~~ 1. edit a camera's configuration file with: :: # Run camera actions. on_camera_lost /home/jobs/scripts/by-user/motion/notify_camera_action.py /home/jobs/scripts/by-user/motion/notify_camera_action.yaml "%$ (id: %t)" "lost" on_camera_found /home/jobs/scripts/by-user/motion/notify_camera_action.py /home/jobs/scripts/by-user/motion/notify_camera_action.yaml "%$ (id: %t)" "found" 2. optionally run common command 1 References ~~~~~~~~~~ - https://motion-project.github.io/motion_config.html Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.8.5 | +----------------------+------------+------------------+ | fpyutils | | 1.2.0 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ A single file is used for all the cameras connected to a system. Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> notify_camera_action.py: category: misc running user: motion configuration files: paths: - notify_camera_action.yaml ---- save_and_notify_file_diffs.py ````````````````````````````` Purpose ~~~~~~~ Track files on the web: when a file changes push it to a VCS repository and send notifications. Examples ~~~~~~~~ I use this script to track changes for assets of `fattura-elettronica-reader `_. See also https://docs.franco.net.eu.org/fattura-elettronica-reader/assets.html Steps ~~~~~ 1. create a VCS repository for example with Git and clone it locally. 2. optionally run common command 1 References ~~~~~~~~~~ None Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | GNU Bash | - bash | 5.1.004 | +----------------------+------------+------------------+ | Python | - python3 | 3.9.1 | +----------------------+------------+------------------+ | fpyutils | | 1.2.2 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ The configuration file uses git but you can adapt it to work with other VCS. .. note:: The running user is ``gitea`` because I use this script with a Gitea instance. .. important:: This script has only been tested with Git. Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> save_and_notify_file_diffs.py: category: misc running user: gitea configuration files: paths: - save_and_notify_file_diffs.myrepo.yaml systemd unit files: paths: service: - save-and-notify-file-diffs.myrepo.service timer: - save-and-notify-file-diffs.myrepo.timer ---- feed_proxy.py ````````````` Purpose ~~~~~~~ I use this to get some feeds which result to be unreadable from `TT-RSS `_. In my case, the `Apache `_ webserver is used as a MITM agent. Steps ~~~~~ 1. run common command 2 using ``rss`` as parameter 2. add the rss user to the group of the user running the webserver (``www-data`` for Apache on Debian GNU/Linux) 3. edit the configuration so that ``files.base path`` points to a readable directory of the webserver, for example ``/var/www`` Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | fpyutils | | 1.2.2 | +----------------------+------------+------------------+ | Requests | | 2.25.1 | +----------------------+------------+------------------+ | Python | - python3 | 3.7.3 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Systemd unit files ~~~~~~~~~~~~~~~~~~ By default torsocks is used to run the python script. If you don't care about getting your feeds through TOR just remove ``/usr/bin/torsocks`` from ``ExecStart``. Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> feed_proxy.py: category: misc running user: rss configuration files: paths: - feed_proxy.mypurpose.yaml systemd unit files: paths: service: - feed-proxy.mypurpose.service timer: - feed-proxy.mypurpose.timer ---- monthly_attendance_paper.py ``````````````````````````` Purpose ~~~~~~~ Automatically print a 2 cloumn monthly attendace paper to be used between an employer and employee. Steps ~~~~~ 1. configure a default printer. See also `https://blog.franco.net.eu.org/notes/cups-simple-shared-printer.html `_ Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.7.3 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ .. note:: The YAML configuration file contains raw tab and newline characters Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> monthly_attendance_paper.py: category: misc running user: myuser configuration files: paths: - monthly_attendance_paper.mypurpose.yaml systemd unit files: paths: service: - monthly-attendance-paper.mypurpose.service timer: - monthly-attendance-paper.mypurpose.timer ---- cryptod.bitcoin.service ``````````````````````` Purpose ~~~~~~~ Run the Bitcoin daemon (or any other Bitcoin-based daemon). Steps ~~~~~ 1. run common command 0 using ``cryptocurrencies`` as parameter 2. install the bitcoin core: ``bitcoind`` and ``bitcoin-cli`` are the only two binaries needed :: $ git clone "https://github.com/bitcoin/bitcoin.git" $ git checkout v0.21.1 # apt-get install build-essential libtool autotools-dev automake pkg-config bsdmainutils python3 libboost-all-dev libevent-dev $ ./autogen.sh $ ./configure --with-incompatible-bdb --without-gui --without-miniupnpc $ make # make install 3. make sure that you have enough space for the blockchain in ``/home/cryptocurrencies/bitcoin``. If that is not the case you can change the ``datadir`` option in ``cryptod.bitcoin.service``. 4. run the daemon: the whole blockchain needs to be downloaded before doing transactions 5. create a file called ``~/.bitcoin/bitcoin.conf`` with the user running ``bitcoin-cli``, with the username and password corresponding to the ones of ``bitcoin.conf``: :: rpcuser=cryptocurrencies rpcpassword= 6. try running ``bitcoin-cli help`` or ``bitcoin-cli -getinfo`` to check the connection to the daemon .. note:: If you want to run other Bitcoin-core based cryptocurrencies (for example Vertcoin, Dogecoin, etc...) it is sufficient to change the occurrencies of `bitcoin` in the service file. You also have to change the path of the files accordingly. .. note:: Instead of having to save all the transactions (>400GB at the time of writing) you can set pruning. This will save a lot space (about 100 times smaller). Just set ``prune=550`` in the configuration file. See also: - https://www.reddit.com/r/Bitcoin/comments/5ywru2/what_is_the_prune_mode_exactly/ - https://coinguides.org/bitcoin-blockchain-pruning/ References ~~~~~~~~~~ - https://github.com/bitcoin/bitcoin/ - https://raw.githubusercontent.com/bitcoin/bitcoin/v0.21.1/contrib/init/bitcoind.service Programming languages ~~~~~~~~~~~~~~~~~~~~~ - shell Dependencies ~~~~~~~~~~~~ +----------------------+---------------+------------------+ | Name | Binaries | Version | +======================+===============+==================+ | Bitcoin Core | - bitcoind | v0.21.1 | | | - bitcoin-cli | | +----------------------+---------------+------------------+ | GNU Bash | - bash | 5.0.3 | +----------------------+---------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ 1. replace ``rpcpassword`` in ``bitcoin.conf`` and in ``~/.bitcoin/bitcoin.conf`` with an appropriate one Licenses ~~~~~~~~ - MIT YAML data ~~~~~~~~~ :: <--YAML--> cryptod.bitcoin.service: category: misc running user: cryptocurrency configuration files: paths: - bitcoin.conf systemd unit files: paths: service: - cryptod.bitcoin.service ---- mine_coins.py ````````````` Purpose ~~~~~~~ Mine some cryptocurrencies Steps ~~~~~ 1. run common command 0 using ``miner`` as parameter 2. install a crypto miner such as cpuminer-opt: :: # apt-get install build-essential automake libssl-dev libcurl4-openssl-dev libjansson-dev libgmp-dev zlib1g-dev git $ git clone "https://github.com/JayDDee/cpuminer-opt" $ cd cpuminer-opt && ./build.sh # make install .. note:: We don't mine Dogecoins directly but other cryptocoins based on the lyra2z330 hashing algorithm. These coins will be exchanged with Dogecoins by the mining pool. References ~~~~~~~~~~ - https://www.reddit.com/r/dogecoin/comments/7oyn8b/currently_i_mine_2500_doge_a_day_using_1_cpu_and/ - https://www.reddit.com/r/dogecoin/comments/kq8j38/dogecoin_mining/ Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+---------------+------------------+ | Name | Binaries | Version | +======================+===============+==================+ | cpuminer-opt | - cpuminer | 3.16.3 | +----------------------+---------------+------------------+ | Python | - python3 | 3.7.3 | +----------------------+---------------+------------------+ | fpyutils | | 2.0.0 | +----------------------+---------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ 1. This example works for Dogecoin but you can mine any coin depending on your hardware and on the mining pool. Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> mine_coins.py: category: misc running user: miner configuration files: paths: - mine_coins.yaml systemd unit files: paths: service: - mine-coins.dogecoin.service ---- firefox_profile_runner.py ````````````````````````` Purpose ~~~~~~~ Use a GUI menu to launch multiple firefox profiles in firejail sandboxes at the same time Steps ~~~~~ 1. create new firefox profile for each of your purposes. Run :: firefox-esr --ProfileManager 2. run the script like this: :: ./firefox_profile_runner.py ./firefox_profile_runner.yaml .. note:: To have even more isolation use private home directories for the sandbox. This is achieved through the ``--private=`` argument in the firejail options. If you run the script with that option enabled Firefox will prompt you to create a new profile. The name you specify must correspond to the profile name in the configuration. You can also import an existing profile: :: cp -aR ~/.mozilla/firefox/${existing_firefox_profile} ${virtual_home}/.mozilla/firefox cp -a ~/.mozilla/firefox/profiles.ini ${virtual_home}/.mozilla/firefox Keep only the imported profile in ``${virtual_home}/.mozilla/firefox/profiles.ini`` and rename the INI key to ``Profile0``: :: [Profile0] Name=personal IsRelative=1 Path=${existing_firefox_profile} Programming languages ~~~~~~~~~~~~~~~~~~~~~ - Python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.9.2 | +----------------------+------------+------------------+ | fpyutils | | 2.1.0 | +----------------------+------------+------------------+ | Firejail | | 0.9.64.4 | +----------------------+------------+------------------+ | Firefox | | 78.14.0esr | +----------------------+------------+------------------+ | python-yad | | 0.9.11 | +----------------------+------------+------------------+ | YAD | | 0.40.0 | +----------------------+------------+------------------+ | PyYAML | | 5.3.1 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> firefox_profile_runner.py: category: misc running user: mydesktopuser configuration files: paths: - firefox_profile_runner.yaml ---- push_files.py ````````````` Purpose ~~~~~~~ Push selected files to all existing remotes of a repository. Examples ~~~~~~~~ The `fattura-elettronica-reader-assets-checksums `_ repository is handled by this script. Steps ~~~~~ .. note:: We will use git as VCS in this example. 1. create a git repository :: git init myrepo 2. add remotes using SSH URIs. You will need SSh keys to make this working :: git remote add origin myremote@my.domain:/myuser/myrepo.git git remote add secondary myremote@my.other.domain:/myuser/myrepo.git 3. create some initial commits and push then to the remotes Programming languages ~~~~~~~~~~~~~~~~~~~~~ - Python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.7.3 | +----------------------+------------+------------------+ | fpyutils | | 2.1.0 | +----------------------+------------+------------------+ | PyYAML | | 6.0 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> push_files.py: category: misc running user: myuser configuration files: paths: - push_files.mypurpose.yaml ---- System ------ hblock_unbound.py ````````````````` .. image:: assets/images/hblock_unbound.sh_0.png Purpose ~~~~~~~ I use this script to block malicious domains at a DNS level for the whole internal network. .. important:: We will assume that `Unbound `_ is configured and running. Steps ~~~~~ 1. separate Unbound's configuration into a header and footer file. Have a look at the provided configuration files. 2. clone the hblock repository: ``$ git clone https://github.com/hectorm/hblock.git`` 3. configure you hblock lists in the ``hblock_list.txt`` file References ~~~~~~~~~~ - https://github.com/hectorm/hblock Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Unbound | - unbound | 1.12.0 | +----------------------+------------+------------------+ | Git | - git | 2.20.1 | +----------------------+------------+------------------+ | hblock | - hblock | 3.2.3 | +----------------------+------------+------------------+ | GNU Make | - make | 4.2.1 | +----------------------+------------+------------------+ | fpyutils | | 1.2.3 | +----------------------+------------+------------------+ | Python | - python3 | 3.8.6 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ In case something goes wrong you can use this fallback command: :: # cat hblock_unbound.header.conf hblock_unbound.footer.conf > /etc/unbound/unbound.conf .. note:: The provided configuration files are designed to work along with `dnscrypt-proxy 2 `_ Licenses ~~~~~~~~ - MIT YAML data ~~~~~~~~~ :: <--YAML--> hblock_unbound.py: category: system running user: root configuration files: paths: - hblock_unbound.yaml - hblock_unbound_list.txt - hblock_unbound.footer.conf - hblock_unbound.header.conf - hblock_unbound.post_commands.conf systemd unit files: paths: service: - hblock-unbound.service timer: - hblock-unbound.timer ---- clean_pacman.py ``````````````` Purpose ~~~~~~~ I use this very simple script to clean the cache generated by `Pacman `_. References ~~~~~~~~~~ - https://wiki.archlinux.org/index.php/Pacman#Cleaning_the_package_cache - https://wiki.archlinux.org/index.php/Pacman/Tips_and_tricks#Removing_unused_packages_.28orphans.29 Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | GNU Bash | - bash | 5.1.004 | +----------------------+------------+------------------+ | pacman-contrib | - paccache | 1.4.0 | +----------------------+------------+------------------+ | Python | - python3 | 3.9.1 | +----------------------+------------+------------------+ | fpyutils | | 1.2.2 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GFDLv1.3+ YAML data ~~~~~~~~~ :: <--YAML--> clean_pacman.py: category: system running user: root configuration files: paths: - clean_pacman.yaml systemd unit files: paths: service: - clean-pacman.service timer: - clean-pacman.timer ---- iptables_geoport.py ``````````````````` Purpose ~~~~~~~ I use this script to block IP addresses by country for inbound ports on a server. Examples ~~~~~~~~ I use this script essentially to avoid bruteforce SSH attacks. However, since I use a remote scanner with SANE, some extra steps are required to make things work: 1. open tcp and udp ports 6566 2. ``# echo "options nf_conntrack nf_conntrack_helper=1" > /etc/modprobe.d/nf_conntrack.conf`` 3. ``# echo "nf_conntrack_sane" > /etc/modules-load.d/nf_conntrack_sane.conf`` 4. reboot 5. ``# cat /proc/sys/net/netfilter/nf_conntrack_helper`` should return ``1`` Steps ~~~~~ 1. run the script 2. make the rules persistent. For example, have a look at `this Arch wiki page `_ Debian support is already active in the service file. References ~~~~~~~~~~ - https://software.franco.net.eu.org/frnmst/iptables-geoport-directives - http://www.cyberciti.biz/faq/?p=3402 - https://wiki.archlinux.org/index.php/Simple_stateful_firewall - https://wiki.archlinux.org/index.php/Iptables - http://www.thegeekstuff.com/2011/06/iptables-rules-examples/ - https://home.regit.org/wp-content/uploads/2011/11/secure-conntrack-helpers.html Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.8.1 | +----------------------+------------+------------------+ | Requests | | 2.23.0 | +----------------------+------------+------------------+ | PyYAML | | 5.3 | +----------------------+------------+------------------+ | iptables | | 1:1.8.4 | +----------------------+------------+------------------+ | fpyutils | | 1.2.0 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ .. warning:: The ``patch rules`` directive contains a list of shell commands that are executed directly! It is your responsability to avoid putting malicious code there. Licenses ~~~~~~~~ - GPLv2+ - GFDLv1.3+ YAML data ~~~~~~~~~ :: <--YAML--> iptables_geoport.py: category: system running user: root configuration files: paths: - iptables_geoport.yaml systemd unit files: paths: service: - iptables-geoport.service timer: - iptables-geoport.timer ---- roothints ````````` Purpose ~~~~~~~ I use this service to update the list of servers, authoritative for the root domain. .. important:: We will assume that `Unbound `_ is configured and running. References ~~~~~~~~~~ - https://wiki.archlinux.org/index.php/Unbound#Root_hints - https://wiki.archlinux.org/index.php/Unbound#Roothints_systemd_timer - https://kb.isc.org/docs/aa-01309 Programming languages ~~~~~~~~~~~~~~~~~~~~~ - bash Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Unbound | - unbound | 1.10.0 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GFDLv1.3+ YAML data ~~~~~~~~~ :: <--YAML--> roothints: category: system running user: root systemd unit files: paths: service: - roothints.service timer: - roothints.timer ---- notify_unit_status.py ````````````````````` Purpose ~~~~~~~ I use this script to notify when a Systemd service fails. Examples ~~~~~~~~ My `Gitea `_ instance `could not start after an update `_. If I used this script I would have known immediately about the problem instead of several days later. Steps ~~~~~ 1. to monitor a service run ``# systemctl edit ${unit_name}`` 2. copy and save the following in the text editor :: [Unit] OnFailure=notify-unit-status@%n.service .. important:: It is assumed that you can send emails using `Msmtp `_ like this: 1. run common command 0 using ``email`` as parameter 2. make sure that the ``root`` user is able to connect to the ``email`` user using an SSH key 3. configure `Msmtp `_ as described in `this section `_ 4. configure email aliases in ``/etc/aliases`` References ~~~~~~~~~~ - https://serverfault.com/questions/694818/get-notification-when-systemd-monitored-service-enters-failed-state Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.8.4 | +----------------------+------------+------------------+ | fpyutils | | 1.2.3 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Systemd unit files ~~~~~~~~~~~~~~~~~~ The provided Systemd service unit file represents a `template `_. Licenses ~~~~~~~~ - CC-BY-SA 4.0 YAML data ~~~~~~~~~ :: <--YAML--> notify_unit_status.py: category: system running user: root configuration files: paths: - notify_unit_status.yaml systemd unit files: paths: service: - notify-unit-status@.service ---- command_assert.py ````````````````` .. image:: assets/images/command_assert.py_0.png Purpose ~~~~~~~ I use this script to check that the result of shell commands correspond to some expected output. The script also creates an RSS feed to complement the standard notifications. Examples ~~~~~~~~~ You can use this if you need to check if some websites or services are reachable. Steps ~~~~~ 1. run common command 0 using ``command-assert`` as parameter 2. optionally run common command 1 Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | Python | - python3 | 3.7.3 | +----------------------+------------+------------------+ | PyYAML | | 6.0 | +----------------------+------------+------------------+ | fpyutils | | 2.1.0 | +----------------------+------------+------------------+ | feedgenerator | | 2.0.0 | +----------------------+------------+------------------+ Licenses ~~~~~~~~ - GPLv3+ - CC-BY-SA 4.0 YAML data ~~~~~~~~~ :: <--YAML--> command_assert.py: category: system running user: command-assert configuration files: paths: - command_assert.py - command_assert.mypurpose.yaml systemd unit files: paths: service: - command-assert.mypurpose.service timer: - command-assert.mypurpose.timer ---- qvm.py `````` Purpose ~~~~~~~ I use this script to run virtual machines via QEMU and to connect to them through other clients. Steps ~~~~~ 1. create a new virtual hard disk: .. code-block:: bash qemu-img create -f qcow2 development.qcow2 64G 2. modify the configuration to point to ``development.qcow2``, the virtual hdd. 3. run the installation having the cdrom set to ``true``, and corresponding to the ISO installation medium: .. code-block:: bash ./qvm ./qvm.py local development 4. create a backup virtual hard disk: .. code-block:: bash qemu-img create -f qcow2 -b development.qcow2 development.qcow2.mod 5. run the installed machine using ``development.qcow2.mod`` as virtual hard disk 6. if needed, modify iptables to let data through the shared ports 7. on the guest system create a ``powermanager`` user and add ``powermanager ALL=(ALL) NOPASSWD:/sbin/poweroff`` using ``visudo``. 8. on the host system create an SSH key so that the ``qvm`` host user can connect to the ``powermanager`` guest user. Have a look at the ``ExecStop`` command in the service unit file. 9. in the host machine, configure the ssh config file (``~/.ssh/config``) like this: .. code-block:: ini # See https://superuser.com/a/870918 # CC BY-SA 3.0 # (C) Kenster, 2015 Match host 127.0.0.1 user powermanager exec "test %p = 2222" IdentitiesOnly yes IdentityFile ~/.ssh/powermanager_test Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+-----------------------+------------------+ | Name | Binaries | Version | +======================+=======================+==================+ | GNU Bash | - bash | 5.1.004 | +----------------------+-----------------------+------------------+ | Python | - python3 | 3.9.1 | +----------------------+-----------------------+------------------+ | fpyutils | | 1.2.2 | +----------------------+-----------------------+------------------+ | QEMU | - qemu-system-x86_64 | 5.2.0 | +----------------------+-----------------------+------------------+ | OpenSSH | - ssh | 8.4p1 | +----------------------+-----------------------+------------------+ | TigerVNC | - vncviewer | 1.11.0 | +----------------------+-----------------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+-----------------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ - if you use a QCOW2 disk you can run the script with an unprivileged user. - you can run directly from phisical partitions. Permissions depend from the filesystem. Use this snippet as reference. .. warning:: Remember NOT to mount the partitions while running. Data loss may occur in that case. :: # Mass memory. Use device name with options. drives: - '/dev/sda,format=raw' - '/dev/sdb,format=raw' - if you need to share a directory you can use a 9p filesystem if the guest kernel supports it. Run ``modprobe 9pnet_virtio`` as ``root``: this is the case for Debian's ``linux-image-cloud-amd64`` package where 9p is not supported: :: modprobe: FATAL: Module 9pnet_virtio not found in directory /lib/modules/5.10.0-9-cloud-amd64 If you kernel has the ``9pnet_virtio`` module you can add a shared directory. This might be an extract of ``qvm.yaml``: :: mount: enabled: true mountpoints: - shared: path: '/home/qvm/shares/development/shared' mount tag: 'shared' In this case an ``/etc/fstab`` entry might look like this: :: shared /home/vm/shared 9p auto,access=any,x-systemd.automount,msize=268435456,trans=virtio,version=9p2000.L 0 0 Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> qvm.py: category: system running user: qvm configuration files: paths: - qvm.yaml systemd unit files: paths: service: - qvm.local_test.service ---- update_action.py ```````````````` Purpose ~~~~~~~ I use this script to update some software not supported by the package manager, for example Docker images. Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+-----------------------+------------------+ | Name | Binaries | Version | +======================+=======================+==================+ | GNU Bash | - bash | 5.0.3 | +----------------------+-----------------------+------------------+ | Python | - python3 | 3.7.3 | +----------------------+-----------------------+------------------+ | fpyutils | | 1.2.2 | +----------------------+-----------------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+-----------------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ .. warning:: No filtering is performed for the configuration file. It is your responsability for its content. Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> update_action.py: category: system running user: root configuration files: paths: - update_action.mypurpose.yaml systemd unit files: paths: service: - update-action.mypurpose.service timer: - update-action.mypurpose.timer ---- Video ----- record_motion.py ```````````````` .. image:: assets/images/record_motion.sh_0.png Purpose ~~~~~~~ I use this script to record video streams captured by webcams with `Motion `_. .. important:: We will assume that Motion is already configured and running. Steps ~~~~~ 1. make sure to have a *big enough* hard drive References ~~~~~~~~~~ Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+------------+------------------+ | Name | Binaries | Version | +======================+============+==================+ | GNU Bash | - bash | 5.1.0 | +----------------------+------------+------------------+ | FFmpeg | - ffmpeg | 2:4.3.1 | +----------------------+------------+------------------+ | fpyutils | | 1.2.2 | +----------------------+------------+------------------+ | Python | - python3 | 3.9.1 | +----------------------+------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ You can use hardware acceleration instead of using software for the encoding process. Using hardware acceleration should reduce the load on the processor: "Hardware encoders typically generate output of significantly lower quality than good software encoders like x264, but are generally faster and do not use much CPU resource. (That is, they require a higher bitrate to make output with the same perceptual quality, or they make output with a lower perceptual quality at the same bitrate.)" -- `HWAccelIntro `_ page Since we are dealing with video surveillance footage we don't care about quality so much. In the configuration file you will find an example for `Intel VAAPI `_. In this case you need to set ``use global quality`` to ``true`` and use the ``global quality`` variable instead. You can adapt the script and/or the configuration to work for other types of hardware acceleration. See also the `Arch Wiki page `_. Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> record_motion.py: category: video running user: surveillance configuration files: paths: - record_motion.camera1.yaml systemd unit files: paths: service: - record-motion.camera1.service ---- convert_videos.py ````````````````` Purpose ~~~~~~~ I use this script to capture, encode and transcode videos from different hardware sources. Steps ~~~~~ 1. make sure to have a *big enough* hard drive: encoding requires a lot of space 2. `follow this tutorial `_ 3. if you are going to use multiple devices you must be able to identify them: - in case of v4l devices you can use ``$ ls -l /dev/v4l/by-path/`` - in case of DVD devices, you can use ``$ ls -l /dev/disk/by-path`` in combination with ``$ eject`` - in case of ALSA devices you can follow `this tutorial `_ to get persistent naming 4. I strongly suggest installing something like `Ananicy `_ which automatically sets functional priority levels for processes like the ones run by ffmpeg which is heaviliy used in this script. 5. have a look at ``$ ./convert_videos.py --help``. You can add descriptions as embedded subtitles using the ``--description`` option. Examples ~~~~~~~~ My purpose is to digitize VHS cassettes and DVDs. For VHSs I use `this easycap device from CSL `_ which uses the ``stk1160`` kernel module and a proper VCR. Have a look at `this LinuxTVWiki wiki page `_. For DVDs I use a standard 5.25'' SATA DVD drive. When everything is set I start to encode a video. Transcoding is done on a different computer, a server, because its processor has a couple of extra cores and it is much more recent. References ~~~~~~~~~~ - https://linuxtv.org/wiki/index.php/V4L_capturing - https://linuxtv.org/wiki/index.php/V4L_capturing/script - https://linuxtv.org/wiki/index.php/Easycap - https://linuxtv.org/wiki/index.php/Stk1160_based_USB_2.0_video_and_audio_capture_devices - https://alsa.opensrc.org/Udev#Identify_two_identical_audio_devices Programming languages ~~~~~~~~~~~~~~~~~~~~~ - python Dependencies ~~~~~~~~~~~~ +----------------------+-------------------+------------------+ | Name | Binaries | Version | +======================+===================+==================+ | GNU Bash | - bash | 5.0.017 | +----------------------+-------------------+------------------+ | FFmpeg | - ffmpeg | 1:4.2.3 | +----------------------+-------------------+------------------+ | PyYAML | | 5.3.1 | +----------------------+-------------------+------------------+ | HandBrake CLI | - HandBrakeCLI | 1.3.0 | +----------------------+-------------------+------------------+ | libdvdcss | | 1.4.2 | +----------------------+-------------------+------------------+ | libdvdnav | | 6.1.0 | +----------------------+-------------------+------------------+ | VLC media player | - cvlc | 3.0.10 | +----------------------+-------------------+------------------+ | v4l-utils | - v4l-ctl | 1.18.1 | +----------------------+-------------------+------------------+ | gst-plugins-bad | | 1.16.2 | +----------------------+-------------------+------------------+ | gst-plugins-base | | 1.16.2 | +----------------------+-------------------+------------------+ | gst-plugins-good | | 1.16.2 | +----------------------+-------------------+------------------+ | gst-plugins-ugly | | 1.16.2 | +----------------------+-------------------+------------------+ | GStreamer | - gst-launch-1.0 | 1.16.2 | +----------------------+-------------------+------------------+ | Python | - python3 | 3.8.3 | +----------------------+-------------------+------------------+ | fpyutils | | 1.2.0 | +----------------------+-------------------+------------------+ | PyYAML | | 5.4.1 | +----------------------+-------------------+------------------+ Configuration files ~~~~~~~~~~~~~~~~~~~ The configuration file is designed so that you can you can reuse different parts of it for different sources and actions. .. important:: the default transcoding options are set up to get the best quality possible. The order of magnitude I get is 24 hours of transcoding time for 1 hour of encoded video (at full system load). If you feel that is too much you can change the preset to ``slow`` or ``medium``. .. warning:: To simplify the development, shell commands are executed directly! It is your responsability to avoid putting malicious code. Licenses ~~~~~~~~ - GPLv3+ YAML data ~~~~~~~~~ :: <--YAML--> record_motion.sh: category: video running user: myuser configuration files: paths: - convert_videos.yaml systemd unit files: paths: service: - convert-videos.samsung.service