Package server#

Mirroring#

Debmirror#

Debmirror is one of the existing programs which are able to mirror Debian APT packages. In this example we are going to use the Apache HTTP server as webserver.

In this example we will use the Apache HTTP server to serve the packages.

See also

  • Debmirror: creiamo un mirror Debian - Guide@Debianizzati.Org 1

  • Mirantis Documentation: Usage ~ DEBMIRROR 2

  • Apache Tips & Tricks: Hide a file type from directory indexes · MD/Blog 3

  • Set Up A Local Ubuntu / Debian Mirror with Apt-Mirror | Programster’s Blog 4

  • Debian – Mirror Size 5

  • Comment #4 : Bug #882941 : Bugs : debmirror package : Ubuntu 6

  • Debian – Setting up a Debian archive mirror 7

  • Better Default Directory Views with HTAccess | Perishable Press 8

  1. install

    apt-get install debmirror bindfs

  2. create a new user

    useradd -m -s /bin/bash -U debmirror
passwd debmirror
usermod -aG jobs debmirror
mkdir /home/debmirror/data
chown debmirror:debmirror /home/debmirror/data

    Note

    In this example /home/debmirror/data is the base directory where all packages are served from.

    Tip

    I suggest using normal HDDs rather than SSDs. Size in more important than speed in this case.

  3. create the jobs directories

    mkdir -p /home/jobs/{services,scripts}/by-user/debmirror

  4. load APT’s keyring into a new keyring owned by the debmirror user

    sudo -i -u debmirror
gpg --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export \
    | gpg --no-default-keyring --keyring trustedkeys.gpg --import
exit

  5. add this configuration for the standard Debian repository

    /home/jobs/scripts/by-user/debmirror/debmirror.debian.conf#
    # The config file is a perl script so take care to follow perl syntax.
# Any setting in /etc/debmirror.conf overrides these defaults and
# ~/.debmirror.conf overrides those again. Take only what you need.
#
# The syntax is the same as on the command line and variable names
# loosely match option names. If you don't recognize something here
# then just stick to the command line.
#
# Options specified on the command line override settings in the config
# files.

# Location of the local mirror (use with care)
# $mirrordir="/path/to/mirrordir"

# Output options
$verbose=1;
$progress=1;
$debug=0;

$remoteroot="debian";
# Download options
$host="debian.netcologne.de";
$download_method="rsync";
@dists="stable,oldstable,buster-updates,bullseye-updates,buster-backports,bullseye-backports";
@sections="main";
@arches="amd64,all,any";
$omit_suite_symlinks=0;
$skippackages=0;
# @rsync_extra="none";
$i18n=1;
$getcontents=1;
$do_source=1;
$max_batch=0;

# Includes other translations as well.
# See the exclude option in
# https://help.ubuntu.com/community/Debmirror
@includes="Translation-(en|it).*";

# @di_dists="dists";
# @di_archs="arches";

# Save mirror state between runs; value sets validity of cache in days
$state_cache_days=0;

# Security/Sanity options
$ignore_release_gpg=0;
$ignore_release=0;
$check_md5sums=0;
$ignore_small_errors=1;

# Cleanup
$cleanup=0;
$post_cleanup=1;

# Locking options
$timeout=300;

# Rsync options
$rsync_batch=200;
$rsync_options="-aIL --partial --bwlimit=10240";

# FTP/HTTP options
$passive=0;
# $proxy="http://proxy:port/";

# Dry run
$dry_run=0;

# Don't keep diff files but use them
$diff_mode="use";

# The config file must return true or perl complains.
# Always copy this.
1;

  6. create the Systemd service unit file

    /home/jobs/services/by-user/debmirror/debmirror.debian.service#
    [Unit]
Description=Debmirror debian
Requires=network-online.target
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/debmirror --config-file=/home/jobs/scripts/by-user/debmirror/debmirror.debian.conf /home/debmirror/data/debian
User=debmirror
Group=debmirror

  7. create the Systemd service timer unit file

    /home/jobs/services/by-user/debmirror/debmirror.debian.timer#
    [Unit]
Description=Once a day debmirror debian

[Timer]
OnCalendar=*-*-* 1:30:00
Persistent=true

[Install]
WantedBy=timers.target

  8. create a directory readable be Apache

    mkdir -p /srv/http/debian
chown www-data:www-data /srv/http/debian
chmod 700 /srv/http/debian

  9. Add this to the fstab file

    /etc/fstab#
    /home/debmirror/data /srv/http/debian fuse.bindfs  auto,force-user=www-data,force-group=www-data,ro 0 0

    Note

    This mount command makes the directory exposed to the webserver readonly, in this case /srv/http/debian

  10. serve the files via HTTP by creating a new Apache virtual host. Replace FQDN with the appropriate domain and include this file from the Apache configuration

    /etc/apache2/debian_mirror.apache.conf#
    <IfModule mod_ssl.c>
<VirtualHost *:443>

    UseCanonicalName on

    Keepalive On
    RewriteEngine on

    ServerName ${FQDN}

    # Set the icons also to avoid 404 errors.
    Alias /icons/ "/usr/share/apache2/icons/"

    DocumentRoot "/srv/http/debian"
    <Directory "/srv/http/debian">
        Options -ExecCGI -Includes
        Options +Indexes +SymlinksIfOwnerMatch
        IndexOptions NameWidth=* +SuppressDescription FancyIndexing Charset=UTF-8 VersionSort FoldersFirst

        ReadmeName footer.html
        IndexIgnore header.html footer.html
        #
        # AllowOverride controls what directives may be placed in .htaccess files.
        # It can be "All", "None", or any combination of the keywords:
        #   AllowOverride FileInfo AuthConfig Limit
        #
        AllowOverride All

        #
        # Controls who can get stuff from this server.
        #
        Require all granted
    </Directory>

    SSLCompression      off

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/${FQDN}/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/${FQDN}/privkey.pem
</VirtualHost>
</IfModule>

  11. create a new text file that will serve as basic instructions for configuring the APT sources file. Replace FQDN with the appropriate domain

    /home/debmirror/data/footer.html#
    <h1>Examples</h1>

Change <code>/etc/apt/sources.list</code> to one of these:

<h2>Bullseye distribution</h2>

<pre>
deb  https://${FQDN}/debian          bullseye            main
deb  https://${FQDN}/debian          bullseye-updates    main
deb-src https://${FQDN}/debian-security bullseye-security    main
deb  https://${FQDN}/debian           bullseye-backports  main

deb [arch=amd64] https://${FQDN}/docker bullseye stable
deb [arch=amd64] https://${FQDN}/gitea gitea main
deb [arch=amd64] https://${FQDN}/postgresql bullseye-pgdg main
</pre>

<h2>Buster distribution</h2>

<pre>
deb  https://${FQDN}/debian          buster            main
deb  https://${FQDN}/debian          buster-updates    main
deb-src https://${FQDN}/debian-security buster/updates    main
deb  https://${FQDN}/debian           buster-backports  main

deb [arch=amd64] https://${FQDN}/docker buster stable
deb [arch=amd64] https://${FQDN}/gitea gitea main
deb [arch=amd64] https://${FQDN}/postgresql buster-pgdg main
</pre>

<h1>Repositories</h1>

<code>stable</code> and <code>oldstable</code> distributions are available if applicable.

<h2>debian</h2>
<p>Supported architectures:</p>
<ul>
<li><code>amd64</code></li>
<li><code>all</code></li>
<li><code>any</code></li>
</ul>

<h2>debian-security</h2>
<p>Supported architectures:</p>
<ul>
<li><code>amd64</code></li>
<li><code>all</code></li>
<li><code>any</code></li>
</ul>

<h2>docker</h2>
<p>Supported architectures:</p>
<ul>
<li><code>amd64</code></li>
</ul>

<h2>gitea</h2>
<p>Supported architectures:</p>
<ul>
<li><code>amd64</code></li>
</ul>

<h2>postgresql</h2>
<p>Supported architectures:</p>
<ul>
<li><code>amd64</code></li>
</ul>

    Note

    This example includes some unofficial repositories.

  12. restart the Apache webserver

    systemctl restart apache2

  13. fix the permissions

    chown -R debmirror:debmirror /home/jobs/{services,scripts}/by-user/debmirror
chmod 700 -R /home/jobs/{services,scripts}/by-user/debmirror

  14. run the deploy script

Unofficial Debian sources#

In case you want to mirror unofficial Debian sources the same instrctions apply. You just need to change the key import step

sudo -i -u debmirror
gpg \
    --no-default-keyring \
    --keyring trustedkeys.gpg \
    --import ${package_signing_key}
exit

Note

package_signing_key is provided by the repository maintainers.

PyPI server#

Build Python packages using git sources and push them to a self-hosted PyPI server.

Server#

See also

  • Minimal PyPI server for uploading & downloading packages with pip/easy_install Resources 9

  1. follow the Docker instructions

  2. create the jobs directories

    mkdir -p /home/jobs/scripts/by-user/root/docker/pypiserver
chmod 700 /home/jobs/scripts/by-user/root/docker/pypiserver

  3. install and run pypiserver. Use this Docker compose file

    /home/jobs/scripts/by-user/root/docker/pypiserver/docker-compose.yml#
    version: '3.7'

services:
    pypiserver-authenticated:
        image: pypiserver/pypiserver:latest
        volumes:
            # Authentication file.
            - type: bind
              source: /home/jobs/scripts/by-user/root/docker/pypiserver/auth
              target: /data/auth

            # Python files.
            - type: bind
              source: /data/pypiserver/packages
              target: /data/packages
        ports:
            - "127.0.0.1:4000:8080"

        # I have purposefully removed the
        #    --fallback-url https://pypi.org/simple/
        # option to have a fully isolated environment.
        command: --disable-fallback --passwords /data/auth/.htpasswd --authenticate update /data/packages

  4. create a Systemd unit file. See also the Docker compose services section

    /home/jobs/services/by-user/root/docker-compose.pypiserver.service#
    [Unit]
Requires=docker.service
Requires=network-online.target
After=docker.service
After=network-online.target

[Service]
Type=simple
WorkingDirectory=/home/jobs/scripts/by-user/root/docker/pypiserver

ExecStart=/usr/bin/docker-compose up --remove-orphans
ExecStop=/usr/bin/docker-compose down --remove-orphans

Restart=always

[Install]
WantedBy=multi-user.target

  5. fix the permissions

    chmod 700 /home/jobs/scripts/by-user/root/docker/pypiserver
chmod 700 -R /home/jobs/services/by-user/root

  6. run the deploy script

  7. modify the reverse proxy port of your webserver configuration with 4000

Apache configuration#

If you use Apache as webserver you should enable caching. The upstream documentation shows how to configure pypiserver for Nginx but not for Apache.

See also

  • GitHub - pypiserver/pypiserver: Minimal PyPI server for uploading & downloading packages with pip/easy_install - Serving Thousands of Packages 10

  • Caching Guide - Apache HTTP Server Version 2.4 11

  • mod_cache - Apache HTTP Server Version 2.4 12

  1. create a new Apache virtual host. Replace FQDN with the appropriate domain

    /etc/apache2/pypi_server.apache.conf#
    ###############
# pypiserver  #
###############
<IfModule mod_ssl.c>
<VirtualHost *:443>
    UseCanonicalName on

    Keepalive On
    RewriteEngine on

    ServerName ${FQDN}

    SSLCompression      off

    RewriteRule    ^/simple$  /simple/  [R]
    ProxyPass      / http://127.0.0.1:4000/ Keepalive=On max=50 timeout=300 connectiontimeout=10
    ProxyPassReverse   / http://127.0.0.1:4000/
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Port "443"
    RequestHeader set X-Forwarded-Host "${FQDN}"

    Header set Service "pypi"

    CacheRoot "/var/cache/apache"
    CacheEnable disk /
    CacheDirLevels 4
    CacheDirLength 1

    CacheDefaultExpire 3600
    CacheIgnoreNoLastMod On
    CacheIgnoreCacheControl On
    CacheMaxFileSize 640000
    CacheReadSize 1024
    CacheIgnoreNoLastMod On
    CacheIgnoreQueryString On
    CacheIgnoreHeaders X-Forwarded-Proto X-Forwarded-For X-Forwarded-Host

    # Debug. Turn these two variables off after testing.
    CacheHeader on
    CacheDetailHeader On

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/${FQDN}/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/${FQDN}/privkey.pem
</VirtualHost>
</IfModule>

    Warning

    The included Cache* options are very aggressive!

  2. create the cache directory

    mkdir /var/cache/apache
chown www-data:www-data /var/cache/apache
chmod 775 /var/cache/apache

  3. enable the Apache modules

    a2enmod cache cache_disk
systemctl start apache-htcacheclean.service
systemctl restart apache2

  4. check for a cache hit. Replace FQDN with the appropriate domain

    curl -s https://${FQDN} 1>/dev/null 2>/dev/null
curl -s -D - https://${FQDN} | head -n 20

    Note

    The /packages/ page does not get cached.

  5. set CacheHeader and CacheDetailHeader to Off

  6. restart Apache

    systemctl restart apache2

Virtual machine compiling the packages#

I suggest using a virtual machine to compile packages to improve isolation and security: arbitrary code might be executed when compiling a package.

See also

  • A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks 13

  • Git repository pointers and configurations to build Python packages from source 14

  • python - pushd through os.system - Stack Overflow 15

  • Pass options to `build_ext · Issue #328 · pypa/build · GitHub` 16

  1. create a virtual machine with Debian Bullseye (stable) and transform it into Sid (unstable). Using the unstable version will provide more up to date software for development.

    See the QEMU server section. You might need to assign a lost of disk space.

  2. connect to the virtual machine. See the QEMU client section

  3. create a new user

    sudo -i
useradd --system -s /bin/bash -U python-source-package-updater
passwd python-source-package-updater
usermod -aG jobs python-source-package-updater

  4. create the jobs directories. See reference

    mkdir -p /home/jobs/{scripts,services}/by-user/python-source-package-updater
chown -R python-source-package-updater:python-source-package-updater /home/jobs/{scripts,services}/by-user/python-source-package-updater
chmod 700 -R /home/jobs/{scripts,services}/by-user/python-source-package-updater

  5. install these packages in the virtual machine:

    apt-get install build-essential fakeroot devscripts git python3-dev python3-all-dev \
    games-python3-dev libgmp-dev libssl-dev libssl1.1=1.1.1k-1 libcurl4-openssl-dev \
    python3-pip python3-build twine libffi-dev graphviz libgraphviz-dev pkg-config \
    clang-tools libblas-dev astro-all libblas-dev libatlas-base-dev libopenblas-dev \
    libgsl-dev libblis-dev liblapack-dev liblapack3 libgslcblas0 libopenblas-base \
    libatlas3-base libblas3 clang-9 clang-13 clang-12 clang-11 sphinx-doc \
    libbliss-dev libblis-dev libbliss2 libblis64-serial-dev libblis64-pthread-dev \
    libblis64-openmp-dev libblis64-3-serial libblis64-dev libblis64-3-pthread \
    libblis64-3-openmp libblis64-3 libblis3-serial libblis3-pthread \
    libblis-serial-dev libblis-pthread-dev libargon2-dev libargon2-0 libargon2-1

    Note

    This is just a selection. Some Python packages need other dependencies not listed here.

  6. install the dependencies of the script

    apt-get install python3-yaml python3-fpyutils
pip3 install --user platformdirs

  7. install fpyutils. See reference

  8. add the script

    /home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.py#
    #!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# build_python_packages.py
#
# Copyright (C) 2023 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.
#

import logging
import os
import pathlib
import shlex
import signal
import subprocess
import sys

import fpyutils
import platformdirs
import yaml


def _setup_logging() -> logging.Logger:
    # See
    # https://python3docs.franco.net.eu.org/howto/logging.html#logging-advanced-tutorial
    logger = logging.getLogger('build_python_packages.py')
    logger.setLevel(logging.DEBUG)

    # Console logging
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)

    formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    ch.setFormatter(formatter)

    logger.addHandler(ch)

    return logger


def _read_yaml_file(yaml_file: str) -> dict:
    data: dict = dict()
    if pathlib.Path(yaml_file).is_file():
        data = yaml.load(open(yaml_file, 'r'), Loader=yaml.SafeLoader)

    return data


def _write_yaml_file(data: dict, yaml_file: str) -> dict:
    with open(yaml_file, 'w') as f:
        f.write(yaml.dump(data))


# This class represents a single submodule.
class SubmoduleConfiguration:

    def __init__(self,
                 path: str,
                 skip_repository: bool = False,
                 mark_skip_repository_successfull: bool = False,
                 mark_failed_build_or_upload_successfull: bool = False,
                 relative_base_directory_override: str = str(),
                 override_commands: dict = dict(),
                 ref_checkout: list = list()):
        r"""Create a SubmoduleConfiguration."""
        self.path: str = shlex.quote(path)
        self.skip_repository: bool = skip_repository

        # Common to all SubmoduleConfiguration instances.
        self.mark_skip_repository_successfull = mark_skip_repository_successfull
        self.mark_failed_build_or_upload_successfull = mark_failed_build_or_upload_successfull

        # Remote configuration
        self.relative_base_directory_override: str = relative_base_directory_override
        self.override_commands: dict = override_commands
        self.ref_checkout: list = ref_checkout

    def _execute_override_commands(self, command_type: str = 'pre'):
        cmd: list
        if (isinstance(self.override_commands, dict)
                and self.override_commands != dict()):
            # Command type must be {pre,build,post}
            for block in self.override_commands[command_type]:
                cmd = list()
                for c in self.override_commands[command_type][block]:
                    cmd.append(shlex.quote(c))
                try:
                    subprocess.run(cmd, check=True)
                    logger.info('override command executed correctly')
                except subprocess.CalledProcessError as e:
                    # Print but do not abort the program.
                    logger.warning(e)
                    logger.info('error executing override command')


class Cache:

    def __init__(self):
        r"""Create the Cache structure."""
        self.path: str = str()

        # cache[base_path] = [tag_0, tag_1, ..., tag_n]
        self.cache: dict = dict()

    def _init(self):
        platformdir: platformdirs.AppDirs = platformdirs.AppDirs(
            'build_python_packages')
        platformdir.user_cache_path.mkdir(mode=0o700,
                                          exist_ok=True,
                                          parents=True)
        self.path = pathlib.Path(platformdir.user_cache_dir, 'cache.yml')

    def _read(self):
        self.cache = _read_yaml_file(self.path)
        logger.info('cache read')

    def _write(self):
        _write_yaml_file(self.cache, self.path)
        logger.info('cache written')

    def _update(self, package_path: str, tag: str):
        if tag != str():
            if package_path not in self.cache:
                self.cache[package_path]: list = list()
            self.cache[package_path].append(tag)
            logger.info('cache updated')
        else:
            logger.info('cache not updated because of tagless repository')


class Executables:

    def __init__(self,
                 git: str = 'git',
                 python: str = 'python3',
                 twine: str = 'twine',
                 rm: str = 'rm'):
        r"""Save the paths of all necessary executables."""
        self.git: str = shlex.quote(git)
        self.python: str = shlex.quote(python)
        self.twine: str = shlex.quote(twine)
        self.rm: str = shlex.quote(rm)


class PypiCredentials:

    def __init__(self, url: str, user: str, password: str):
        r"""Save the PyPI credentials to be used for a mirror."""
        self.url = shlex.quote(url)
        self.user = user
        self.password = password


class GitRepository:

    def __init__(self, path: str, executables: Executables):
        r"""Initialize a generic empty GIT repository."""
        self.path: str = shlex.quote(path)
        self.executables: str = executables

    def _get_tags(self) -> list:
        s = subprocess.run([self.executables.git, '-C', self.path, 'tag'],
                           check=True,
                           capture_output=True)
        logger.info('obtained git tags')
        return s.stdout.decode('UTF-8').rstrip().split('\n')

    def _remove_untracked_files(self):
        fpyutils.shell.execute_command_live_output(self.executables.git +
                                                   ' -C ' + self.path +
                                                   ' checkout --force --')
        fpyutils.shell.execute_command_live_output(self.executables.git +
                                                   ' -C ' + self.path +
                                                   ' clean -d -x --force')

    def _get_last_ref_timestamp(self) -> str:
        return subprocess.run(
            [
                self.executables.git, '-C', self.path, 'log', '-1',
                '--pretty=%ct'
            ],
            check=True,
            capture_output=True).stdout.decode('UTF-8').strip()

    def _tag_checkout(self, tag: str):
        # Checkout repository with tags: avoid checking out tagless repositories.
        if tag != str():
            fpyutils.shell.execute_command_live_output(
                shlex.quote(self.executables.git) + ' -C ' + self.path +
                ' checkout ' + tag)


class Dist:

    def __init__(self, path: str, executables: Executables):
        r"""Initialize a Dist which is a subset of a package."""
        self.path = shlex.quote(path)
        self.executables = executables

    def _build(self, git_repository_timestamp: str):
        git_repository_timestamp = shlex.quote(git_repository_timestamp)
        r"""Build the Python package in a reproducable way.

        Remove all dev, pre-releases, etc information from the package name.
        Use a static timestamp.
        See
        https://github.com/pypa/build/issues/328#issuecomment-877028239
        """
        subprocess.run([
            self.executables.python, '-m', 'build', '--sdist', '--wheel',
            '-C--build-option=egg_info', '-C--build-option=--no-date',
            '-C--build-option=--tag-build=', self.path
        ],
                       check=True,
                       env=dict(os.environ,
                                SOURCE_DATE_EPOCH=git_repository_timestamp))

    def _upload(self, pypi_credentials: PypiCredentials):
        r"""Push the compiled package to a remote PyPI server."""
        subprocess.run([
            self.executables.twine, 'upload', '--repository-url',
            pypi_credentials.url, '--non-interactive', '--skip-existing',
            str(pathlib.Path(self.path, 'dist/*'))
        ],
                       check=True,
                       env=dict(os.environ,
                                TWINE_PASSWORD=pypi_credentials.password,
                                TWINE_USERNAME=pypi_credentials.user))


class Package:

    def __init__(self, path: str, tag: str, repo: GitRepository,
                 submodule_configuration: SubmoduleConfiguration, cache: Cache,
                 executables: Executables):
        r"""Initialize a Package which is a subset of a worker."""
        self.path: str = shlex.quote(path)
        # Do not qute tag: str() == '', shlex.quote(str()) == "''"
        self.tag = tag
        self.repo = repo
        self.submodule_configuration: SubmoduleConfiguration = submodule_configuration
        self.cache: Cache = cache
        self.executables = executables

        self.dist: Dist = Dist(self.path, self.executables)

    def _clean(self):
        fpyutils.shell.execute_command_live_output(
            self.executables.rm + ' -rf ' +
            fpyutils.path.add_trailing_slash(self.path) + 'build ' +
            fpyutils.path.add_trailing_slash(self.path) + 'dist')

    def _work(self, pypi_credentials: PypiCredentials) -> bool:
        # Retuns True if build and upload are successfull
        # False otherwise
        update_cache: bool = False
        successfull: bool = False

        self.repo._remove_untracked_files()
        self._clean()

        # Do not checkout empty tags
        if self.tag != str():
            self.repo._tag_checkout(self.tag)

        try:
            self.submodule_configuration._execute_override_commands('pre')

            # Replace build command if necessary.
            if self.submodule_configuration.override_commands != dict():
                self.submodule_configuration._execute_override_commands(
                    'build')
            else:
                self.dist._build(self.repo._get_last_ref_timestamp())

            # Post
            self.submodule_configuration._execute_override_commands('post')

            self.dist._upload(pypi_credentials)

            if self.tag != str():
                update_cache = True
            successfull = True
            logger.info('package build successfully')
        except subprocess.CalledProcessError:
            logger.info('error building package')
            if (self.submodule_configuration.
                    mark_failed_build_or_upload_successfull
                    and self.tag != str):
                successfull = True
                update_cache = True

        if update_cache:
            self.cache._update(pathlib.Path(self.path).stem, self.tag)

        self.repo._remove_untracked_files()

        return successfull


class RepositoryWorker:

    def __init__(self, path: str,
                 submodule_configuration: SubmoduleConfiguration,
                 executables: Executables):
        r"""Initialize a worker which corresponds to a GIT submodule."""
        # Working directory full path of Python code.
        self.path = shlex.quote(path)

        self.submodule_configuration: SubmoduleConfiguration = submodule_configuration
        self.executables: Executables = executables
        self.successfull_tags: int = 0
        self.total_tags: int = 0

    def _work(self, cache: Cache, pypi_credentials: PypiCredentials):
        repo: GitRepository = GitRepository(self.path, self.executables)
        tags: list = repo._get_tags()
        self.total_tags: int = len(tags)
        for i, tag in enumerate(tags):
            logger.info('processing git tag ' + str(i + 1) + ' of ' +
                        str(len(tags)))

            if self.submodule_configuration.skip_repository:
                logger.info('git tag ' + str(i + 1) + ' is skipped')
                if self.submodule_configuration.mark_skip_repository_successfull:
                    self.successfull_tags += 1
                    logger.info('marking skipped git tag ' + str(i + 1) +
                                ' as successfull')
                    cache._update(pathlib.Path(self.path).stem, tag)

            elif (pathlib.Path(self.path).stem in cache.cache
                  and tag in cache.cache[pathlib.Path(self.path).stem]):
                self.successfull_tags += 1
                logger.info('git tag ' + str(i + 1) + ' already in cache')

            else:
                p = Package(
                    path=self.path,
                    tag=tag,
                    repo=repo,
                    submodule_configuration=self.submodule_configuration,
                    cache=cache,
                    executables=self.executables)
                self.successfull_tags += int(p._work(pypi_credentials))


class GitParentRepository(GitRepository):

    def __init__(self, path: str, remote: str, checkout_branch: str,
                 local_sumodules_configuration: dict, cache: Cache,
                 executables: Executables):
        r"""Initialize the main repository."""
        super().__init__(path, executables)

        # usually set to 'origin'
        self.remote: str = shlex.quote(remote)
        self.checkout_branch: str = shlex.quote(checkout_branch)

        # self.configuration = SubmoduleConfiguration.all()
        self.submodules_configuration: dict = dict()

        self.submodules: list = list()
        self.local_sumodules_configuration = local_sumodules_configuration
        self.cache: Cache = cache

        self.total_successfull_tags: int = 0
        self.total_tags: int = 0

    def _get_updates(self):
        logger.info(
            'pulling parent repository changes, this might take a while')
        fpyutils.shell.execute_command_live_output(self.executables.git +
                                                   ' -C ' + self.path +
                                                   ' pull ' + self.remote +
                                                   ' ' + self.checkout_branch)
        fpyutils.shell.execute_command_live_output(
            self.executables.git + ' -C ' + self.path +
            ' submodule foreach --recursive git reset --hard')
        fpyutils.shell.execute_command_live_output(self.executables.git +
                                                   ' -C ' + self.path +
                                                   ' submodule sync')
        # We might need to add the '--recursive' option for 'git submodule update'
        # to build certain packages. This means that we still depend from external
        # services at build time if we use that option.
        fpyutils.shell.execute_command_live_output(
            self.executables.git + ' -C ' + self.path +
            ' submodule update --init --remote')
        fpyutils.shell.execute_command_live_output(
            self.executables.git + ' -C ' + self.path +
            ' submodule foreach git fetch --tags --force')
        logger.info('parent repository changes pulled')

    def _append_local_submodules_configuration(self):
        self.submodules_configuration[
            'local'] = self.local_sumodules_configuration

    # Read the 'configuration.yaml' file in the repository
    def _read_submodules_configuration(self):
        remote_configuration_file = pathlib.Path(self.path,
                                                 'configuration.yaml')
        if remote_configuration_file.is_file():
            self.submodules_configuration = yaml.load(open(
                remote_configuration_file, 'r'),
                                                      Loader=yaml.SafeLoader)
            self.submodules_configuration[
                'remote'] = self.submodules_configuration['submodules']
            del self.submodules_configuration['submodules']
            logger.info(
                'parent repository submodules configuration was read correctly'
            )
        else:
            logger.info('no repository submodules configuration present')
        self._append_local_submodules_configuration()
        logger.info('all submodules configuration was read')

    def _get_submodules(self):
        self.submodules = [
            x for x in pathlib.Path(self.path, 'submodules').iterdir()
        ]
        logger.info('got submodules directories list')

    def _call_worker(self, pypi_credentials: PypiCredentials):
        self._get_submodules()

        signal.signal(signal.SIGINT,
                      lambda signal, frame: self._signal_handler())
        signal.signal(signal.SIGTERM,
                      lambda signal, frame: self._signal_handler())

        for i in range(0, len(self.submodules)):
            logger.info('remaining ' + str(len(self.submodules) - i + 1) +
                        ' submodules')
            d: pathlib.Path = self.submodules[i]

            dirname: str = pathlib.Path(self.path, d).stem
            skip_repository: bool = False
            relative_base_directory_ovr: str = str()
            override_commands: dict = dict()
            ref_checkout: list = list()

            if dirname in self.submodules_configuration['local'][
                    'skip_repository']:
                skip_repository = True
            if dirname in self.submodules_configuration['remote']:
                relative_base_directory_ovr = self.submodules_configuration[
                    'remote'][dirname]['base_directory_override']
                override_commands = self.submodules_configuration['remote'][
                    dirname]['override_commands']
                ref_checkout = self.submodules_configuration['remote'][
                    dirname]['ref_checkout']

            submodule_cfg = SubmoduleConfiguration(
                path=d.stem,
                skip_repository=skip_repository,
                mark_skip_repository_successfull=self.submodules_configuration[
                    'local']['mark_skip_repository_successfull'],
                mark_failed_build_or_upload_successfull=self.
                submodules_configuration['local']
                ['mark_failed_build_or_upload_successfull'],
                relative_base_directory_override=relative_base_directory_ovr,
                override_commands=override_commands,
                ref_checkout=ref_checkout)

            worker = RepositoryWorker(path=str(d),
                                      submodule_configuration=submodule_cfg,
                                      executables=self.executables)
            worker._work(cache=self.cache, pypi_credentials=pypi_credentials)

            self.total_successfull_tags += worker.successfull_tags
            self.total_tags += worker.total_tags

    def _signal_handler(self):
        logger.info('signal received. finished queued workers and writing ' +
                    str(len(self.cache.cache)) +
                    ' repository elements to cache before exit')
        self.cache._write()
        sys.exit(1)

    def _stats(self):
        logger.info('total successfull tags: ' +
                    str(self.total_successfull_tags))
        logger.info('total tags: ' + str(self.total_tags))


class Notify:

    def __init__(self, gotify: dict, email: dict):
        r"""Save data for the notifications."""
        self.message: str = str()
        self.gotify: dict = gotify
        self.email: dict = email

    def _get_message(self, parent_repo: GitParentRepository):
        self.message = ''.join([
            'total tags: ',
            str(parent_repo.total_tags), '\n', 'total successfull tags: ',
            str(parent_repo.total_successfull_tags), '\n',
            'tag successfull ratio: ',
            str(parent_repo.total_successfull_tags / parent_repo.total_tags)
        ])

    def _send(self):
        m = self.gotify['message'] + '\n' + self.message
        if self.gotify['enabled']:
            fpyutils.notify.send_gotify_message(self.gotify['url'],
                                                self.gotify['token'], m,
                                                self.gotify['title'],
                                                self.gotify['priority'])
        if self.email['enabled']:
            fpyutils.notify.send_email(
                self.message, self.email['smtp_server'], self.email['port'],
                self.email['sender'], self.email['user'],
                self.email['password'], self.email['receiver'],
                self.email['subject'])


logger: logging.Logger = _setup_logging()


def main():
    config = _read_yaml_file(shlex.quote(sys.argv[1]))
    if config == dict():
        raise ValueError

    cache = Cache()
    cache._init()
    cache._read()
    execs = Executables(git=config['executables']['git'],
                        python=config['executables']['python'],
                        twine=config['executables']['twine'],
                        rm=config['executables']['rm'])
    pypi = PypiCredentials(config['pypi']['url'], config['pypi']['user'],
                           config['pypi']['password'])
    parent_repo = GitParentRepository(config['repository']['path'],
                                      config['repository']['remote'],
                                      config['repository']['checkout_branch'],
                                      config['submodules'], cache, execs)
    parent_repo._read_submodules_configuration()
    parent_repo._get_updates()
    parent_repo._call_worker(pypi)
    cache._write()

    n = Notify(config['notify']['gotify'], config['notify']['email'])
    n._get_message()
    n._send()


if __name__ == '__main__':
    main()

  9. add the configuration

    /home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.yaml#
    #
# build_python_packages.yaml
#
# Copyright (C) 2021-2022 Franco Masotti (franco \D\o\T masotti {-A-T-} tutanota \D\o\T com)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

notify:
    email:
        enabled: false
        smtp_server: 'smtp.gmail.com'
        port: 465
        sender: 'myusername@gmail.com'
        user: 'myusername'
        password: 'my awesome password'
        receiver: 'myusername@gmail.com'
        subject: 'update action'
    gotify:
        enabled: false
        url: '<gotify url>'
        token: '<app token>'
        title: 'update action'
        message: 'update action'
        priority: 5


repository:
    path: '/home/jobs/scripts/by-user/pypi-source-packages-updater/python-packages-source'
    remote: 'origin'
    checkout_branch: 'dev'

submodules:
    mark_skip_repository_successfull: true
    mark_failed_build_or_upload_successfull: true

    # Directory names.
    skip_repository: []

executables:
    git: 'git'
    python: 'python3'
    rm: 'rm'
    twine: 'twine'

pypi:
    url: '<PyPI URL>'
    user: '<PyPI username>'
    password: '<PyPI password>'

  10. add the helper script. update_and_build_python_packages.sh clones and updates the python-packages-source 14 repository and compiles all the packages

    /home/jobs/scripts/by-user/python-source-packages-updater/update_and_build_python_packages.sh#
    #!/usr/bin/env bash

REPOSITORY='https://software.franco.net.eu.org/frnmst/python-packages-source.git'

pushd /home/jobs/scripts/by-user/python-source-packages-updater
export PATH=$PATH:/home/python-source-packages-updater/.local/bin/

git clone "${REPOSITORY}"

pushd python-packages-source

git checkout dev
git pull

# Always commit and push to dev only.
[ "$(git branch --show-current)" = 'dev' ] || exit 1

# Update all submodules and the stats.
make install-dev
make submodules-update
make submodules-add-gitea
make stats
git add -A
git commit -m "Submodule updates."
git push

popd

# Compile the packages.
./build_python_packages.py ./build_python_packages.yaml

# Cleanup.
rm -rf python-packages-source

popd

  11. add the Systemd service file

    /home/jobs/services/by-user/python-source-packages-updater/build-python-packages.service#
    [Unit]
Description=Build python packages

[Service]
Type=simple
ExecStart=/home/jobs/scripts/by-user/python-source-packages-updater/update_and_build_python_packages.sh
User=python-source-packages-updater
Group=python-source-packages-updater
StandardOutput=null
StandardError=null

  12. add the Systemd timer unit file

    /home/jobs/services/by-user/python-source-packages-updater/build-python-packages.timer#
    [Unit]
Description=Once every week build python packages

[Timer]
OnCalendar=Weekly
Persistent=true

[Install]
WantedBy=timers.target

  13. fix the permissions

    chown -R python-source-packages-updater:python-source-packages-updater /home/jobs/{scripts,services}/by-user/python-source-packages-updater
chmod 700 -R /home/jobs/{scripts,services}/by-user/python-source-packages-updater

  14. run the deploy script

  15. To be able to compile most packages you need to manually compile at least these basic ones and push them to you local PyPI server.

    • setuptools

    • setuptools_scm

    • wheel

    You can clone the python-packages-source 14 respository then compile and upload these basic packages.

    sudo -i -u python-source-package-updater
git clone https://software.franco.net.eu.org/frnmst/python-packages-source.git
cd python-packages-source/setuptools
python3 -m build --sdist --wheel
twine upload --repository-url ${your_pypi_index_url} dist/*
exit

    Important

    Some packages might need different dependencies. Have a look at the setup_requires variable in setup.py or in setup.cfg or requires in the pyproject.toml file. If you cannot compile some, download them directly from pypi.python.org.

Updating the package graph#

When you run the helper script you can update the stats graph automatically by using a GIT commit hook. The following script generates the graph and copies it to a webserver directory.

  1. connect via SSH to the GIT remote machine and install the dependencies

    sudo -i
apt-get install git python3 make pipenv
exit

  2. connect to the git deploy user

    sudo -i -u git-deploy

  3. configure your remote: add this to the post-receive hooks

    ${remote_git_repository}/hooks/post-receive#
    #!/usr/bin/bash -l

IMAGE=""$(echo -n 'frnmst/python-packages-source' | sha1sum | awk '{print $1 }')"_graph0.png"
DOMAIN='assets.franco.net.eu.org'

TMP_GIT_CLONE=""${HOME}"/tmp/python-packages-source"
PUBLIC_WWW="/var/www/${DOMAIN}/image/${IMAGE}"

git clone "${GIT_DIR}" "${TMP_GIT_CLONE}"

pushd "${TMP_GIT_CLONE}"
make install
make plot OUTPUT="${PUBLIC_WWW}"
chmod 770 "${PUBLIC_WWW}"
popd

rm --recursive --force "${TMP_GIT_CLONE}"

Using the PyPI server#

  1. change the PyPI index of your programs. See for example https://software.franco.net.eu.org/frnmst/python-packages-source#client-configuration

Footnotes

1

https://guide.debianizzati.org/index.php/Debmirror:_creiamo_un_mirror_Debian CC BY-SA 4.0, Copyright (c) guide.debianizzati contributors

2

https://docs.mirantis.com/mcp/q4-18/mcp-dev-resources/debmirror/debmirror_readme.html unknown license

3

https://www.ducea.com/2006/06/19/apache-tips-tricks-hide-a-file-type-from-directory-indexes/ unknown license

4

https://blog.programster.org/set-up-a-local-ubuntu-mirror-with-apt-mirror unknown license

5

https://www.debian.org/mirror/size MIT License, Copyright © 1997-2021 Software in the Public Interest, Inc. and others

6

https://bugs.launchpad.net/ubuntu/+source/debmirror/+bug/882941/comments/4 unknown license

7

https://people.debian.org/~koster/www/mirror/ftpmirror Open Publication License, Draft v1.0 or later, Copyright © 1997-2011 Software in the Public Interest, Inc.

8

https://perishablepress.com/better-default-directory-views-with-htaccess/ CC BY 4.0, © 2004–2022 Perishable Press

9

https://github.com/pypiserver/pypiserver zlib/libpng + MIT license, Copyright (c) 2011-2014 Ralf Schmitt

10

https://github.com/pypiserver/pypiserver#serving-thousands-of-packages zlib/libpng + MIT license, Copyright (c) 2011-2014 Ralf Schmitt

11

https://httpd.apache.org/docs/2.4/caching.html Apache License, Version 2.0. Copyright (c) Apache HTTPD contributors

12

https://httpd.apache.org/docs/2.4/mod/mod_cache.html Apache License, Version 2.0. Copyright (c) Apache HTTPD contributors

13

https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti

14(1,2,3)

https://software.franco.net.eu.org/frnmst/python-packages-source GPLv3+, Copyright (C) 2021-2022, Franco Masotti

15

https://stackoverflow.com/a/13847807 CC BY-SA 4.0, Copyright (c) 2012, 2020, spiralman, bryant1410 (at stackoverflow.com)

16

https://github.com/pypa/build/issues/328#issuecomment-877028239 unknown license