Package server ============== Mirroring --------- Debmirror ````````` Debmirror is one of the existing programs which are able to mirror Debian APT packages. In this example we are going to use the Apache HTTP server as webserver. In this example we will use the Apache HTTP server to serve the packages. .. seealso:: - _`Debmirror: creiamo un mirror Debian - Guide@Debianizzati.Org` [#f1]_ - _`Mirantis Documentation: Usage ~ DEBMIRROR` [#f2]_ - _`Apache Tips & Tricks: Hide a file type from directory indexes · MD/Blog` [#f3]_ - _`Set Up A Local Ubuntu / Debian Mirror with Apt-Mirror | Programster's Blog` [#f4]_ - _`Debian -- Mirror Size` [#f5]_ - _`Comment #4 : Bug #882941 : Bugs : debmirror package : Ubuntu` [#f6]_ - _`Debian -- Setting up a Debian archive mirror` [#f7]_ - _`Better Default Directory Views with HTAccess | Perishable Press` [#f8]_ #. install .. code-block:: shell-session apt-get install debmirror bindfs #. create a new user .. code-block:: shell-session useradd -m -s /bin/bash -U debmirror passwd debmirror usermod -aG jobs debmirror mkdir /home/debmirror/data chown debmirror:debmirror /home/debmirror/data .. note:: In this example ``/home/debmirror/data`` is the base directory where all packages are served from. .. tip:: I suggest using normal HDDs rather than SSDs. Size in more important than speed in this case. #. create the jobs directories .. code-block:: shell-session mkdir -p /home/jobs/{services,scripts}/by-user/debmirror #. _`load APT's keyring` into a new keyring owned by the ``debmirror`` user .. code-block:: shell-session sudo -i -u debmirror gpg --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export \ | gpg --no-default-keyring --keyring trustedkeys.gpg --import exit #. add this :download:`configuration ` for the standard Debian repository .. literalinclude:: includes/home/jobs/scripts/by-user/debmirror/debmirror.debian.conf :language: perl :caption: /home/jobs/scripts/by-user/debmirror/debmirror.debian.conf #. create the :download:`Systemd service unit file ` .. literalinclude:: includes/home/jobs/services/by-user/debmirror/debmirror.debian.service :language: ini :caption: /home/jobs/services/by-user/debmirror/debmirror.debian.service #. create the :download:`Systemd service timer unit file ` .. literalinclude:: includes/home/jobs/services/by-user/debmirror/debmirror.debian.timer :language: ini :caption: /home/jobs/services/by-user/debmirror/debmirror.debian.timer #. create a directory readable be Apache .. code-block:: shell-session mkdir -p /srv/http/debian chown www-data:www-data /srv/http/debian chmod 700 /srv/http/debian #. Add this to the fstab file .. code-block:: ini :caption: /etc/fstab /home/debmirror/data /srv/http/debian fuse.bindfs auto,force-user=www-data,force-group=www-data,ro 0 0 .. note:: This mount command makes the directory exposed to the webserver readonly, in this case ``/srv/http/debian`` #. serve the files via HTTP by creating a new :download:`Apache virtual host `. Replace ``FQDN`` with the appropriate domain and include this file from the Apache configuration .. literalinclude:: includes/etc/apache2/debian_mirror.apache.conf :language: apache :caption: /etc/apache2/debian_mirror.apache.conf #. create a new :download:`text file ` that will serve as basic instructions for configuring the APT sources file. Replace ``FQDN`` with the appropriate domain .. literalinclude:: includes/home/debmirror/data/footer.html :language: html :caption: /home/debmirror/data/footer.html .. note:: This example includes some unofficial repositories. #. restart the Apache webserver .. code-block:: shell-session systemctl restart apache2 #. fix the permissions .. code-block:: shell-session chown -R debmirror:debmirror /home/jobs/{services,scripts}/by-user/debmirror chmod 700 -R /home/jobs/{services,scripts}/by-user/debmirror #. run the :ref:`deploy script ` Unofficial Debian sources ~~~~~~~~~~~~~~~~~~~~~~~~~ In case you want to mirror unofficial Debian sources the same instrctions apply. You just need to change the :ref:`key import step ` .. code-block:: shell-session sudo -i -u debmirror gpg \ --no-default-keyring \ --keyring trustedkeys.gpg \ --import ${package_signing_key} exit .. note:: ``package_signing_key`` is provided by the repository maintainers. PyPI server ----------- Build Python packages using git sources and push them to a self-hosted PyPI server. Server `````` .. seealso:: - _`Minimal PyPI server for uploading & downloading packages with pip/easy_install Resources` [#f9]_ #. follow the :ref:`Docker ` instructions #. create the jobs directories .. code-block:: shell-session mkdir -p /home/jobs/scripts/by-user/root/docker/pypiserver chmod 700 /home/jobs/scripts/by-user/root/docker/pypiserver #. install and run `pypiserver `_. Use this :download:`Docker compose file ` .. literalinclude:: includes/home/jobs/scripts/by-user/root/docker/pypiserver/docker-compose.yml :language: yaml :caption: /home/jobs/scripts/by-user/root/docker/pypiserver/docker-compose.yml #. create a :download:`Systemd unit file `. See also the :ref:`Docker compose services ` section .. literalinclude:: includes/home/jobs/services/by-user/root/docker-compose.pypiserver.service :language: ini :caption: /home/jobs/services/by-user/root/docker-compose.pypiserver.service #. fix the permissions .. code-block:: shell-session chmod 700 /home/jobs/scripts/by-user/root/docker/pypiserver chmod 700 -R /home/jobs/services/by-user/root #. run the :ref:`deploy script ` #. modify the reverse proxy port of your webserver configuration with ``4000`` Apache configuration ```````````````````` If you use Apache as webserver you should enable caching. The upstream documentation shows how to configure pypiserver for Nginx but not for Apache. .. seealso:: - _`GitHub - pypiserver/pypiserver: Minimal PyPI server for uploading & downloading packages with pip/easy_install - Serving Thousands of Packages` [#f10]_ - _`Caching Guide - Apache HTTP Server Version 2.4` [#f11]_ - _`mod_cache - Apache HTTP Server Version 2.4` [#f12]_ #. create a new :download:`Apache virtual host `. Replace ``FQDN`` with the appropriate domain .. literalinclude:: includes/etc/apache2/pypi_server.apache.conf :language: apache :caption: /etc/apache2/pypi_server.apache.conf .. warning:: The included ``Cache*`` options are very aggressive! #. create the cache directory .. code-block:: shell-session mkdir /var/cache/apache chown www-data:www-data /var/cache/apache chmod 775 /var/cache/apache #. enable the Apache modules .. code-block:: shell-session a2enmod cache cache_disk systemctl start apache-htcacheclean.service systemctl restart apache2 #. check for a cache hit. Replace ``FQDN`` with the appropriate domain .. code-block:: shell-session curl -s https://${FQDN} 1>/dev/null 2>/dev/null curl -s -D - https://${FQDN} | head -n 20 .. note:: The ``/packages/`` page does not get cached. #. set ``CacheHeader`` and ``CacheDetailHeader`` to ``Off`` #. restart Apache .. code-block:: shell-session systemctl restart apache2 Virtual machine compiling the packages `````````````````````````````````````` I suggest using a virtual machine to compile packages to improve isolation and security: arbitrary code might be executed when compiling a package. .. seealso:: - A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks [#f13]_ - _`Git repository pointers and configurations to build Python packages from source` [#f14]_ - _`python - pushd through os.system - Stack Overflow` [#f15]_ - _`Pass options to `build_ext` · Issue #328 · pypa/build · GitHub` [#f16]_ #. create a virtual machine with Debian Bullseye (stable) and transform it into `Sid `_ (unstable). Using the unstable version will provide more up to date software for development. See the :ref:`QEMU server ` section. You might need to assign a lost of disk space. #. connect to the virtual machine. See the :ref:`QEMU client ` section #. create a new user .. code-block:: shell-session sudo -i useradd --system -s /bin/bash -U python-source-package-updater passwd python-source-package-updater usermod -aG jobs python-source-package-updater #. create the jobs directories. See :ref:`reference ` .. code-block:: shell-session mkdir -p /home/jobs/{scripts,services}/by-user/python-source-package-updater chown -R python-source-package-updater:python-source-package-updater /home/jobs/{scripts,services}/by-user/python-source-package-updater chmod 700 -R /home/jobs/{scripts,services}/by-user/python-source-package-updater #. install these packages in the virtual machine: .. code-block:: shell-session apt-get install build-essential fakeroot devscripts git python3-dev python3-all-dev \ games-python3-dev libgmp-dev libssl-dev libssl1.1=1.1.1k-1 libcurl4-openssl-dev \ python3-pip python3-build twine libffi-dev graphviz libgraphviz-dev pkg-config \ clang-tools libblas-dev astro-all libblas-dev libatlas-base-dev libopenblas-dev \ libgsl-dev libblis-dev liblapack-dev liblapack3 libgslcblas0 libopenblas-base \ libatlas3-base libblas3 clang-9 clang-13 clang-12 clang-11 sphinx-doc \ libbliss-dev libblis-dev libbliss2 libblis64-serial-dev libblis64-pthread-dev \ libblis64-openmp-dev libblis64-3-serial libblis64-dev libblis64-3-pthread \ libblis64-3-openmp libblis64-3 libblis3-serial libblis3-pthread \ libblis-serial-dev libblis-pthread-dev libargon2-dev libargon2-0 libargon2-1 .. note:: This is just a selection. Some Python packages need other dependencies not listed here. #. install the dependencies of the script .. code-block:: shell-session apt-get install python3-yaml python3-fpyutils pip3 install --user platformdirs #. install fpyutils. See :ref:`reference ` #. add the :download:`script ` .. literalinclude:: includes/home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.py :language: python :caption: /home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.py #. add the :download:`configuration ` .. literalinclude:: includes/home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.yaml :language: yaml :caption: /home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.yaml #. add the :download:`helper script `. _`update_and_build_python_packages.sh` clones and updates the python-packages-source [#f14]_ repository and compiles all the packages .. literalinclude:: includes/home/jobs/scripts/by-user/python-source-packages-updater/update_and_build_python_packages.sh :language: bash :caption: /home/jobs/scripts/by-user/python-source-packages-updater/update_and_build_python_packages.sh #. add the :download:`Systemd service file ` .. literalinclude:: includes/home/jobs/services/by-user/python-source-packages-updater/build-python-packages.service :language: ini :caption: /home/jobs/services/by-user/python-source-packages-updater/build-python-packages.service #. add the :download:`Systemd timer unit file ` .. literalinclude:: includes/home/jobs/services/by-user/python-source-packages-updater/build-python-packages.timer :language: ini :caption: /home/jobs/services/by-user/python-source-packages-updater/build-python-packages.timer #. fix the permissions .. code-block:: shell-session chown -R python-source-packages-updater:python-source-packages-updater /home/jobs/{scripts,services}/by-user/python-source-packages-updater chmod 700 -R /home/jobs/{scripts,services}/by-user/python-source-packages-updater #. run the :ref:`deploy script ` #. To be able to compile most packages you need to manually compile at least these basic ones and push them to you local PyPI server. - ``setuptools`` - ``setuptools_scm`` - ``wheel`` You can clone the python-packages-source [#f14]_ respository then compile and upload these basic packages. .. code-block:: shell-session sudo -i -u python-source-package-updater git clone https://software.franco.net.eu.org/frnmst/python-packages-source.git cd python-packages-source/setuptools python3 -m build --sdist --wheel twine upload --repository-url ${your_pypi_index_url} dist/* exit .. important:: Some packages might need different dependencies. Have a look at the ``setup_requires`` variable in ``setup.py`` or in ``setup.cfg`` or ``requires`` in the ``pyproject.toml`` file. If you cannot compile some, download them directly from `pypi.python.org `_. Updating the package graph ~~~~~~~~~~~~~~~~~~~~~~~~~~ When you run the :ref:`helper script ` you can update the stats graph automatically by using a GIT commit hook. The following script generates the graph and copies it to a webserver directory. #. connect via SSH to the GIT remote machine and install the dependencies .. code-block:: shell-session sudo -i apt-get install git python3 make pipenv exit #. connect to the git deploy user .. code-block:: bash sudo -i -u git-deploy #. configure your remote: add this to the post-receive hooks .. code-block:: bash :caption: ${remote_git_repository}/hooks/post-receive #!/usr/bin/bash -l IMAGE=""$(echo -n 'frnmst/python-packages-source' | sha1sum | awk '{print $1 }')"_graph0.png" DOMAIN='assets.franco.net.eu.org' TMP_GIT_CLONE=""${HOME}"/tmp/python-packages-source" PUBLIC_WWW="/var/www/${DOMAIN}/image/${IMAGE}" git clone "${GIT_DIR}" "${TMP_GIT_CLONE}" pushd "${TMP_GIT_CLONE}" make install make plot OUTPUT="${PUBLIC_WWW}" chmod 770 "${PUBLIC_WWW}" popd rm --recursive --force "${TMP_GIT_CLONE}" Using the PyPI server ````````````````````` #. change the PyPI index of your programs. See for example https://software.franco.net.eu.org/frnmst/python-packages-source#client-configuration .. rubric:: Footnotes .. [#f1] https://guide.debianizzati.org/index.php/Debmirror:_creiamo_un_mirror_Debian CC BY-SA 4.0, Copyright (c) guide.debianizzati contributors .. [#f2] https://docs.mirantis.com/mcp/q4-18/mcp-dev-resources/debmirror/debmirror_readme.html unknown license .. [#f3] https://www.ducea.com/2006/06/19/apache-tips-tricks-hide-a-file-type-from-directory-indexes/ unknown license .. [#f4] https://blog.programster.org/set-up-a-local-ubuntu-mirror-with-apt-mirror unknown license .. [#f5] https://www.debian.org/mirror/size MIT License, Copyright © 1997-2021 Software in the Public Interest, Inc. and others .. [#f6] https://bugs.launchpad.net/ubuntu/+source/debmirror/+bug/882941/comments/4 unknown license .. [#f7] https://people.debian.org/~koster/www/mirror/ftpmirror Open Publication License, Draft v1.0 or later, Copyright © 1997-2011 Software in the Public Interest, Inc. .. [#f8] https://perishablepress.com/better-default-directory-views-with-htaccess/ CC BY 4.0, © 2004–2022 Perishable Press .. [#f9] https://github.com/pypiserver/pypiserver zlib/libpng + MIT license, Copyright (c) 2011-2014 Ralf Schmitt .. [#f10] https://github.com/pypiserver/pypiserver#serving-thousands-of-packages zlib/libpng + MIT license, Copyright (c) 2011-2014 Ralf Schmitt .. [#f11] https://httpd.apache.org/docs/2.4/caching.html Apache License, Version 2.0. Copyright (c) Apache HTTPD contributors .. [#f12] https://httpd.apache.org/docs/2.4/mod/mod_cache.html Apache License, Version 2.0. Copyright (c) Apache HTTPD contributors .. [#f13] https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti .. [#f14] https://software.franco.net.eu.org/frnmst/python-packages-source GPLv3+, Copyright (C) 2021-2022, Franco Masotti .. [#f15] https://stackoverflow.com/a/13847807 CC BY-SA 4.0, Copyright (c) 2012, 2020, spiralman, bryant1410 (at stackoverflow.com) .. [#f16] https://github.com/pypa/build/issues/328#issuecomment-877028239 unknown license