Package server ============== Mirroring --------- Debmirror ````````` .. seealso:: - _`Debmirror: creiamo un mirror Debian - Guide@Debianizzati.Org` [#f1]_ - _`Mirantis Documentation: Usage ~ DEBMIRROR` [#f2]_ - _`Apache Tips & Tricks: Hide a file type from directory indexes · MD/Blog` [#f3]_ - _`Set Up A Local Ubuntu / Debian Mirror with Apt-Mirror | Programster's Blog` [#f4]_ - _`Debian -- Mirror Size` [#f5]_ - _`Comment #4 : Bug #882941 : Bugs : debmirror package : Ubuntu` [#f6]_ - _`Debian -- Setting up a Debian archive mirror` [#f7]_ - _`Better Default Directory Views with HTAccess | Perishable Press` [#f8]_ +--------------+--------------------+ | Run as user | Instruction number | +--------------+--------------------+ | ``root`` | 1-3,5-14 | +--------------+--------------------+ | ``debmirror``| 4 | +--------------+--------------------+ Debmirror is one of the existing programs which are able to mirror Debian APT packages. In this example we are going to use the Apache HTTP server as webserver. In this example we will use the Apache HTTP server to serve the packages. #. install .. code-block:: shell-session apt-get install debmirror bindfs #. create a new user .. code-block:: shell-session useradd -m -s /bin/bash -U debmirror passwd debmirror usermod -aG jobs debmirror mkdir /home/debmirror/data chown debmirror:debmirror /home/debmirror/data .. note:: In this example ``/home/debmirror/data`` is the base directory where all packages are served from. .. tip:: I suggest using normal HDDs rather than SSDs. Size in more important than speed in this case. #. create the jobs directories .. code-block:: shell-session mkdir -p /home/jobs/{services,scripts}/by-user/debmirror chown -R debmirror:debmirror /home/jobs/{services,scripts}/by-user/debmirror chmod 700 -R /home/jobs/{services,scripts}/by-user/debmirror #. _`load APT's keyring` into a new keyring owned by the ``debmirror`` user .. code-block:: shell-session gpg --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export \ | gpg --no-default-keyring --keyring trustedkeys.gpg --import #. add this :download:`configuration ` for the standard Debian repository .. literalinclude:: includes/home/jobs/scripts/by-user/debmirror/debmirror.debian.conf :language: perl :linenos: :caption: /home/jobs/scripts/by-user/debmirror/debmirror.debian.conf #. change permissions .. code-block:: shell-session chown -R debmirror:debmirror /home/jobs/scripts/by-user/debmirror/debmirror.debian.conf chmod 700 -R /home/jobs/scripts/by-user/debmirror/debmirror.debian.conf #. create the :download:`Systemd service unit file ` .. literalinclude:: includes/home/jobs/services/by-user/debmirror/debmirror.debian.service :language: ini :linenos: :caption: /home/jobs/services/by-user/debmirror/debmirror.debian.service #. create the :download:`Systemd service timer unit file ` .. literalinclude:: includes/home/jobs/services/by-user/debmirror/debmirror.debian.timer :language: ini :linenos: :caption: /home/jobs/services/by-user/debmirror/debmirror.debian.timer #. create a directory readable be Apache .. code-block:: shell-session mkdir -p /srv/http/debian chown www-data:www-data /srv/http/debian chmod 700 /srv/http/debian #. Add this to the fstab file .. code-block:: ini :caption: /etc/fstab /home/debmirror/data /srv/http/debian fuse.bindfs auto,force-user=www-data,force-group=www-data,ro 0 0 .. note:: This mount command makes the directory exposed to the webserver readonly, in this case ``/srv/http/debian`` #. serve the files via HTTP by creating a new :download:`Apache virtual host `. Replace ``FQDN`` with the appropriate domain and include this file from the Apache configuration .. literalinclude:: includes/etc/apache2/debian_mirror.apache.conf :language: apache :linenos: :caption: /etc/apache2/debian_mirror.apache.conf #. create a new :download:`text file ` that will serve as basic instructions for configuring the APT sources file. Replace ``FQDN`` with the appropriate domain .. literalinclude:: includes/home/debmirror/data/footer.html :language: html :linenos: :caption: /home/debmirror/data/footer.html .. note:: This example includes some unofficial repositories. #. restart the Apache webserver .. code-block:: shell-session systemctl restart apache2 #. run the :ref:`deploy script ` Unofficial Debian sources ~~~~~~~~~~~~~~~~~~~~~~~~~ +--------------+--------------------+ | Run as user | Instruction number | +--------------+--------------------+ | ``debmirror``| \* | +--------------+--------------------+ In case you want to mirror unofficial Debian sources the same instrctions apply. You just need to change the :ref:`key import step ` .. code-block:: shell-session gpg \ --no-default-keyring \ --keyring trustedkeys.gpg \ --import ${package_signing_key} .. note:: ``package_signing_key`` is provided by the repository maintainers. PyPI server ----------- Build Python packages using git sources and push them to a self-hosted PyPI server. Server `````` .. seealso:: - _`Minimal PyPI server for uploading & downloading packages with pip/easy_install Resources` [#f9]_ +--------------+--------------------+ | Run as user | Instruction number | +--------------+--------------------+ | ``root`` | \* | +--------------+--------------------+ #. follow the :ref:`Docker ` instructions #. create the jobs directories .. code-block:: shell-session mkdir -p /home/jobs/scripts/by-user/root/docker/pypiserver chmod 700 /home/jobs/scripts/by-user/root/docker/pypiserver #. install and run `pypiserver `_. Use this :download:`Docker compose file ` .. literalinclude:: includes/home/jobs/scripts/by-user/root/docker/pypiserver/docker-compose.yml :language: yaml :linenos: :caption: /home/jobs/scripts/by-user/root/docker/pypiserver/docker-compose.yml #. create a :download:`Systemd unit file `. See also the :ref:`Docker compose services ` section .. literalinclude:: includes/home/jobs/services/by-user/root/docker-compose.pypiserver.service :language: ini :linenos: :caption: /home/jobs/services/by-user/root/docker-compose.pypiserver.service #. run the :ref:`deploy script ` #. modify the reverse proxy port of your webserver configuration with ``4000`` Apache configuration ```````````````````` .. seealso:: - _`GitHub - pypiserver/pypiserver: Minimal PyPI server for uploading & downloading packages with pip/easy_install - Serving Thousands of Packages` [#f10]_ - _`Caching Guide - Apache HTTP Server Version 2.4` [#f11]_ - _`mod_cache - Apache HTTP Server Version 2.4` [#f12]_ +--------------+--------------------+ | Run as user | Instruction number | +--------------+--------------------+ | ``root`` | \* | +--------------+--------------------+ If you use Apache as webserver you should enable caching. The upstream documentation shows how to configure pypiserver for Nginx but not for Apache. #. create a new :download:`Apache virtual host `. Replace ``FQDN`` with the appropriate domain .. literalinclude:: includes/etc/apache2/pypi_server.apache.conf :language: apache :linenos: :caption: /etc/apache2/pypi_server.apache.conf .. warning:: The included ``Cache*`` options are very aggressive! #. create the cache directory .. code-block:: shell-session mkdir /var/cache/apache chown www-data:www-data /var/cache/apache chmod 775 /var/cache/apache #. enable the Apache modules .. code-block:: shell-session a2enmod cache cache_disk systemctl start apache-htcacheclean.service systemctl restart apache2 #. check for a cache hit. Replace ``FQDN`` with the appropriate domain .. code-block:: shell-session curl -s https://${FQDN} 1>/dev/null 2>/dev/null curl -s -D - https://${FQDN} | head -n 20 .. note:: The ``/packages/`` page does not get cached. #. set ``CacheHeader`` and ``CacheDetailHeader`` to ``Off`` #. restart Apache .. code-block:: shell-session systemctl restart apache2 Virtual machine compiling the packages `````````````````````````````````````` .. seealso:: - A collection of scripts I have written and/or adapted that I currently use on my systems as automated tasks [#f13]_ - _`Git repository pointers and configurations to build Python packages from source` [#f14]_ - _`python - pushd through os.system - Stack Overflow` [#f15]_ - _`Pass options to `build_ext` · Issue #328 · pypa/build · GitHub` [#f16]_ +----------------------------------------+--------------------+ | Run as user | Instruction number | +----------------------------------------+--------------------+ | ``qvm`` | 1-2 | +----------------------------------------+--------------------+ | ``root`` (VM) | 3-13 | +----------------------------------------+--------------------+ | ``python-source-package-updater`` (VM) | 14 | +----------------------------------------+--------------------+ #. create a virtual machine with Debian Bullseye (stable) and transform it into `Sid `_ (unstable). Using the unstable version will provide more up to date software for development. See the :ref:`QEMU server ` section. You might need to assign a lost of disk space. #. connect to the virtual machine. See the :ref:`QEMU client ` section #. create a new user .. code-block:: shell-session useradd --system -s /bin/bash -U python-source-package-updater passwd python-source-package-updater usermod -aG jobs python-source-package-updater #. create the jobs directories. See :ref:`reference ` .. code-block:: shell-session mkdir -p /home/jobs/{scripts,services}/by-user/python-source-package-updater chown -R kiwix:kiwix /home/jobs/{scripts,services}/by-user/python-source-package-updater chmod 700 -R /home/jobs/{scripts,services}/by-user/python-source-package-updater #. install these packages in the virtual machine: .. code-block:: shell-session apt-get install build-essential fakeroot devscripts git python3-dev python3-all-dev \ games-python3-dev libgmp-dev libssl-dev libssl1.1=1.1.1k-1 libcurl4-openssl-dev \ python3-pip python3-build twine libffi-dev graphviz libgraphviz-dev pkg-config \ clang-tools libblas-dev astro-all libblas-dev libatlas-base-dev libopenblas-dev \ libgsl-dev libblis-dev liblapack-dev liblapack3 libgslcblas0 libopenblas-base \ libatlas3-base libblas3 clang-9 clang-13 clang-12 clang-11 sphinx-doc \ libbliss-dev libblis-dev libbliss2 libblis64-serial-dev libblis64-pthread-dev \ libblis64-openmp-dev libblis64-3-serial libblis64-dev libblis64-3-pthread \ libblis64-3-openmp libblis64-3 libblis3-serial libblis3-pthread \ libblis-serial-dev libblis-pthread-dev libargon2-dev libargon2-0 libargon2-1 .. note:: This is just a selection. Some Python packages need other dependencies not listed here. #. install the dependencies of the script .. code-block:: shell-session apt-get install python3-yaml python3-appdirs #. install fpyutils. See :ref:`reference ` #. add the :download:`script ` .. literalinclude:: includes/home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.py :language: python :linenos: :caption: /home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.py #. add the :download:`configuration ` .. literalinclude:: includes/home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.yaml :language: yaml :linenos: :caption: /home/jobs/scripts/by-user/python-source-packages-updater/build_python_packages.yaml #. add the :download:`helper script `. _`update_and_build_python_packages.sh` clones and updates the python-packages-source [#f14]_ repository and compiles all the packages .. literalinclude:: includes/home/jobs/scripts/by-user/python-source-packages-updater/update_and_build_python_packages.sh :language: bash :linenos: :caption: /home/jobs/scripts/by-user/python-source-packages-updater/update_and_build_python_packages.sh #. add the :download:`Systemd service file ` .. literalinclude:: includes/home/jobs/services/by-user/python-source-packages-updater/build-python-packages.service :language: ini :linenos: :caption: /home/jobs/services/by-user/python-source-packages-updater/build-python-packages.service #. add the :download:`Systemd timer unit file ` .. literalinclude:: includes/home/jobs/services/by-user/python-source-packages-updater/build-python-packages.timer :language: ini :linenos: :caption: /home/jobs/services/by-user/python-source-packages-updater/build-python-packages.timer #. run the :ref:`deploy script ` #. To be able to compile most packages you need to manually compile at least these basic ones and push them to you local PyPI server. - ``setuptools`` - ``setuptools_scm`` - ``wheel`` You can clone the python-packages-source [#f14]_ respository then compile and upload these basic packages. .. code-block:: shell-session git clone https://software.franco.net.eu.org/frnmst/python-packages-source.git cd python-packages-source/setuptools python3 -m build --sdist --wheel twine upload --repository-url ${your_pypi_index_url} dist/* .. important:: Some packages might need different dependencies. Have a look at the ``setup_requires`` variable in ``setup.py`` or in ``setup.cfg`` or ``requires`` in the ``pyproject.toml`` file. If you cannot compile some, download them directly from `pypi.python.org `_. Updating the package graph ~~~~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------+--------------------+ | Run as user | Instruction number | +---------------------------+--------------------+ | ```` | 1 | +---------------------------+--------------------+ | ``root`` | 2 | +---------------------------+--------------------+ | ```` | 3 | +---------------------------+--------------------+ When you run the :ref:`helper script ` you can update the stats graph automatically by using a GIT commit hook. The following script generates the graph and copies it to a webserver directory. #. connect via SSH to the GIT remote machine #. install the dependencies .. code-block:: shell-session apt-get install git python3 make pipenv #. configure your remote: add this to the post-receive hooks .. code-block:: bash :linenos: :caption: ${remote_git_repository}/hooks/post-receive #!/usr/bin/bash -l IMAGE=""$(echo -n 'frnmst/python-packages-source' | sha1sum | awk '{print $1 }')"_graph0.png" DOMAIN='assets.franco.net.eu.org' TMP_GIT_CLONE=""${HOME}"/tmp/python-packages-source" PUBLIC_WWW="/var/www/${DOMAIN}/image/${IMAGE}" git clone "${GIT_DIR}" "${TMP_GIT_CLONE}" pushd "${TMP_GIT_CLONE}" make install make plot OUTPUT="${PUBLIC_WWW}" chmod 770 "${PUBLIC_WWW}" popd rm --recursive --force "${TMP_GIT_CLONE}" Using the PyPI server ````````````````````` #. change the PyPI index of your programs. See for example https://software.franco.net.eu.org/frnmst/python-packages-source#client-configuration .. rubric:: Footnotes .. [#f1] https://guide.debianizzati.org/index.php/Debmirror:_creiamo_un_mirror_Debian CC BY-SA 4.0, Copyright (c) guide.debianizzati contributors .. [#f2] https://docs.mirantis.com/mcp/q4-18/mcp-dev-resources/debmirror/debmirror_readme.html unknown license .. [#f3] https://www.ducea.com/2006/06/19/apache-tips-tricks-hide-a-file-type-from-directory-indexes/ unknown license .. [#f4] https://blog.programster.org/set-up-a-local-ubuntu-mirror-with-apt-mirror unknown license .. [#f5] https://www.debian.org/mirror/size MIT License, Copyright © 1997-2021 Software in the Public Interest, Inc. and others .. [#f6] https://bugs.launchpad.net/ubuntu/+source/debmirror/+bug/882941/comments/4 unknown license .. [#f7] https://people.debian.org/~koster/www/mirror/ftpmirror Open Publication License, Draft v1.0 or later, Copyright © 1997-2011 Software in the Public Interest, Inc. .. [#f8] https://perishablepress.com/better-default-directory-views-with-htaccess/ CC BY 4.0, © 2004–2022 Perishable Press .. [#f9] https://github.com/pypiserver/pypiserver zlib/libpng + MIT license, Copyright (c) 2011-2014 Ralf Schmitt .. [#f10] https://github.com/pypiserver/pypiserver#serving-thousands-of-packages zlib/libpng + MIT license, Copyright (c) 2011-2014 Ralf Schmitt .. [#f11] https://httpd.apache.org/docs/2.4/caching.html Apache License, Version 2.0. Copyright (c) Apache HTTPD contributors .. [#f12] https://httpd.apache.org/docs/2.4/mod/mod_cache.html Apache License, Version 2.0. Copyright (c) Apache HTTPD contributors .. [#f13] https://software.franco.net.eu.org/frnmst/automated-tasks GNU GPLv3+, copyright (c) 2019-2022, Franco Masotti .. [#f14] https://software.franco.net.eu.org/frnmst/python-packages-source GPLv3+, Copyright (C) 2021-2022, Franco Masotti .. [#f15] https://stackoverflow.com/a/13847807 CC BY-SA 4.0, Copyright (c) 2012, 2020, spiralman, bryant1410 (at stackoverflow.com) .. [#f16] https://github.com/pypa/build/issues/328#issuecomment-877028239 unknown license