Packaging and uploading any Dune module to PyPI can be done using the dunepackaging.py script in dune-common/bin. This page discusses the creating of Python bindings for a new dune module. For the usage of existing Python bindings, see Use Python bindings.

Preparing the module

To provide sufficient metadata for the package additional fields should be added to dune.module. These fields are:

Author: S. Body
Description: Some DUNE mdoule
URL: e.g. GitLab link
Python-Requires: e.g. numpy scipy

The last line is only required in case Python bindings are available that require additional packages to be installed. Here the usual Python syntax can be used, e.g., numpy>=1.19. The string for the description entry can be taken from the pc.in file and replaced there with the line

Description: @DESCRIPTION@

To generate the distribution package run

PATH_TO_DUNE_COMMON/bin/dunepackaging.py --onlysdist

This generates the files setup.py, pyproject.toml and the folder dist containing the tar.gz file ready for upload to PyPI.

Using dunecontrol multiple packages can be built and uploaded for all available source repos by executing

PATH_TO_DUNE_COMMON/bin/dunecontrol exec PATH_TO_DUNE_COMMON/bin/dunepackaging.py --onlysdist

Note that this is executed in the source directory of each module so using an absolute path to dune-common is the easiest approach to guarantee that dunepackaging.py is found. The version string can be changed by providing a --version parameter to dunepackaging.py.

Note: We use scikit-build as glue between the setuptools python module and CMake. So scikit-build must also be available while packaging a module. All files versioned under git are included in the final package.

Local testing

Local testing of the package can be done using

pip install -v --pre --log logfile --find-links file://$PWD/dist dune-foo==version

where dist is a folder containing all the generated dist files. Packages not found there will be downloaded from PyPI. Note that the wheel cache is used (and is crucial for efficient building of the modules) so removing ~/.cache/pip can be required to force pip to use the new packages.

Uploading

If you have an account on PyPI and an active token it is simple to upload packages PyPI.

NOTE: Once uploaded packages cannot be changed any more! Test your packages thoroughly before uploading, e.g., by uploading them to GitLab for testing.

After testing simply running

PATH_TO_DUNE_COMMON/bin/dunepackaging.py --upload pypi

will build the package and upload to PyPI.

Other repositories can be provided: testpypi and gitlab. Like pypi each requires an entry in ~/.pypirc. An example .pypirc could look like this:

[distutils]
  index-servers =
    gitlab
    pypi
[gitlab]
  repository = https://gitlab.dune-project.org/api/v4/projects/812/packages/pypi/
  username = __token__
  password = ....
[pypi]
  username = __token__
  password = ....

Note that the repository URL for GitLab must be stated without the simple at the end.

The DUNE GitLab repositories provide their own PyPI registry and packages can also be uploaded there, provided that the GitLab entry in .pypirc is correct. The GitLab repository is identified through the repository id. In this case 812 points to https://gitlab.dune-project.org/dune-fem/dune-fem-dev. Replace this id with the id from the repository you wish to use. The access token can be generated via Settings --> Access Tokens. Once uploaded the packages can be retrieved using something like

pip install --index-url https://gitlab.dune-project.org/api/v4/projects/812/packages/pypi/simple --extra-index-url https://pypi.org/simple

Note 1: The url after ‘–index-url’ overrides the default index provided by ‘–extra-index-url’ in this case.

Note 2: Observe the extra simple at the end of the urls.

Versioning

The version number specification is parsed from the dune.module file or can be passed as argument --version to the dunepackaging script. If the version specifier contains a -git postfix (e.g. 2.8-git) the postfix is removed and the major.minor.revision version specifier (2.8.0) is appended by .devDATE where DATE is the current day in YYYYMMDD format (2.8.0.dev31012020). Using dune-foo==dev will download the newest developer snapshot while dune-foo=dev2.8.0.dev31012020 will download a specific version. Once release versions are available pip install dune-foo will retrieve the latest stable release.

Dependencies

There are two kinds of dependencies of python packages:

1. Build time dependency (in pyproject.toml)
2. Run time dependency (in setup.py’s install_requires)

The modules given in the dune.module file are used as build/run time dependency as follows:

Versioning of dependencies

If the version specifier contains a -git postfix or a version number was passed as command line argument, the version number is set as maximum version requirement (<=) for all dependent dune modules. This ensures that a published version of a module will not be invalidated by an updated version of some dependency.