Installation Dune and its Python bindings from PyPi

Download and install DUNE modules from PyPI

We have added installation support of Dune modules via pip using the Python package index (PyPI). The module and the modules it depends on are downloaded, build, and installed either into the system or into a python virtual environment (the latter might be of more interest as it is an option to install Dune without root privileges). This could also be of interest for C++ users of Dune as Python bindings are not mandatory. Scripts like dunecontrol and duneproject can be used the same way as with a system installation so that installed and source modules can easily be used together.

Starting with the 2.8 release, we are planning to upload stable release versions as well as regular developer snapshots of all core modules. That should make it easy for users to work with an installed Dune which can be updated to a developer version if a newer feature is required.

Installing the main core modules (common, geometry, and grid) can be easily installed into a virtual environment without requiring root prviliges by running

python3 -m venv ~/dune-env
source ~/dune-env/bin/activate
pip install dune-grid

You should now be able to test with one of your own Dune modules or you can run a Python test:

python -m dune.grid
cd grid_tutorial
python example.py

Simply remove ~/dune-env to get rid of everything.

All core modules are available on the Python Package Index (pypi), as well as some additional grid managers (e.g. pip install dune-alugrid), and the discretization module dune-fem.

We have tested on a number of Linux distributions and on MacOS. Please report if you encounter any problems with the setup.

Know issue

If some software component was found through the environment variable CMAKE_PREFIX_PATH then there might be problems later when loading some shared libraries requiring to add those paths also the LD_LIBRARY_PATH.

Adding support to existing modules

Packaging and uploading any Dune module to PyPi can be easily done using the new dunepackaging.py script in dune-common/bin.

Preparing the module

To provide sufficient metadata for the package additional field should be added to dune.module. These fields are:

Author: S. Body
Description: Some dune mdoule
URL: e.g. gitlab link
Python-Requires: e.g. numpy scipy

The last line is only required in case Python bindings are available that require additional packages to be installed. Here the usual Python syntax can be used, e.g., numpy==1.19.4. The string for the description entry can be taken from the pc.in file and replaced there with the line

Description: @DESCRIPTION@

To generate the distribution package run

PATH_TO_DUNE_COMMON/bin/dunepackaging.py --onlysdist

this generate the files setup.py, pyproject.toml and the folder dist containing the tar.gz file ready for upload to pypi.

Using dunecontrol packages can be build an uploaded for all available source repos by executing

PATH_TO_DUNE_COMMON/bin/dunecontrol exec PATH_TO_DUNE_COMMON/bin/dunepackaging.py --onlysdist

The version string can be changed by providing a --version parameter to dunepackaging.py.

Local testing

Local testing of the package can be done using

pip install -v --log logfile --find-links file://$PWD/dist dune-foo==version

where dist is a folder containing all the generated dist files. Packages not found there will be downloaded from PyPi. Note that the wheel cache is used (and is crucial for efficient building of the modules) so removing ~/.cache/pip can be required to force pip to use the new packages.

Uploading

If you have an account on PyPi and an active token it is simply to upload packages pypi. NOTE: Once uploaded packages cannot be removed or changed so test packages before a upload to gitlab for testing. After a testing simply running

PATH_TO_DUNE_COMMON/bin/dunepackaging.py --upload pypi

will build the package and upload to pypi. Other repositories can be provided: testpypi and gitlab - like pypi each requires an entry in ~/.pypirc. An example .pypirc could look like this (note the repository url for gitlab without the simple at the end):

[distutils]
  index-servers =
    gitlab
    pypi
[gitlab]
  repository = https://gitlab.dune-project.org/api/v4/projects/812/packages/pypi/
  username = __token__
  password = ....
[pypi]
  username = __token__
  password = ....

Note that the Dune gitlab repositories provide their own PyPi registry so packages can also be uploaded there, provided that the gitlab entry in .pypirc is correct. The gitlab repository is identified through the repository id, 812 in this case which points to https://gitlab.dune-project.org/dune-fem/dune-fem-dev. Replace this id with the id from the repository you wish to use. The access token can be generated via Settings --> Access Tokens. Once uploaded the packages can be retrieved using something like

pip install --index-url https://gitlab.dune-project.org/api/v4/projects/812/packages/pypi/simple --extra-index-url https://pypi.org/simple

Note 1: The url after ‘–index-url’ overrides the default index provided by ‘–extra-index-url’ in this case. Note 2: Observe the extra simple at the end of the urls.

Versioning

The version number specification is parsed from the dune.module file or can be passed as argument --version to the dunepackaging script. If the version specifier contains a -git postfix (e.g. 2.8-git) the postfix is removed and the major.minor.revision version specifier (2.8.0) is appended by .devDATE where DATE is the current day in YYYYMMDD format (2.8.0.dev31012020). Using dune-foo==dev will download the newest developer snapshot while dune-foo=dev2.8.0.dev31012020 will download a specific version. Once release version are available pip install dune-foo will retrieve the latest stable release.

Dependencies

There are two kinds of dependencies of python packages:

  1. Build time dependency (in pyproject.toml)
  2. Run time dependency (in setup.py’s install_requires)

The modules given in the dune.module file are used as build/run time dependency as follows:

dune.module pyproject.toml     setup.py
Depends x x
Suggests
Python-Requires:    x x

Versioning of dependencies

If the version specifier contains a -git postfix or a version number was passed as command line argument, the version number is set as maximum version requirement (<=) for all dependent dune modules. This ensures that a published version of a module will not be invalidated by an updated version of some dependency.

More information

We use scikit-build as glue between the setuptools Python module and CMake. So scikit-build must also be available while packaging a module. All files versioned under git are included in the final package.

Creative Commons License   |  Legal Statements / Impressum  |  generated with Hugo v0.80.0 (Jun 24, 22:25, 2021)