Download and install DUNE modules from PyPI
We have added installation support of Dune modules via
pip using the Python package
index (PyPI). The module and the modules it depends on are downloaded, build, and
installed either into the system or into a python virtual environment (the latter
might be of more interest as it is an option to install Dune without root privileges).
This could also be of interest for C++ users of Dune as Python bindings are not mandatory.
duneproject can be used the same way as with a system installation so that
installed and source modules can easily be used together.
Starting with the 2.8 release, we are planning to upload stable release versions as well as regular developer snapshots of all core modules. That should make it easy for users to work with an installed Dune which can be updated to a developer version if a newer feature is required.
Installing the main core modules (
grid) can be
easily installed into a virtual environment without requiring root
prviliges by running
python3 -m venv ~/dune-env source ~/dune-env/bin/activate pip install dune-grid
You should now be able to test with one of your own Dune modules or you can run a Python test:
python -m dune.grid cd grid_tutorial python example.py
~/dune-env to get rid of everything.
core modules are available on the Python Package Index (pypi), as well
as some additional grid managers (e.g.
pip install dune-alugrid),
and the discretization module
We have tested on a number of Linux distributions and on MacOS. Please report if you encounter any problems with the setup.
If some software component was found through the environment variable
then there might be problems later when loading some shared libraries requiring to add those paths also the
Adding support to existing modules
Packaging and uploading any Dune module to PyPi can be easily done
using the new
dunepackaging.py script in
Preparing the module
To provide sufficient metadata for the package additional field should be added to
dune.module. These fields are:
Author: S. Body Description: Some dune mdoule URL: e.g. gitlab link Python-Requires: e.g. numpy scipy
The last line is only required in case Python bindings are available that require additional packages to be installed. Here the usual Python syntax can be used, e.g.,
The string for the description entry can be taken from the
pc.in file and replaced there with the line
To generate the distribution package run
this generate the files
setup.py, pyproject.toml and the folder
dist containing the
tar.gz file ready for upload to
dunecontrol packages can be build an uploaded for all available source repos by executing
PATH_TO_DUNE_COMMON/bin/dunecontrol exec PATH_TO_DUNE_COMMON/bin/dunepackaging.py --onlysdist
The version string can be changed by providing a
--version parameter to
Local testing of the package can be done using
pip install -v --log logfile --find-links file://$PWD/dist dune-foo==version
dist is a folder containing all the generated
dist files. Packages not found there will be downloaded from PyPi.
Note that the wheel cache is used (and is crucial for efficient building of the modules) so removing
~/.cache/pip can be required to force
pip to use the new packages.
If you have an account on PyPi and an active token it is simply to upload packages pypi. NOTE: Once uploaded packages cannot be removed or changed so test packages before a upload to gitlab for testing. After a testing simply running
PATH_TO_DUNE_COMMON/bin/dunepackaging.py --upload pypi
will build the package and upload to pypi.
Other repositories can be provided:
gitlab - like
pypi each requires an entry in
.pypirc could look like this (note the repository url for gitlab without the simple at the end):
[distutils] index-servers = gitlab pypi [gitlab] repository = https://gitlab.dune-project.org/api/v4/projects/812/packages/pypi/ username = __token__ password = .... [pypi] username = __token__ password = ....
Note that the Dune
gitlab repositories provide their own
PyPi registry so packages can also be uploaded there, provided that the gitlab entry in
.pypirc is correct. The gitlab repository is identified through the repository id,
812 in this case which points to https://gitlab.dune-project.org/dune-fem/dune-fem-dev. Replace this id with the id from the repository you wish to use. The access token can be generated via
Settings --> Access Tokens. Once uploaded the packages can be retrieved using something like
pip install --index-url https://gitlab.dune-project.org/api/v4/projects/812/packages/pypi/simple --extra-index-url https://pypi.org/simple
Note 1: The url after ‘–index-url’ overrides the default index provided by ‘–extra-index-url’ in this case.
Note 2: Observe the extra
simple at the end of the urls.
The version number specification is parsed from the
dune.module file or can be passed as argument
--version to the dunepackaging script.
If the version specifier contains a
-git postfix (e.g. 2.8-git) the postfix is removed and the major.minor.revision version specifier (2.8.0) is appended by
DATE is the current day in YYYYMMDD format (2.8.0.dev31012020).
dune-foo==dev will download the newest developer snapshot while
dune-foo=dev2.8.0.dev31012020 will download a specific version.
Once release version are available
pip install dune-foo will retrieve the latest stable release.
There are two kinds of dependencies of python packages:
- Build time dependency (in
- Run time dependency (in
The modules given in the dune.module file are used as build/run time dependency as follows:
Versioning of dependencies
If the version specifier contains a
-git postfix or a version number was passed as command line argument, the version number is set as maximum version requirement (
<=) for all dependent dune modules. This ensures that a published version of a module will not be invalidated by an updated version of some dependency.
We use scikit-build as glue between the setuptools Python module and CMake. So
scikit-build must also be available while packaging a module. All files versioned under git are included in the final package.