Merge branch 'update/python-readme' into 'master'

[python] update readme for new setup

See merge request !3474

(cherry picked from commit c8125fd2)

8dead24a [doxygen] Add filter for lines that should only be displayed in Doxygen
9690d896 [python] update readme for new setup
d5a2162a [doxygen] Add Python readme to doxygen

doc/doxygen/DoxygenDumuxLayout.xml

@@ -16,6 +16,7 @@
     <!-- The modules contains pages for each group and subpages for each subgroup -->
     <!-- Note that extra documentation can be added to groups in separate markdown files -->
     <!-- For this write @addtogroup <groupname> at the top of that markdown page -->
+    <tab type="user" url="@ref python-bindings" visible="yes" title="Python bindings" intro=""/>
     <tab type="modules" visible="yes" title="Module description" intro=""/>
     <tab type="namespaces" visible="yes" title="">
       <tab type="namespacelist" visible="yes" title="" intro=""/>

doc/doxygen/Doxylocal

@@ -9,6 +9,7 @@ PROJECT_NUMBER        = @DUNE_MOD_VERSION@
 INPUT                 += @top_srcdir@/README.md \
                          @srcdir@/pages \
                          @srcdir@/groups \
+                         @top_srcdir@/python/README.md \
                          @top_srcdir@/dumux \
                          @srcdir@/extradoc/parameterlist.txt
 
@@ -25,7 +26,8 @@ FILE_PATTERNS         = *.md,*.cc,*.hh
 EXCLUDE               += @top_srcdir@/dumux/io/format/fmt \
                          @top_srcdir@/dumux/porenetwork/util/README.md
 
-EXAMPLE_PATH          += @srcdir@/extradoc
+EXAMPLE_PATH          += @srcdir@/extradoc \
+                         @top_srcdir@/test/python
 
 IMAGE_PATH             = @srcdir@/images
 

doc/doxygen/markdown-filter.py

@@ -34,6 +34,12 @@ def _is_main_readme_file(path) -> bool:
     return path == MAIN_README
 
 
+def _enable_doxygen_only_content(line: str) -> str:
+    if line.startswith("<!--DOXYGEN_ONLY "):
+        return line.replace("<!--DOXYGEN_ONLY ", "").replace("-->", "")
+    return line
+
+
 assert len(sys.argv[1]) > 1
 filePath = abspath(sys.argv[1])
 
@@ -49,7 +55,7 @@ if _is_main_readme_file(filePath):
 # correctly (may be fixed in the most recent Doxygen version)
 result_lines = []
 for line in result.split("\n"):
-    result_lines.append(_add_header_label(line))
+    result_lines.append(_enable_doxygen_only_content(_add_header_label(line)))
 
 # Print the final result for Doxygen to pick it up
 print("\n".join(result_lines))

python/README.md

-# Python bindings for DuMu<sup>x</sup>
+# Python bindings
 
-The Python bindings have two objectives
+The DuMu<sup>x</sup> Python bindings have two objectives
 * making it possible to use DuMu<sup>x</sup> from Python
 * use Python code in DuMu<sup>x</sup>
 
+The current Python bindings are far from complete and only cover a
+a small subset of DuMu<sup>x</sup> functionality
+(see [test/python](https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/tree/master/test/python)
+for some example Python scripts).
 The binding are experimental until further notice which means
 the API might undergo unannounced changes breaking backwards
 compatibility. Track changes by regularly checking for current merge requests
 and newly merged Dumux commits on GitLab. Python bindings require Python 3.
+Feedback over the mailing list or the issue tracker is highly welcome.
 
-##  Installation
+[TOC]
 
-Python bindings are part of the dune core modules >= version 2.8.
-We recommend to use the Dune modules on the master branch to try out
-Python bindings.
+## Installation of Python bindings
 
-### Example development setup (Dune 2.9-git)
+Using Python bindings require at least Dune core modules version 2.9
+and at least DuMu<sup>x</sup> version 3.7. Python bindings are
+configured **automatically** with `dunecontrol` if Python is found on your system.
+Nothing special needs to be done if you following the installation instructions.
 
-This section is rather new and experimental. Please help improving
-this documentation in the future.
+### Running a test
+
+After configuring and installing DuMu<sup>x</sup> with `dunecontrol`
+(or the installscript) you are ready to run the Python tests.
 
-Checkout the `master` branch of the Dune core modules and DuMu<sup>x</sup>
+From the top-level dumux folder,
+run your first DuMu<sup>x</sup> Python test using the helper script
 
 ```
-git clone https://gitlab.dune-project.org/core/dune-common.git
-git clone https://gitlab.dune-project.org/core/dune-geometry.git
-git clone https://gitlab.dune-project.org/core/dune-grid.git
-git clone https://gitlab.dune-project.org/core/dune-localfunctions.git
-git clone https://gitlab.dune-project.org/core/dune-istl.git
-git clone https://git.iws.uni-stuttgart.de/dumux-repositories/dumux.git
+./build-cmake/run-in-dune-env dumux/test/python/test_gridgeometry.py
+```
+
+See below what this script does and how you can get better control over
+what is exactly happening.
+
+The Python bindings are based on just-in-time compilation of C++ code,
+so the first execution might take considerably longer than subsequent executions.
+In particular, compiling the Dune grid interface may take a few minutes.
 
-cp dumux/cmake.opts .
+### Running all tests
+
+To check that everything works correctly, you can
+also run all currently existing DuMu<sup>x</sup> Python tests with
+```
+cd build-cmake
+ctest -L python
 ```
 
-Enable shared libraries as required for the Python bindings by adding the flag in `cmake.opts` (see
-comments inside the `.opts` file). Not setting this option is often responsible for error messages
-related to parameters not being found.
+## Setup with a Python virtual environment
 
-Create and activate a new virtual environment in which the
-Python modules will be installed in editable mode (symlinked)
+When configuring `dune-common`, by default, an internal Python virtual environment setup is created
+in the build folder of `dune-common` for the Python bindings. Hence, in the description above,
+you were already running the script in a virtual environment. In this virtual environment,
+all Dune module Python bindings are automatically installed
+in editable mode (symlinked) when running `dunecontrol`.
+After running `dunecontrol` the internal virtual environment can be
+activated with
 
 ```
-python3 -m venv venv
-source ./venv/bin/activate
+source ./dune-common/build-cmake/dune-env/bin/activate
 ```
 
-Run dunecontrol which will setup both C++ and Python bindings and modules.
+Then you can run Python tests *without* the helper script like this
 
 ```
-./dune-common/bin/dunecontrol --opts=cmake.opts all
+python3 dumux/test/python/test_gridgeometry.py
 ```
 
-### Running a test
+To have better control of the virtual environment, you can create and
+activate a new virtual environment yourself before running `dunecontrol`.
+The CMake build system of `dune-common` detects the activated virtual environment
+and installs Python bindings into that virtual environment (again in editable mode (symlinked)).
+This looks like this:
 
-Run your first DuMu<sup>x</sup> Python test
+```
+python3 -m venv venv
+source ./venv/bin/activate
+./dune-common/bin/bexec rm -r CMakeFiles CMakeCache.txt
+./dune-common/bin/dunecontrol --opts=dumux/cmake.opts all
+```
+
+This installs everything necessary into your virtual environment `venv`.
+(You may choose any other name of course.) With activated virtual
+environment, you can run Python script using Python bindings like this
 
 ```
 python3 dumux/test/python/test_gridgeometry.py
 ```
 
-The Python bindings are based on just-in-time compilation of C++ code,
-so the first execution might take considerably longer than subsequent executions.
-In particular compiling the Dune grid interface may a few minutes.
+## DuMux Python example
+
+As an example of how to use the Python bindings have a look at this
+[Python script solving a problem with one-phase flow in porous media](https://git.iws.uni-stuttgart.de/dumux-repositories/dumux/-/tree/master/test/python/test_1p.py).
+
+<!--DOXYGEN_ONLY @include test_1p.py-->
+
+
+##  Development of Python bindings
+
+You can install all development requirements with
 
-You can run all currently existing DuMu<sup>x</sup> Python tests with
 ```
-cd dumux/build-cmake
-ctest -L python
+python3 -m pip install -r requirements.txt
 ```
 
-##  Development
+The `requirements.txt` can be found in the DuMux repository in the top level.
+We recommend to use a virtual environment for development as described above.
 
-All Python files should be linted by the tool [`black`](https://pypi.org/project/black/).
+All Python files are linted by the tool [`black`](https://pypi.org/project/black/).
 You can install `black` with `pip install black` and run it from the dumux top-directory
 
 ```
@@ -80,6 +120,7 @@ black ./python
 
 You can also run it on a specific file (replace `./python` by file name)
 This will automatically format the Python files. Run black before every commit changing Python files.
+The CI pipeline prevents merging code that would reformat when running `black`.
 
 The `dumux` Python module should be get a score of `10` from
 the tool [`pylint`](https://pypi.org/project/pylint/).
@@ -89,9 +130,10 @@ You can install `pylint` with `pip install pylint` and run it from the dumux top
 pylint build-cmake/python/dumux
 ```
 
-Pylint needs to be able to check imports so the modules need to be properly set up
-with `setup-python-env.sh` for Dune versions 2.8 or older (see above). The `pylint` configuration file `dumux/.pylintrc` can
+The `pylint` configuration file `dumux/.pylintrc` can
 be used to configure `pylint`. Some exceptions or other parameters than the default
 might be sensible in the future but generally advice given by `pylint` leads to better code.
 Different from `black`, `pylint` does no itself fix the code, you need to do this yourself.
 Always run `black` before checking `pylint`.
+We also run `flake8` on Python code. The CI pipeline automatically checks
+that `pylint` and `flake8` return without warning.