Use gitlab ci for automated testing
It could be useful to integrate build and test jobs into our ci such that one can conveniently let the runner test everything in a merge request. That makes it less probable that stuff gets merged that breaks anything.
I would suggest to create pipelines for commits to master, tags and merge requests. Regarding automatic start of the pipelines, I would suggest the following strategy:
- add manual trigger for build & test pipeline for merge requests. Especially on Dumux days, when there is a lot of pushing to open MRs, we would trigger a lot of unneccesary pipeline runs, I guess. I would prefer a manual trigger here.
- automatic start of the pipeline for master and tags. The first is useful as you always see which commit destroyed the pipeline on master. This is of course a bit redundant as one maybe already triggered and verified a successful pipeline on the branch just before merging. In some cases, it could still break on master if someone didn't rebase before running the pipeline. For safety reasons, I think it could be nice to always test master.
I am thinking of some default setup (Dune2.7, release opts) to be used, but we could schedule one nightly build with different setups as on buildbot. However, that is basically currently done on buildbot, so this would only make sense if we want to stop using buildbot at one point. So, this is probably irrelevant for now.
Another point is that our build and test times are quite long, and it would be nice to build and run only those tests that are necessary, at least for MRs. Here's a first idea for merge requests:
- 1 use
git diff-tree -r --name-only $CI_COMMIT_SHA $CI_MERGE_REQUEST_TARGET_BRANCH_SHA
to list the modified files - 2 use
ctest -N
to list all tests we currently have - 3 do something like in
extractmodulepart.sh
to determine required headers of the tests - 4 select those tests which use any of the files obtained in (1)
- 5 compile and run those tests only
point (2) and (3) might be expensive. We could also store that info in a json file or so, but on the other hand, that file would need to be up-to-date, and maybe someone adds a test in the MR thats not in the file. Or, maybe someone adds a dependency with the MR. For now, I'd opt for always doing that, should be much less expensive than building and running a lot of tests.
Docker images
Dune's Docker images for testing are built here: https://gitlab.dune-project.org/docker/ci
It would be nice to reuse some of the base images they use there. Also I propose a dumux-docker-ci
project that
- automatically builds the dumux test images in GitLabCI (services images from Dune, e.g.
buildah
can be hopefully reused for that purpose) - pushes the images to the
dumux-docker-ci
container registry (see gitlab-ci.yaml in the dune project) - reuse base images from Dune if possible
BuildBot
Our current images are in https://git.iws.uni-stuttgart.de/timok/dumux-bb-ci and our current buildbot setup is here: https://git.iws.uni-stuttgart.de/timok/buildbot-master
Multi module setup
After new commits in dumux we also want to trigger builds of dumux-lecture (maybe dumux-course). There is also (a bit outdated) dumux-testing with the first system test for dumux. See !2396 (closed).
Coverage reports
The coverage report is generated here in a weekly build. It already uses GitlabCI. https://git.iws.uni-stuttgart.de/dumux-repositories/dumux-coverage