Introduction
This effort aims at ensuring that Maven in Fedora can be bootstrapped, i.e. built and run with the set of current packages, at any time. We aim to preserve this feature with any change in the set of packages.
Part of the motivation for this effort lies in some of the complexities of updating some core packages required by Maven at runtime. In some cases, a change in such a core package led to breaking Maven. When that happened, neither Maven nor most of the required packages could not be rebuilt because they are built with Maven - and at this point Maven does not work.
Solving such cases required manual inspection of the problem and steps like rollback of the change, introducing a temporary change that would ensure working Maven, the required change itself, and a rollback of the temporary change. This approach would have to be discovered with each breaking modification.
This led us to creating the javapackages-bootstrap (JPB) project. It requires only very low level components like the C compiler, core utilities and OpenJDK. The project fetches the upstream core packages required for a working Maven and the build process is simplified not to use maven during its build. The result of the build of JPB is a working set of executables, libraries and configuration files required for javapackages-tools (JPT) to work. A working JPT can be then used to build Java packages in a traditional way in Fedora.
The problem of cyclic dependencies affects a specific set of packages: those that build-require themselves, either directly or transitively by having runtime Requires on other packages which in turn require the original one. Our packages, which have this property, were modified to be able to be built in two modes: non-bootstrap (default) and bootstrap mode. When bootstrap mode is set, it usually means that the package build dependencies are reduced to JPB and the build instructions are set to invoke xmvn from bootstrap rather than the package xmvn. This means we are able to avoid circular dependencies among our packages.
MBI (Maven Bootstrap Initiative) Continuous Integration
To ensure that modifying packages retains the bootstrapping property of Maven in Fedora, a specific test was implemented.
Simply put, It builds JPB, then the rest of the packages in bootstrap mode and then all the packages again in non-bootstrap mode.
When a package that implements bootstrap mode or is an Ant dependency, is changed, MBI CI ensures that Maven and Ant in Fedora will work with the modified version. The test is time-consuming and resource-intensive. The test is currently triggered manually for pull requests. There are plans to set these tests as gating for our set of packages. On top of that, the tests should be run regularly to ensure that external modifications to our buildroot, for example changes in OpenJDK do not break the bootstrap property.
When the test succeeds, it means that all the packages that were tested can be built from scratch with no pre-existing Java packages except for OpenJDK.
The tool used for testing generates test reports. These are useful in case of failed tests. In such a situation, the report should be linked as a pull request comment. The report should contain information about the first component that failed to build. This can be then used to fix the failed build.
An example can be seen on this ant pull request: the test named “MBI - Maven Bootstrap Initiative”.
Executing the test
In order to run the test yourself, you need to prepare a Test Subject XML document containing the URLs of dist-git repositories of the Fedora Maven packages. Other XML documents: Build Plan and Build Platform are available in this repository. A script that can be used to generate the Test Subject from the Build Plan is available here as subject-from-fedora-prs.py.
Example of how to create a Test Subject from the latest packages in Fedora Rawhide:
./mbici-fedora-pr/subject-from-fedora-prs.py -plan ./mbici-config/plan/bootstrap-maven-rawhide.xml > ./mbici-fedora-pr/test-subject-rawhide.xml
If you wish to create a Test Subject containing URLs of some pull requests you can either add the URLs separated by space as additional parameters and recreate the Test Subject or you can edit the XML document yourself by modifying the <commit> node of the respective package.
Then follow the instructions of the mbi-workflow. Build:
./mbici-workflow/build.sh
Generate workflow document:
./mbici-workflow/target/mbici-wf generate \
-plan ./mbici-config/plan/bootstrap-maven-rawhide.xml \
-platform ./mbici-config/platform/rawhide-jdk.xml \
-subject ./mbici-fedora-pr/test-subject-rawhide.xml \
-workflow ${workspace}/mbici-wf.xml
Execute the test (very time consuming, may take from half an hour up to more than an hour):
./mbici-workflow/target/mbici-wf run \
-workflow ${workspace}/mbici-wf.xml \
-resultDir ${workspace}/mbici-result \
-cacheDir ${workspace}/mbici-cache \
-workDir ${workspace}/mbici-work
If the test fails, relevant logs are stored inside resultDir in the particular directory with the corresponding name. This includes files created during a normal rpm build. After modifying the files, you can rerun the run command and it will continue from where the test failed previously.
After the test is finished, you may want to generate an HTML report, that can be done by:
./mbici-workflow/target/mbici-wf report \
-plan ./mbici-config/plan/bootstrap-maven-rawhide.xml \
-platform ./mbici-config/platform/rawhide-jdk.xml \
-subject ./mbici-fedora-pr/test-subject-rawhide.xml \
-workflow ${workspace}/mbici-wf.xml \
-resultDir ${workspace}/mbici-result \
-reportDir ${workspace}/mbici-report
The generated report is the same that is used as the result of CI when a pull request is tested.