Adding and building a module for Fedora
This document describes the process of adding a new module to the Fedora Modularity project, how to build it locally and how to build it in Fedora infrastructure
Process and policy for how to add a module to Fedora
Adding a module repository is a manual process atm. Find someone from the release-engineering group to add the new repository and give you write access. Later on this will be automated by MBS, the module build server, but this is still being worked on.
Writing a new modulemd file
A modulemd file is a yaml file that contains the module metadata like description, license and dependencies. The sample file in the upstream git repository of modulemd contains a complete documentation of the required and optional yaml tags.
The Modularity team uses a shorter modulemd file to test builds, but it can also be used as a base for new modules. Another good example is base-runtime.yml
Lets use the vim modulemd as an example for this document. It is in the /home/karsten/Modularity/modules/vim/ directory on my system.
document: modulemd
version: 1
data:
summary: The best text editor and IDE
description: The classic, extensible text editor, the legend.
license:
module: [ MIT ]
dependencies:
buildrequires:
base_runtime: master
requires:
base_runtime: master
references:
community: https://fedoraproject.org/wiki/Modularity
documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
tracker: https://taiga.fedorainfracloud.org/project/modularity
profiles:
default:
rpms:
- vim-enhanced
- vim-common
- vim-filesystem
minimal:
rpms:
- vim-minimal
api:
rpms:
- vim-common
components:
rpms:
vim:
rationale: Provides API for this module
ref: f25
buildorder: 10
generic-release:
rationale: build dependency
ref: f25
perl-Carp:
rationale: build dependency
ref: f25
gpm:
rationale: build dependency
ref: f25
perl-Exporter:
rationale: build dependency
ref: f25
All dependencies of vim need to be listed under components/rpms, except those that are already included in Base Runtime. Here's how you can get the list of vim dependencies that are not in Base Runtime:
wget https://raw.githubusercontent.com/asamalik/fake-base-runtime-module-image/master/packages/gen-core-binary-pkgs.txt
for i in `repoquery --requires --recursive --resolve --qf "%{SOURCERPM}\n" \
vim-enhanced vim-minimal vim-common vim-filesystem \
| sed -e "s/-[^-]*-[^-]*$//" | sort -n | uniq` ; do
grep -wq $i gen-core-binary-pkgs.txt || echo $i
done
verifying the syntax of the new modulemd file and bug fixing
Once the modulemd file is finished, it is a good idea to check if there any errors in the yaml syntax. The check_modulemd program checks modulemd files for errors. You need to install some packages to use this:
- python2-aexpect - dependency for python-avocado
- python2-avocado - avocado testing framework
- python2-modulemd - Module metadata manipulation library
- python-enchant - spell checker library (needed only for check_modulemd.py)
- hunspell-en-US - English dictionary (needed only for check_modulemd.py)
Then run
./run-checkmmd.sh /home/karsten/Modularity/modules/vim/vim.yaml
and check the output for errors:
Running: avocado run ./check_modulemd.py --mux-inject 'run:modulemd:/home/karsten/Modularity/modules/vim/vim.yaml' JOB ID : 51581372fec0086a50d9be947ea06873b33dd0e5 JOB LOG : /home/karsten/avocado/job-results/job-2017-01-19T11.28-5158137/job.log TESTS : 11 (01/11) ./check_modulemd.py:ModulemdTest.test_debugdump: PASS (0.16 s) (02/11) ./check_modulemd.py:ModulemdTest.test_api: PASS (0.15 s) (03/11) ./check_modulemd.py:ModulemdTest.test_components: PASS (0.16 s) (04/11) ./check_modulemd.py:ModulemdTest.test_dependencies: WARN (0.02 s) (05/11) ./check_modulemd.py:ModulemdTest.test_description: PASS (0.16 s) (06/11) ./check_modulemd.py:ModulemdTest.test_description_spelling: PASS (0.16 s) (07/11) ./check_modulemd.py:ModulemdTest.test_summary: PASS (0.16 s) (08/11) ./check_modulemd.py:ModulemdTest.test_summary_spelling: WARN (0.02 s) (09/11) ./check_modulemd.py:ModulemdTest.test_rationales: ERROR (0.04 s) (10/11) ./check_modulemd.py:ModulemdTest.test_rationales_spelling: PASS (0.16 s) (11/11) ./check_modulemd.py:ModulemdTest.test_component_availability: WARN (0.02 s) RESULTS : PASS 7 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 3 | INTERRUPT 0 TESTS TIME : 1.20 s
So this isn't quite right yet, lets have a look at the logfile mentioned in the output.
grep -i error /home/karsten/avocado/job-results/job-2017-01-19T11.28-5158137 .... TestError: Rationale for component RPM generic-release should end with a period: build dependency
It seems that rationales need to end with a period. Change all those lines so that they look like this:
vim:
rationale: Provides API for this module.
ref: f25
buildorder: 10
generic-release:
rationale: build dependency.
ref: f25
gpm:
rationale: build dependency.
ref: f25
perl:
rationale: build dependency.
ref: f25
perl-Carp:
rationale: build dependency.
ref: f25
perl-Exporter:
rationale: build dependency.
ref: f25
Another run of check_modulemd.py completes without errors.
Building the module locally
The build_module script from the build-module repository on github makes local module builds really easy. It sets up the environment and then builds a module and its components locally with mock. One requirement is to have docker installed and running on your system. It is also required that the name of the new modulemd file, the repository name of that module and the name of the module itself match in order to use the build_module script. As build_module builds the latest commit in the master branch of the module git repository, changes need to be checked into git, a push to upstream (dist-git) is not required at this stage.
The basic usage of build_module is
./build_module /home/karsten/Modularity/modules/vim/ /tmp/results
This will download a container with the Module build service and rebuild the dependencies that are listed in the modulemd file. This step can take quite some time, depending on the module and how many components need to be built.
When build_module is done there will be a couple of rebuilt rpms in /tmp/results/module-vim-master-*:
find /tmp/results/module-vim-master-* -name "*.rpm" /tmp/results/module-vim-master-20170119120233/results/vim-8.0.206-1.fc25.src.rpm /tmp/results/module-vim-master-20170119120233/results/vim-debuginfo-8.0.206-1.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/vim-X11-8.0.206-1.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/vim-filesystem-8.0.206-1.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/vim-enhanced-8.0.206-1.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/vim-minimal-8.0.206-1.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/vim-common-8.0.206-1.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/perl-Carp-1.40-365.fc25.src.rpm /tmp/results/module-vim-master-20170119120233/results/perl-Carp-1.40-365.fc25.noarch.rpm /tmp/results/module-vim-master-20170119120233/results/perl-Exporter-5.72-366.fc25.src.rpm /tmp/results/module-vim-master-20170119120233/results/perl-Exporter-5.72-366.fc25.noarch.rpm /tmp/results/module-vim-master-20170119120233/results/gpm-1.20.7-9.fc25.src.rpm /tmp/results/module-vim-master-20170119120233/results/gpm-debuginfo-1.20.7-9.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/gpm-static-1.20.7-9.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/gpm-devel-1.20.7-9.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/gpm-libs-1.20.7-9.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/gpm-1.20.7-9.fc25.x86_64.rpm /tmp/results/module-vim-master-20170119120233/results/generic-release-25-1.src.rpm /tmp/results/module-vim-master-20170119120233/results/generic-release-notes-25-1.noarch.rpm /tmp/results/module-vim-master-20170119120233/results/generic-release-25-1.noarch.rpm /tmp/results/module-vim-master-20170119120233/results/module-build-macros-0.1-1.module_vim_master_20170119120233.src.rpm /tmp/results/module-vim-master-20170119120233/results/module-build-macros-0.1-1.module_vim_master_20170119120233.noarch.rpm <pre> These will now be put into a container. == putting the packages into a container == For this step you'll need to create a rpm repository of the new packages. <pre> cd /tmp/results/module-vim-master-20170119120233/ mkdir vim-module-repo cp results/*.rpm vim-module-repo cd vim-module-repo createrepo .
The /tmp/results/module-vim-master-20170119120233/vim-module-repo need to uploaded somewhere public so that docker can access it. O good place for that is p.e. fedorapeople
scp -r /tmp/results/module-vim-master-20170119120233/vim-module-repo karsten.fedorapeople.org:public_html/
You'll also need a dnf/yum configfile (/home/karsten/Modularity/modules/vim/vimmodule.repo) that points at this new repo:
cat /home/karsten/Modularity/modules/vim/vimmodule.repo [vimmodule] name=VIM module failovermethod=priority baseurl=https://karsten.fedorapeople.org/vim-module-repo/ enabled=1 metadata_expire=7d gpgcheck=0 skip_if_unavailable=True
Now put everything into a Dockerfile. We're using Adama Samalik's fake-gen-core-module as there is no usable base-runtime module yet:
cat /home/karsten/Modularity/modules/vim/Dockerfile FROM asamalik/fake-gen-core-module ADD vimmodule.repo /etc/yum.repos.d/vimmodule.repo RUN dnf -y install vim-enhanced
Building the module in Fedora infrastructure
This step uses a local module-build-service and other components in containers and passes results on to the Fedora staging infrastructure. A checkout of the module-build-service is required. Change into your local copy of this repository and run
docker-compose down docker-compose up --build
This will start the module-build-service frontend and scheduler as well as fedmsg and you'll see when messages about module builds are coming in over the Federated Message Bus.