Using Mock to test package builds

This page needs some love
This page should be revised or reconstructed to be more helpful. Problems may include being out of step with current team or project status or process.

Following chapters are no longer applicable for recent Fedora releases.
Mock now uses DNF instead of YUM, therefore some information is outdated

This page is a collection of bits people have found useful when working with mock. A more up-to-date version of this information is in the Mock user documentation.

What is Mock?

Mock takes a srpm and builds it in a chroot. This ensures that your BuildRequires lines are correct, that there are no missing dependencies, and that it builds cleanly. The Mock user documentation has more information on Mock.

How do I set up Mock?

First, install the mock package, which is available from Fedora Package Collection.

Default configuration

When using mock, you can use the "-r" option to select a particular test configuration. If there's one configuration that system users will normally use, you can set that as the default:

# cd /etc/mock
# ln -s --force SOMECONFIG.cfg default.cfg

(An example of SOMECONFIG.cfg is "fedora-14-x86_64.cfg"; see the /etc/mock directory for the existing mock configurations.)

Setting up local mirror

If you want to set up a local mirror, refer Docs/Drafts/MockSetupUsingLocalMirror

Proxy Configurations

If you're behind a proxy, add the proxy info to the desired cfg file in /etc/mock in the same format that you would add it to your yum.conf. See man yum.conf for details on how to do it.

How do I use Mock?

Mock accepts a number of commands. Mock accepts commands in old-style format (e.g., "rebuild" or "init" without a leading "--"), but these are deprecated; use commands with a leading "--" instead.

Using Mock outside your git sandbox

Add your user name to the mock group

sudo usermod -a -G mock myusername

Create your srpm using 'rpmbuild -bs'. Then change to the directory where your srpm was created.

Now you can start mock with

mock -r <configfile> --rebuild package-1.2-3.src.rpm

where <configfile> is the name of a configuration from /etc/mock/ (do not include the .cfg extension). Passing -r is not necessary if you have set a default config file via a symlink as shown above. --rebuild is the default option. So the following would work equally well

mock package-1.2-3.src.rpm

The --verbose option is useful if you want more output to be shown for debugging purposes.

If using mock version older than 0.8.8 or on a system with python 2.4, and building i386 packages on x86_64, prepend setarch i386 to the mock command line:

setarch i386 mock -r <configfile> --rebuild package-1.2-3.src.rpm

Newer versions of mock no longer need the setarch command, although it does not hurt anything if it is there.

Using Mock inside your git sandbox

You only need to type 'fedpkg mockbuild' to start a mock build. The used architecture depends on the directory where you start the mock build.

Building packages that depend on packages not in a repository

If you're building some package P that depends on another package Q, and Q is not in a repository, you can still use mock. Here's one way to do that.

First, initialize the mock repository:

mock -r MOCK_CONFIG --init

Note: If you get the error "Error: There are no enabled repos." then this is because (like me) you're on current Fedora and the fake O/S inside the chroot is attempting downloads using yum. In that case you have to put the "--dnf" to force it to use DNF.

mock -r MOCK_CONFIG --dnf --init

(An example of MOCK_CONFIG is "fedora-14-x86_64"; see the /etc/mock directory for the existing mock configurations.)

Install the packages you need to build the program (named in BuildRequires) from the yum repositories and/or local RPMs. In practice, you may need to do a number of these (for the different package dependencies); you can list multiple packages on the same line:

mock -r MOCK_CONFIG --install PACKAGE_NAME_OR_PATH_TO_RPM

Current versions of mock should permit you to now build from the source RPM. If that does not work then the "--copyin" method below should be used. The --no-clean option is necessary since by default the chroot is cleaned prior to building.

mock -r MOCK_CONFIG --no-clean /PATH/TO/SRPM

Copy in the source RPM into /tmp (we'll copy in and do a build inside the shell, to work around the checks that detect that the packages aren't in the repository). Do not copy into /tmp directory, because it gets cleared before other mock command:

mock -r MOCK_CONFIG --copyin /PATH/TO/SRPM /var/tmp

Shell into the mock environment and perform the build:

mock -r MOCK_CONFIG --shell
cd
rpmbuild --rebuild /var/tmp/SRPM_NAME

Speeding up

• /etc/mock/site-defaults.cfg
  • Disable the package state plugin:

    config_opts['plugin_conf']['package_state_enable'] = False

  • Parallelize builds:

    config_opts['macros']['%_smp_mflags'] = "-j17"

  • Improve ccache hits between package version builds (may break -debuginfos!):

    config_opts['files']['etc/profile.d/zz-local.sh'] = """
    unset CCACHE_HASHDIR
    """

  • Compress ccache:

    config_opts['plugin_conf']['ccache_opts']['compress'] = True

  • Use lzo compression for root cache:

    config_opts['plugin_conf']['root_cache_opts']['compress_program'] = "lzop"
    config_opts['plugin_conf']['root_cache_opts']['extension'] = ".lzo"

• Per-repo config files
  • Parallelize (de)compression within rpmbuild:

    config_opts['chroot_setup_cmd'] = 'install @buildsys-build /usr/bin/pigz /usr/bin/lbzip2'
    config_opts['macros']['%__gzip'] = '/usr/bin/pigz'
    config_opts['macros']['%__bzip2'] = '/usr/bin/lbzip2'

Using scan-build for unmodified packages

• Add /usr/bin/scan-build to per-chroot config files' chroot_setup_cmd for which you want to use it, for example:

config_opts['chroot_setup_cmd'] = 'install @buildsys-build /usr/bin/scan-build'

• Set up a shell alias and use it instead of plain mock (everything on one line):

alias mock-scan-build="mock --define '__scan_build /usr/bin/scan-build' --define '_configure %__scan_build ./configure' --define '__cmake %__scan_build %{_bindir}/cmake' --define '__make %__scan_build %{_bindir}/make' --define '___build_template #!%{___build_shell}\\"$'\n'"alias make=\"%__make\" cmake=\"%__cmake\"\\"$'\n'"%{___build_pre}\\"$'\n'"%{nil}'"

• See generated reports in the chroot's /tmp

Caching in mock 0.8.x and later

Mock 0.8.x introduces three different types of caches: 1) root cache, 2) yum cache, and 3) ccache. These caches greatly speed up mock.

Look in /etc/mock/defaults.cfg for documentation on configuring each of these caches. You can configure the directory where they are stored, the maximum age of the caches, and (for ccache only) the max disk usage for the cache.

Root cache

Starting with mock 0.5, mock can automatically cache the standard buildroot for each environment in a local tar file (/var/cache/mock/$CONFIG/root_cache/*). It then unpacks this tar file to populate the buildroot, rather than downloading the buildroot RPMs fresh each time. After unpacking the buildroot from the root cache, mock does a yum update to make sure that the buildroot is up-to-date prior to installing additional BuildRequires for a package build.

The root cache is enabled by default in mock 0.8.x and later. To disable, see the documentation in /etc/mock/defaults.cfg. The root cache can be disabled globally or per-chroot. The root caches are automatically deleted and recreated every 15 days to prevent them from being too stale.

Keep in mind that the root cache can affect reproducibility of your build, especially in environments where the set of base RPMs is updated often (e.g. rawhide). Disable the root cache in environments where build reproducibility is of utmost importance.

yum cache

By default, yum stores downloaded RPMs in a directory under /var/cache/yum. The yum cache feature bind mounts the chroot /var/cache/yum to a common directory outside the chroot environment (such as /var/cache/mock/$CONFIG/yum_cache/) where it can be saved and re-used by subsequent builds. This ensures that yum does not need to download each RPM fresh over the network for every build. This feature is enabled by default.

ccache

The ccache tool is a utility that wraps calls to compilers such as 'gcc' and caches the output. When called to compile the same program a second time (with the same cmdline args, gcc-version, and header files), ccache will instead pull the cached version rather than running the compiler afresh.

Using Squid to Speed Up Mock package downloads

The cached buildroots trick above is much faster than using squid alone, but you can also use squid in addition to the cached buildroots, particularly if you are at the slow end of an Internet connection.

The root cache will contain the standard buildroot packages, but as it gets older, the yum update after unpacking it may have to update several packages. In addition, some packages aren't in the standard buildroot, but are often needed (e.g. pkgconfig). These can be faster served from a squid cache than from downloading them from the Internet for each package build.

Install the squid package which is available from Fedora Package Collection.

Some items worth changing in /etc/squid/squid.conf:

• Maximum Object Size. Normally, this is 4MB. However, some of the base packages like glibc can be larger than 16MB, and if you're building say Eclipse plugins, that's 60MB. So make it large.

maximum_object_size 409600 KB

• Cache Replacement Policy. Normally, lru is fine. But in our case, we want to keep the largest objects around longer, and just download the smaller objects if we can.

cache_replacement_policy heap LFUDA

• Cache Size. Normal size is 100MB. This isn't going to cut it. This needs to be at least 1GB, more if you regularly build for multiple Fedora releases and multiple architectures.

cache_dir ufs /var/spool/squid 1000 16 256

Then start squid, and configure it to start on startup.

To make the build user automatically use the squid cache, type:

echo export http_proxy=\"http://localhost:3128\" >> /home/build/.bash_profile

If you only want to use squid for particular mock configurations (perhaps you have local mirrors of some distributions and don't need squid for those), you can, instead of setting the http_proxy environment variable, set up the proxy configuration in the yum.conf part of each mock configuration file:

config_opts['yum.conf']  = """
[main]
...
proxy=http://localhost:3128/

...
"""

Edit /etc/mock/*.cfg to not use mirror lists

Don't use the mirrorservice.org mirror; they prevent downstream caching via the HTTP header "Cache-Control: no-store".

If you're using squid, you don't want mock+squid to use a random mirror for downloading packages. Change the config files in /etc/mock/*.cfg to use a single fast-for-you mirror in a baseurl entry, and don't use mirrorlists. You can use this script for that.

Using tmpfs to avoid disk access

It's not clear that tmpfs is a huge win anymore, now that autocache works and is a standard feature.

When having enough RAM (1.5-2 GB), using a tmpfs can speedup the 'prep' phase significantly. With 1.5GB RAM on a Celeron 2.8 GHz, populating the buildroot will take 2-5s (seconds) on subsequent builds.

You can either mount /var/lib/mock as a tmpfs, or you can mount each of the build directories under /var/lib/mock as separate tmpfs file systems. The latter lets you use the autocache above and keeps the autocache files on disk, rather than in the tmpfs. If you erase and recreate the autocaches often, keeping them in tmpfs makes sense. If you don't recreate them often, then you should use per-build-directory tmpfs mounts instead.

Using mock under SELinux

Using SELinux in conjunction with chroot-ed environments such as mock presents a number of problems, such as needing to replicate all default file contexts with the chroot directory prefix, which is an administrative nightmare. The approach used by mock is very simple, though it only works with the targeted policy: preload a dummy libselinux that makes all child processes think that SELinux is disabled. With mock itself running as an unconfined process, this generally works well.

Problems with SELinux memory protection

However, from FC5 onwards, the SELinux unconfined_t domain is no longer fully unconfined by SELinux; memory protection is implemented to protect most user processes against a variety of possible exploits. This presents at least two sets of problems when using mock in FC5:

1. When building packages for legacy distributions , the build process can involve loading old shared libraries that do not have separate sections for, say, executable code and writable data. This leads to execmod denials when a process running in the unconfined_t domain tries to load such a library, unless the library is labelled textrel_shlib_t.
2. When building packages using mono (the same might apply to java), the mono process would normally run in its own less-constrained domain (mono_t); when running under mock, mono runs in the unconfined_t domain, which can result in execstack and execheap denials.

Whilst it is possible to turn off these checks using SELinux booleans, this is not a desirable thing to do for all processes. Another solution would be to ensure that all files were labelled correctly in the chroot and that SELinux domain transitions occurred when needed. However, this is not possible because processes running under mock are told by the dummy libselinux that SELinux is disabled.

SELinux policy module for mock

The following solution to these issues creates an SELinux policy for mock that has two aspects:

1. It runs mock in its own domain (mock_t) that is as unconstrained as any process that needs to run under it needs to be, which so far means that it allows execheap and execmod, just like the mono_t domain does.
2. It labels all files installed under /var/lib/mock (where mock sets up its chroot environments) with the special context type mock_var_lib_t, for which execmod is allowed for processes in the mock_t domain.

This solution relaxes SELinux protection sufficiently for mock to be able to work, without compromising the protection afforded to the normal unconfined domain.

To install the policy module:

• Create a directory to store local policy files, such as /root/selinux.local, and change to that directory
• Download File:PackageMaintainers MockTricks mock.if, File:PackageMaintainers MockTricks mock.fc, and File:PackageMaintainers MockTricks mock.te to this directory
• Build and install the policy module as follows:

# make -f /usr/share/selinux/devel/Makefile
Compiling targeted mock module
/usr/bin/checkmodule:  loading policy configuration from tmp/mock.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 5) to tmp/mock.mod
Creating targeted mock.pp policy package
rm tmp/mock.mod.fc tmp/mock.mod

The selinux-policy-devel and checkpolicy packages are required

If the policy module is loaded before the mock package is installed, /var/lib/mock will be labelled as mock_var_lib_t and /usr/bin/mock will be labelled as mock_exec_t at installation time. Otherwise, it's necessary to use restorecon to fix the file contexts:

# restorecon -R /var/lib/mock /usr/bin/mock

This policy would probably work for mach as well, just by adding these two lines to mock.fc:

/usr/bin/mach       -- gen_context(system_u:object_r:mock_exec_t,s0)
/var/lib/mach(/.*)?    gen_context(system_u:object_r:mock_var_lib_t,s0)

Using mock as a chroot sandbox tool

Mock can be used to create chroots for testing things, not just building packages. Here is a quick howto:

• Create a config file that points to the repo(s) of your choice, where your test packages are

mock -r <config-name> --init
mock -r <config-name> --install <your packages>
mock -r <config-name> --shell

Why use mock to shell, why not chroot directly? Using mock to "shell" will allow mock to create the mountpoints you'll probably need inside the chroot. If you want to manage your mounts manually, do so by all means.

Testing graphical applications inside mock

we need --old-chroot or with --enable-network to "test graphical apps inside mock", https://github.com/rpm-software-management/mock/issues/223

To test graphical applications or tcl/tk bindings inside mock you have to enable other hosts using xhost (or at least the one you need):

xhost +

and then set the DISPLAY environment variable inside the chroot:

export DISPLAY=:0.0

Disable domain name resolution

Sometimes you build you package locally, but it fails as soon as you build it in Koji, since it cannot resolve domain names and connect to remote services. This is because DNS is not configured in mock for Koji. You can achieve the same functionality locally, by adding

config_opts['use_host_resolv'] = False

into your mock config. You can add this option into site-defaults.cfg to enable it by default for every mock configuration.