Workstation/BuildingFlatpaksOldPlan

Basic Idea

Flatpak (formerly xdg-app) is a way of distributing and running applications with more isolation from the system than a traditional packaging format such as RPM. Dependencies for a flatpak are either distributed as part of the runtime or are bundled into the application itself. These - the runtime and the application - are the two types of artifacts we want to build for Fedora.

runtime

A runtime contains files that multiple applications depend on. There can be any number of runtimes installed on a system: for example, there might be both a Fedora 23 and a Fedora 24 runtime installed. A runtime looks like a mini-system image with libraries in /usr/lib, data files in /usr/share, binaries in /usr/bin and so forth.

application

The application contains both the application itself and libraries it depends on that are not part of the runtime. All files in the application are under /app.

Runtimes and applications are typically distributed as ostree repositories - but also can be stored as a single-file bundle. Currently a bundle is a ostree delta, but using the OCI container format instead is being investigated.

When a flatpak is run, an isolated environment is created with the runtime mounted on /usr and the application mounted on /app. Depending on sandboxing, the environment might have full access to the user's home directory, or might only access to a restricted set of D-Bus portals.

We generally want the runtimes and applications to be built out of the same set of sources as the rest of Fedora: when we do a security update for a library, we want to know what applications or runtimes are affected and need to be rebuilt. We don't want to create an entirely new way for sources to get into Fedora bypassing the normal Fedora procedures.

Runtimes can simply be built out of normal Fedora RPMS in the same way as we might compose an ISO or a container image, but to make an application out of RPMs requires rebuilding them with the /app prefix. The most obvious way of doing this would be a separate build target, but the downside of this is that we've now forked the build process. Say we build the 'eog' image viewer as a flatpak - so we rebuild it in the special build target. It requires the 'exempi' library for reading image metadata, so we rebuild that as well. Later - the exempi maintainer updates exempi - somehow the build in the other build target has to be rebuilt in sync, or the eog bundle will contain a stale version of exempi.

What if we could consider "x86_64 but rebuilt with a /app prefix" as a separate architecture, so that whenever exempi is rebuilt, koji rebuilds that version as well? This led to the idea of "architecture variants", where a package can be configured to build not just x86_64, but also x86_64~flatpak. See https://pagure.io/fork/otaylor/koji/blob/xdg-app/f/docs/arch_variants.rst for more details.

Alternate Approaches

• Instead of encoding the "flatpak" nature of the RPM in the dist tag (exempi-2.2.1.fc24~flatpak.x86_64), put it into the name of the RPM instead: 'flatpak-exempi-2.2.1.fc24.x86_64'. This is the approach that SCL's take. The 'eog' package would have Name: %{_flatpak_prefix}eog and BuildRequire: %{_flatpak_prefix}exempi-devel. This would require explicit conversion of all specfiles to support Flatpak, would hard-code what dependencies are part of the Flatpak runtime and which are bundled into the specfile (You would still BuildRequire: _libstdc++-devel, etc), and would make a lot of Fedora specfiles uglier. It would, however, have the advantage of making a build where a build-requires was not rebuilt for flatpak fail immediately in a clean fashion.
• Instead of of encoding the "flatpak" nature of the RPM in the dist tag, use the currently-being-developed koji namespace support to rebuild them, so the same NVR exempi-2.2.1.fc24.x86_64 could be built either for /usr or /app. This would be highly confusing.
• Instead of adding architecture-variant support to Koji, add a separate build target for flatpak that shadows the main build target (f24-flatpak-build) and use an external trigger to shadow build - so each successful build in f24-build would be rebuilt into f24-flatpak-build which would have a different build group which contained a different RPM configuration. If this failed, the packager would be asynchronously notified, and would have to bump-and-rebuild, since the main build would already be imported and tagged into f24-build. There would be problems with keeping the shadow tag exactly in sync with the main tag because packagers might not always fix and rebuild, because of secondary build tags like f24-gnome, and because of packages being manually tagged and untagged.

Packager Workflow

1. A packager identifies a package as a candidate for creating a flatpak for.
2. They mark the application to be built as a flatpak (koji set-pkg-variants 'MAIN flatpak' f25 eog.)
3. They bump and rebuild the package
4. They try and build the flatpak (koji flatpak-build f25 eog eog.json. Extra information about the app - like what resources it wants access to - is included in a JSON file that could also be checked into source control.
5. If that fails because of missing library dependencies, they add the flatpak variant to these dependencies and rebuild them, and repeat the flatpak build.

Phase 1 - manual enablement

• Support in koji for architecture variants
  • Add 'arch_variants' to the tag_packages schema.
  • Add support in the hub to understand variants at the necessary places
  • Add support in the builder to fire off builds for configured architecture variants, and when building for a architecture variant, install '<variant>-build' rather than 'build'
  • A way to set the variants for a packages - perhaps 'koji set-pkg-variants' similar to 'koji set-pkg-arches'.
• A koji plugin to add ‘koji flatpak-build [--runtime] NVR METAINFO.JSON’ which takes an existing package that has been built for the ~flatpak variants, and converts the RPM and its dependencies into a flatpak bundle (either existing ostree delta or future OCI image).

A branch of Koji with prototypes of these changes can be found at: https://pagure.io/fork/otaylor/koji/branch/xdg-app. flatpak-build is done as a built-in rather than a plugin as a first pass for development convenience.

Enhancement: autoimport into a ostree repository

• When new app/runtime bundles are built, automatically import them into a ostree repository. (Or testing/release repositories based upon how the source rpms are tagged) Conceivably this could be handled by a new separate service that listens on the message bus, rather than trying to extend koji.

Enhancement: autorebuild of flatpaks

• When packages are rebuilt that are part of flatpak applications and runtimes, automatically trigger rebuilds of the applications/runtimes. This would require tracking the packages in each application or runtime. This might be handled by the same daemon as above.

N.B. This is already work in progress for OCI/Docker containers and partially completed with Factory 2.0 and OSBS. -- mattdm

Enhancement: generalize this to server containers

• The basic idea of taking RPMs and converting the rpm and dependencies into a container via rpm-ostree is applicable to containers as well. The autorebuild and autoimport into ostree steps would be useful for this as well (but also probably auto-push to a docker registry.) One challenge here is that while an RPM of a desktop application is “self-describing” - it contains the .desktop and appstream files that specify what the application is and how to run it, turning an RPM of, say, etcd into a useful container probably requires extra metadata and configuration files.

Enhancement: merge in ~flatpak architecture variants to an existing build

• It might be nice to avoid the bump+rebuild, as envisioned by the mergeScratch RPC call that was added https://lists.fedoraproject.org/pipermail/buildsys/2015-April/004628.html

Enhancement: auto-rebuilding dependencies

• Tooling so that when someone wants to build an application that has not previously been built for ~flatpak, all the build dependencies are queued for adding to the ~flatpak architecture variants and rebuilt in the right order.