Flatpaks without Modules
Summary
Change how we build Flatpaks in Fedora to remove the dependency on modularity. Instead of using modules to rebuild Fedora packages with prefix=/app, there will be a separate build target that is used for that.
Owner
- Name: Owen Taylor
- Email: otaylor@redhat.com
Current status
- Targeted release: Fedora Linux 39
- Last updated: 2023-06-22
Detailed Description
There are two types of Flatpak containers - runtimes - which contain unmodified Fedora packages, and applications - which contain Fedora packages rebuilt to relocate them from /usr to /app. Currently, the rebuild process is done using modules - packages are either rebuilt in a shared flatpak-common module, or in a module specific to the application. This change replaces this with a separate build target that all rebuilds are done in.
Rebuilding RPMs
For each Fedora release, we'll have two additional targets
f39-flatpak-runtime (inherits f39), tags: f39-flatpak-runtime-build, f39-flatpak-runtime f39-app (inherits f39-flatpak-runtime), tags: f39-app-build, f39-app
The f39-flatpak-runtime target has a limited use - it's used to build packages that go into the Flatpak runtimes (standard and KDE). This would include flatpak-runtime-config, flatpak-rpm-macros, but also rebuilds of standard packages if needed to prune dependencies: currently this is done for gstreamer-plugins-good.
The f39-app target is the main target that is used for prefix=/app rebuilds. flatpak-rpm-macros is part of the build group for this target, resulting in builds that target /app. As currently, spec files can be conditionalized with '%if 0%{?flatpak}'
Builds in f39-flatpak-runtime and f39-app have the dist tags .f39runtime and .f39app respectively. This is necessary so that we can automatically rebuild a package without having to bump the release in the specfile. (Koji requires everything to have a unique NVR.) It also allows humans and tools to easily distinguish these specialized builds.
Any Fedora packager can rebuild a package into f39-app using 'fepdkg build --target=f39-app'. There will also be a new convenience subcommands of fedpkg 'fedpkg build-flatpak-rpms', 'fedpkg build-flatpak-rpms-local' that can be used to rebuild all missing bundled dependencies of a Flatpak, in Koji or locally.
Once a package exists in f39-app or f38-app, then [distrobuildsync] will be used to do a build into f39-app each time a standard build completes.
Container Versioning
Currently, the version is based on the module version:
firefox:stable:3820230427145713:b1edb643 => firefox-stable-3820230427145713.b1edb643
Where the module version is created from platform versions and git commit timestamps for the module repository. We could reproduce something very much like this based on the runtime branch and git commit timestamp, but since application Flatpaks have an identifiable "Main RPM" (unlike the general case of modules and containers), a better scheme is:
firefox-112.0.2-1.fc38 => firefox-flatpak-112.0.2-1 name: main RPM name + -flatpak version: main RPM version release: automatically incremented based on existing Koji builds
Because upgrades are done on Flatpak ID (org.mozilla.Firefox), there is no compatibility impact of switching Flatpaks from 'firefox' to 'firefox-flatpak'.
There is considerable implementation complexity within OSBS to implement this,
because the N/V/R need to be written into generated Dockerfile as labels before building it,
but it should be manageable. If not, we'll need to switch to a different scheme.
(Running dnf with --best when building the container will help make this reliable
and would be a general improvement.)
Updates
The current plan is that builds of RPMs will be tagged directly into f39-app and containers will be composed from the contents of f39-app with no involvement of Bodhi. Bodhi is only involved in releasing the container.
We can add CI checks on container updates to compare the rebuilt packages in the container to the latest released versions for Fedora, though it's not always easy to distinguish:
- Newer because of a RPM change that hasn't passed testing yet
- vs. Newer because a spec file was needed to get the package to rebuild with prefix=/app
Since Fedora does not embargo builds, there's no potential information leak if a change to dist-git is released via container before it is released by RPM.
Disadvantages and issues
In moving to this system, we lose the ability to take advantage of the capabilities of modularity - to use a different stream branch of a library for an application than the one shipped in F39, say. There are essentially no examples of this being used in all the Flatpaks created for Fedora over the last 5 years, probably because modularity has gained little traction within Fedora more broadly.
Some Flatpak modules strip down bundled packages by setting macros in the buildopts/rpms/macros section of the modulemd file. For example, flatpak-common has:
buildopts:
rpms:
macros: |
%_without_mingw 1
%_without_tkinter 1
This is no longer possible, so these changes need to be fed back into spec files with '%if 0%{?flatpak}' style conditionalization.
Additionally a couple of packages (evolution-data-services and tracker-miners) are set up so they can be built with an application-specific D-Bus prefix. Evolution has:
buildopts:
rpms:
macros: |
%_eds_dbus_services_prefix org.gnome.Evolution
This will need to be redone so that evolution-data-services doesn't need recompilation and the prefixing can be done by changing configuration files at container build time.
Feedback
Benefit to Fedora
The experience for Fedora developers who want to create Flatpaks of their applications will be improved:
- They no longer need to understand modularity and modulemd files - instead of a modulemd file *and* a container.yaml, they simply create a container.yaml.
- Rebuilds are automatically shared between Flatpaks, without having to get them added to the flatpak-common module.
- Because the local build experience is no longer using the poorly maintained local-build mode of MBS, it can be more efficient and easier to understand.
This should result in more Fedora Flatpaks and more up-to-date Fedora Flatpaks.
We also greatly reduce the usage of Module Build Service within Fedora. This is important because upstream maintenance is currently reduced, and it's not clear that we'll be able to sustain a deployment indefinitely.
Scope
- Proposal owners:
- Update flatpak-module-tools to be able to work without modularity
- Create changes to fedpkg to add 'rebuild-flatpak-packages' (implementation likely within flatpak-module-tools)
- Update atomic-reactor to be able to build Flatpaks without modularity
- Other developers:
- Flatpaks will need to be migrated to the new system by Flatpak packagers. While we could make the new system opt-in for F39, it's probably easiest to just require all Flatpaks to be moved over. In many cases, this should consist of just a few minor changes to container.yaml.
- Changes to fedpkg and atomic-reactor need to be reviewed, and for atomic-reactor deployed. Note Fedora is on OSBS 1, which is not a current release. Backporting changes to OSBS 1 seems more realistic than upgrading Fedora to OSBS 2 at this point.
- Release engineering: #Releng issue number
- Release engineering will need to manually create the tags/targets for F39 and update the tag maintenance scripts to do this for future releases.
- Policies and guidelines: https://docs.fedoraproject.org/en-US/flatpak/ will need to be updated once we have an initial implementation. No impact to packaging guidelines is expected.
- Trademark approval: N/A (not needed for this Change)
- Alignment with Community Initiatives: No
Upgrade/compatibility impact
The Flatpaks built will look exactly the same as before from a user perspective.
How To Test
Testing here will be done by selecting a few Flatpaks and trying to rebuild them using the new system, first locally and then on infrastructure.
The final validation will be when we mass rebuild Flatpaks for F39.
User Experience
This should be largely invisible to users. Hopefully, by simplifying the developer experience around creating Flatpaks, more Fedora Flatpaks will be available and they will be kept more up-to-date.
Dependencies
This is unrelated to other Fedora changes or to changes within packages.
Contingency Plan
- Contingency mechanism: The current infrastructure for module-based builds will be maintained, so that if we can't get this done for F39, we'll build Flatpaks as modules for F39.
- Contingency deadline: By 2023-08-01 we should have runtimes built and a number of applications or we'll plan on using modules for F39 Flatpaks.
- Blocks release? No
Documentation
At the moment, this change proposal is standalone documentation of the change. As noted above, https://docs.fedoraproject.org/en-US/flatpak/ will need to be updated once we have an initial implementation. If you are interested in getting involved in planing and implementing this change, please contact the Flatpak SIG matrix room.