No edit summary |
|||
| (26 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
'''Note''': This wiki is pretty oudated, refer to https://docs.fedoraproject.org/en-US/flatpak/ for updated documentation | |||
---- | |||
= Basic Idea = | = Basic Idea = | ||
[http://flatpak.org Flatpak] (formerly xdg-app) is a way of distributing and running applications with more isolation from the system than a traditional packaging format such as RPM. Dependencies for a flatpak are either distributed as part of the ''runtime'' or are bundled into the application itself. These - the runtime and the application - are the two types of artifacts we want to build for Fedora. | [http://flatpak.org Flatpak] (formerly xdg-app) is a way of distributing and running applications with more isolation from the system than a traditional packaging format such as RPM. Dependencies for a flatpak are either distributed as part of the ''runtime'' or are bundled into the application itself. These - the runtime and the application - are the two types of artifacts we want to build for Fedora. | ||
| Line 6: | Line 9: | ||
; application: The application contains both the application itself and libraries it depends on that are not part of the runtime. All files in the application are under /app. | ; application: The application contains both the application itself and libraries it depends on that are not part of the runtime. All files in the application are under /app. | ||
Runtimes and applications are typically distributed as ostree repositories - but also can be stored as a single-file bundle. | Runtimes and applications are typically distributed as ostree repositories - but also can be stored as a single-file bundle. A bundle can be an ostree delta, or an OCI Image. | ||
When a flatpak is run, an isolated environment is created with the runtime mounted on /usr and the application mounted on /app. Depending on sandboxing, the environment might have full access to the user's home directory, or might only access to a restricted set of D-Bus ''portals''. | When a flatpak is run, an isolated environment is created with the runtime mounted on /usr and the application mounted on /app. Depending on sandboxing, the environment might have full access to the user's home directory, or might only access to a restricted set of D-Bus ''portals''. | ||
Flatpaks can be built directly from upstream sources using the | Flatpaks can be built directly from upstream sources using the <code>flatpak-builder</code> tool, but in the Fedora context, we want to build runtimes and applications in a way that is integrated with the overall Fedora development process: when we do a security update for a library, we want to know what applications or runtimes are affected and need to be rebuilt. We don't want to create an entirely new way for sources to get into Fedora bypassing the normal Fedora procedures. | ||
There will be a single Fedora runtime maintained on the same schedule as the Platform module for Fedora. It will be be defined as a module that includes libraries that are commonly used for graphical applications. Some of these will inherit from the Platform module. Each application has it's own module in which the relevant RPMs are rebuilt with a special RPM macros (in the | There will be a single Fedora runtime maintained on the same schedule as the Platform module for Fedora. It will be be defined as a module that includes libraries that are commonly used for graphical applications. Some of these will inherit from the Platform module. Each application has it's own module in which the relevant RPMs are rebuilt with a special RPM macros (in the <code>flatpak-rpm-macros</code> package) to relocate the application and bundled libraries into the /app prefix. (This is necessary, because inside an executing Flatpak, the application is mounted at /app, and the runtime at /usr) | ||
The packages are then composed into Flatpaks by the layered image service, sharing as much of the workflow and code as possible with containers. | The packages are then composed into Flatpaks by the layered image service, sharing as much of the workflow and code as possible with containers. We plan to use the OCI image format for Flatpak bundles during the build process, and to distribute them to users via registry.fedoraproject.org. | ||
= Building the Flatpak Runtime = | = Building the Flatpak Runtime = | ||
The 'flatpak-runtime' module ([https://github.com/owtaylor/module-flatpak-runtime prototype]) defines the contents of the runtime. The 'runtime' profile lists exactly what is installed into the flatpak runtime. There is also a 'buildroot' module which lists the | The 'flatpak-runtime' module ([https://github.com/owtaylor/module-flatpak-runtime prototype]) defines the contents of the runtime. The 'runtime' profile lists exactly what is installed into the flatpak runtime. There is also a 'buildroot' module which lists the [https://github.com/owtaylor/rpm-flatpak-rpm-macros flatpak-rpm-macros] package. This special profile is used by the modularity tools to populate the buildroot for building any module that buildrequires flatpak-runtime. The flatpak-rpm-macros package overrides the standard RPM macros so that RPMS locate their content in /app rather than /usr. | ||
The current prototype of flatpak-runtime is based on Fedora 26 and pulls packages from the base-runtime and shared-userspace modules. It is expected that for Fedora 27 it will be switched to depend on the Platform module, and the lifetime will match the lifetime of the Platform module. | The current prototype of flatpak-runtime is based on Fedora 26 and pulls packages from the base-runtime and shared-userspace modules. It is expected that for Fedora 27 it will be switched to depend on the Platform module, and the lifetime will match the lifetime of the Platform module. | ||
The upstream runtimes include both a runtime, and an SDK, where the SDK contains the compiler, developer headers, etc, used to build an applications from source using | The upstream runtimes include both a runtime, and an SDK, where the SDK contains the compiler, developer headers, etc, used to build an applications from source using <code>flatpak-builder</code>. We do not need a SDK for the Fedora runtime, since builds will simply pull in the necessary packages to build against, but we may eventually want to produce one that can be used by third-parties who want a runtime that has the lifetime guarantees and security updates that the Fedora project, but want to build their applications (open-source or closed-source) themselves from source. | ||
=== Work items === | |||
* Rebase flatpak-runtime to F27 | |||
* Extend flatpak-runtime to include additional libraries (possibly matching the org.gnome.Platform module) | |||
= Building Flatpak Modules = | = Building Flatpak Modules = | ||
Each application is then defined as a module that buildrequires flatpak-runtime, and that includes the packages that should be installed into the flatpak application filesystem in its default profile. [https://github.com/owtaylor/module- | Each application is then defined as a module that buildrequires flatpak-runtime, and that includes the packages that should be installed into the flatpak application filesystem in its default profile. [https://github.com/owtaylor/module-eog Prototype eog application module] | ||
Initially, we expect application authors to simply follow the major branch structure of Fedora and maintain the same RPMs for module use and for standard RPM installation, but in the future application authors to be able to follow upstream releases instead, and eventually bundle library versions as needed without reference to the main Fedora schedule - though they will need to keep track of the lifetime of bundled library branches. (See [[Changes/ArbitraryBranching]]) | |||
=== Work Items === | |||
* Make modulemd creation via [https://pagure.io/flatpak-module-tools flatpak-module-tools] more user friendly - reduce the amount of debugging spew, provide better ways to debug failures due to builddep loops. (Or switch to tooling that is more in sync with modularity in general.) | |||
* Create additional test applications | |||
** A largely actively developed application like Inkscape or the GIMP | |||
** An application that is sandboxed and uses the Portal infrastructure | |||
** A simple game using SDL | |||
= Building Flatpak Images = | = Building Flatpak Images = | ||
| Line 35: | Line 51: | ||
* [https://github.com/release-engineering/koji-containerbuild koji-containerbuild]. This is the glue between koji and OSBS and also the command line client. The changes needed here would be pretty minor - it just has a bit of code that parses labels out of the Dockerfile. | * [https://github.com/release-engineering/koji-containerbuild koji-containerbuild]. This is the glue between koji and OSBS and also the command line client. The changes needed here would be pretty minor - it just has a bit of code that parses labels out of the Dockerfile. | ||
* [https://github.com/projectatomic/osbs-client osbs-client]. The headline of this module is that it is is a command-line/python client for OSBS, but it actually has a ton of code that is specific to atomic-reactor and koji and not to OSBS (OSBS in some sense doesn’t exist - image building is just part of OpenShift these days.) It sets up the configuration that gets passed to atomic-reactor. osbs-client would have to detect (or be told) that the build is a flatpak build and set up the atomic-reactor configuration differently. | * [https://github.com/projectatomic/osbs-client osbs-client]. The headline of this module is that it is is a command-line/python client for OSBS, but it actually has a ton of code that is specific to atomic-reactor and koji and not to OSBS (OSBS in some sense doesn’t exist - image building is just part of OpenShift these days.) It sets up the configuration that gets passed to atomic-reactor. osbs-client would have to detect (or be told) that the build is a flatpak build and set up the atomic-reactor configuration differently. | ||
* [https://github.com/projectatomic/atomic-reactor atomic-reactor]. Atomic reactor is where the code that runs inside the OpenShift builder image lives. It has a plugin structure with many more-or-less independent plugins that run before and after the actual build. | * [https://github.com/projectatomic/atomic-reactor atomic-reactor]. Atomic reactor is where the code that runs inside the OpenShift builder image lives. It has a plugin structure with many more-or-less independent plugins that run before and after the actual build. The basic idea is to have a pre-build plugin that creates a Dockerfile which installs the rpms for the flatpak into a chroot in <code>/var/lib/flatpak</code> and a post-build plugin that extracts that part of the filesystem and turns it into a flatpak. | ||
[https://pagure.io/flatpak-module-tools flatpak-module-tools] includes <code>flatpak-module create-flatpak</code> which: | |||
* Downloads modules from Koji | |||
* Installs rpms into a Docker container to build a filesystem image | |||
* Turns that filesystem image and turn it into a flatpak application or runtime. | |||
This serves as an early-prototype of the build step. [[Workstation/FlatpakDevelopmentSetup|Instructions]] | |||
=== Work Items === | |||
* Update koji-cointainerbuild and obs-client to be able to submit a Flatpak build | |||
* Implement building Flatapk OCI Images within atomic-reactor | |||
* Implement some form of OCI image support in atomic-reactor - enough to be able to handle the output from the flatpak build and push it to registry.fedoraproject.org. This will use [https://github.com/projectatomic/skopeo skopeo] | |||
= Distributing Flatpaks = | = Distributing Flatpaks = | ||
Flatpaks will be distributed along side containers on registry.fedoraproject.org, | Flatpaks will be distributed along side containers on registry.fedoraproject.org, and whatever provisions are made for mirroring the Fedora registry will then mirror flatpaks as well. | ||
=== Work Items === | |||
* Allow uploading OCI images to registry.fedoraproject.org [https://github.com/docker/distribution/pull/2076 Upstream pull request] | |||
* Create some way to get summary information for all flatpaks on registry.fedoraproject.org; this might be a patch to the codebase, a look-aside file, or a separate service. | |||
* Modify flatpak and gnome-software so that flatpaks can be installed from registry.fedoraproject.org. (There is already support from installing from a "OCI registry" - but that's just based on alexl's guess about how OCI images would be exported via HTTP - nobody seems to be implementing such a registry.) | |||
= Packager Workflow = | = Packager Workflow = | ||
In order to make a Flatpak of an application, the packager needs to create two extra pieces: | In order to make a Flatpak of an application, the packager needs to create two extra pieces: | ||
* A modulemd (module metadata) file (called <modulename>.yaml) | |||
* A JSON file describing extra-metadata for the application, such as the security permissions that it requires. (Called flatpak.json - the idea is to be similar to Dockerfile where this is a fixed name.) | |||
With current plans, this requires three separate git repositories ( | With current plans, this requires three separate git repositories (<code>rpms/eog</code>, <code>modules/eog</code>, <code>containers/eog</code>), and separate releng steps to create each. We should consider whether it makes sense to allow, for the case where there is a clean rpm <=> module <=> container/flatpak mapping, to allow a packager to put the eog.yaml and flatpak.json files directly in rpms/ and then build a module and flatpak against that. If not, we'll need some tools and expedited process to make this convenient. | ||
Given the name of an existing application-package in Fedora, it | Given the name of an existing application-package in Fedora, it ispossible to write tools to generate a first version of the modulemd and JSON files ([https://pagure.io/flatpak-module-tools flatpak-module-tools] has <code>flatpak-module create-modulemd</code>), though the JSON file will need some editing. | ||
Initially, the packager would need to manually trigger module and flatpak builds, but we'll work together with general module and container efforts to improve automation. Useful automation might include, for example: | Initially, the packager would need to manually trigger module and flatpak builds, but we'll work together with general module and container efforts to improve automation. Useful automation might include, for example: | ||
* Building the flatpak automatically each time the module build completes | |||
* Starting a new module build automatically when an update to a package contained within the package is filed in bodhi. (Or when changes are pushed to dist-git?) | |||
= Handling Updates = | = Handling Updates = | ||
It's not clear whether we want updates for applications to be ''separately'' handled in Bodhi, or whether when package updates are made, applications are automatically rebuilt. | It's not clear whether we want updates for applications to be ''separately'' handled in Bodhi, or whether when package updates are made, applications are automatically rebuilt. Work is underway in Bodhi allow filing separate updates for anything that has a representation in Koji, include containers, and that work should carry through to Flatpaks with few changes, but the necessary work for an integrated package/application workflow still has to be figured out. | ||
= Appendix: Previous Work on Building Flatpaks out of RPMS = | = Appendix: Previous Work on Building Flatpaks out of RPMS = | ||
Latest revision as of 09:19, 6 May 2019
Note: This wiki is pretty oudated, refer to https://docs.fedoraproject.org/en-US/flatpak/ for updated documentation
Basic Idea
Flatpak (formerly xdg-app) is a way of distributing and running applications with more isolation from the system than a traditional packaging format such as RPM. Dependencies for a flatpak are either distributed as part of the runtime or are bundled into the application itself. These - the runtime and the application - are the two types of artifacts we want to build for Fedora.
- runtime
- A runtime contains files that multiple applications depend on. There can be any number of runtimes installed on a system: for example, there might be both a Fedora 23 and a Fedora 24 runtime installed. A runtime looks like a mini-system image with libraries in /usr/lib, data files in /usr/share, binaries in /usr/bin and so forth.
- application
- The application contains both the application itself and libraries it depends on that are not part of the runtime. All files in the application are under /app.
Runtimes and applications are typically distributed as ostree repositories - but also can be stored as a single-file bundle. A bundle can be an ostree delta, or an OCI Image.
When a flatpak is run, an isolated environment is created with the runtime mounted on /usr and the application mounted on /app. Depending on sandboxing, the environment might have full access to the user's home directory, or might only access to a restricted set of D-Bus portals.
Flatpaks can be built directly from upstream sources using the flatpak-builder tool, but in the Fedora context, we want to build runtimes and applications in a way that is integrated with the overall Fedora development process: when we do a security update for a library, we want to know what applications or runtimes are affected and need to be rebuilt. We don't want to create an entirely new way for sources to get into Fedora bypassing the normal Fedora procedures.
There will be a single Fedora runtime maintained on the same schedule as the Platform module for Fedora. It will be be defined as a module that includes libraries that are commonly used for graphical applications. Some of these will inherit from the Platform module. Each application has it's own module in which the relevant RPMs are rebuilt with a special RPM macros (in the flatpak-rpm-macros package) to relocate the application and bundled libraries into the /app prefix. (This is necessary, because inside an executing Flatpak, the application is mounted at /app, and the runtime at /usr)
The packages are then composed into Flatpaks by the layered image service, sharing as much of the workflow and code as possible with containers. We plan to use the OCI image format for Flatpak bundles during the build process, and to distribute them to users via registry.fedoraproject.org.
Building the Flatpak Runtime
The 'flatpak-runtime' module (prototype) defines the contents of the runtime. The 'runtime' profile lists exactly what is installed into the flatpak runtime. There is also a 'buildroot' module which lists the flatpak-rpm-macros package. This special profile is used by the modularity tools to populate the buildroot for building any module that buildrequires flatpak-runtime. The flatpak-rpm-macros package overrides the standard RPM macros so that RPMS locate their content in /app rather than /usr.
The current prototype of flatpak-runtime is based on Fedora 26 and pulls packages from the base-runtime and shared-userspace modules. It is expected that for Fedora 27 it will be switched to depend on the Platform module, and the lifetime will match the lifetime of the Platform module.
The upstream runtimes include both a runtime, and an SDK, where the SDK contains the compiler, developer headers, etc, used to build an applications from source using flatpak-builder. We do not need a SDK for the Fedora runtime, since builds will simply pull in the necessary packages to build against, but we may eventually want to produce one that can be used by third-parties who want a runtime that has the lifetime guarantees and security updates that the Fedora project, but want to build their applications (open-source or closed-source) themselves from source.
Work items
- Rebase flatpak-runtime to F27
- Extend flatpak-runtime to include additional libraries (possibly matching the org.gnome.Platform module)
Building Flatpak Modules
Each application is then defined as a module that buildrequires flatpak-runtime, and that includes the packages that should be installed into the flatpak application filesystem in its default profile. Prototype eog application module
Initially, we expect application authors to simply follow the major branch structure of Fedora and maintain the same RPMs for module use and for standard RPM installation, but in the future application authors to be able to follow upstream releases instead, and eventually bundle library versions as needed without reference to the main Fedora schedule - though they will need to keep track of the lifetime of bundled library branches. (See Changes/ArbitraryBranching)
Work Items
- Make modulemd creation via flatpak-module-tools more user friendly - reduce the amount of debugging spew, provide better ways to debug failures due to builddep loops. (Or switch to tooling that is more in sync with modularity in general.)
- Create additional test applications
- A largely actively developed application like Inkscape or the GIMP
- An application that is sandboxed and uses the Portal infrastructure
- A simple game using SDL
Building Flatpak Images
Flatpak applications and runtimes will be built as an extension of the Layered Build Service. By building the Flatpaks as OCI images, it will be possible to have a very similar pipeline for server-side containers and for flatpaks. Unfortunately, although having OCI image support is a desired future feature for atomic-reactor and other layered build service components, it does not yet exist, and will need to be implemented as part of the Flatpak process.
The modules that would need modification are:
- koji-containerbuild. This is the glue between koji and OSBS and also the command line client. The changes needed here would be pretty minor - it just has a bit of code that parses labels out of the Dockerfile.
- osbs-client. The headline of this module is that it is is a command-line/python client for OSBS, but it actually has a ton of code that is specific to atomic-reactor and koji and not to OSBS (OSBS in some sense doesn’t exist - image building is just part of OpenShift these days.) It sets up the configuration that gets passed to atomic-reactor. osbs-client would have to detect (or be told) that the build is a flatpak build and set up the atomic-reactor configuration differently.
- atomic-reactor. Atomic reactor is where the code that runs inside the OpenShift builder image lives. It has a plugin structure with many more-or-less independent plugins that run before and after the actual build. The basic idea is to have a pre-build plugin that creates a Dockerfile which installs the rpms for the flatpak into a chroot in
/var/lib/flatpakand a post-build plugin that extracts that part of the filesystem and turns it into a flatpak.
flatpak-module-tools includes flatpak-module create-flatpak which:
- Downloads modules from Koji
- Installs rpms into a Docker container to build a filesystem image
- Turns that filesystem image and turn it into a flatpak application or runtime.
This serves as an early-prototype of the build step. Instructions
Work Items
- Update koji-cointainerbuild and obs-client to be able to submit a Flatpak build
- Implement building Flatapk OCI Images within atomic-reactor
- Implement some form of OCI image support in atomic-reactor - enough to be able to handle the output from the flatpak build and push it to registry.fedoraproject.org. This will use skopeo
Distributing Flatpaks
Flatpaks will be distributed along side containers on registry.fedoraproject.org, and whatever provisions are made for mirroring the Fedora registry will then mirror flatpaks as well.
Work Items
- Allow uploading OCI images to registry.fedoraproject.org Upstream pull request
- Create some way to get summary information for all flatpaks on registry.fedoraproject.org; this might be a patch to the codebase, a look-aside file, or a separate service.
- Modify flatpak and gnome-software so that flatpaks can be installed from registry.fedoraproject.org. (There is already support from installing from a "OCI registry" - but that's just based on alexl's guess about how OCI images would be exported via HTTP - nobody seems to be implementing such a registry.)
Packager Workflow
In order to make a Flatpak of an application, the packager needs to create two extra pieces:
- A modulemd (module metadata) file (called <modulename>.yaml)
- A JSON file describing extra-metadata for the application, such as the security permissions that it requires. (Called flatpak.json - the idea is to be similar to Dockerfile where this is a fixed name.)
With current plans, this requires three separate git repositories (rpms/eog, modules/eog, containers/eog), and separate releng steps to create each. We should consider whether it makes sense to allow, for the case where there is a clean rpm <=> module <=> container/flatpak mapping, to allow a packager to put the eog.yaml and flatpak.json files directly in rpms/ and then build a module and flatpak against that. If not, we'll need some tools and expedited process to make this convenient.
Given the name of an existing application-package in Fedora, it ispossible to write tools to generate a first version of the modulemd and JSON files (flatpak-module-tools has flatpak-module create-modulemd), though the JSON file will need some editing.
Initially, the packager would need to manually trigger module and flatpak builds, but we'll work together with general module and container efforts to improve automation. Useful automation might include, for example:
- Building the flatpak automatically each time the module build completes
- Starting a new module build automatically when an update to a package contained within the package is filed in bodhi. (Or when changes are pushed to dist-git?)
Handling Updates
It's not clear whether we want updates for applications to be separately handled in Bodhi, or whether when package updates are made, applications are automatically rebuilt. Work is underway in Bodhi allow filing separate updates for anything that has a representation in Koji, include containers, and that work should carry through to Flatpaks with few changes, but the necessary work for an integrated package/application workflow still has to be figured out.
Appendix: Previous Work on Building Flatpaks out of RPMS
Alex Larsson's first effort using a copr can be found at:
- https://lists.fedoraproject.org/pipermail/desktop/2015-May/012362.html
- https://github.com/alexlarsson/fedora-runtime
Owen Taylor then picked up and did some work on making this work within Koji