Modularity

Summary

Modularization is an an ongoing initiative in Fedora to resolve the issue of divergent, occasionally conflicting lifecycles of different components. A module provides functionality (for instance a web server) and includes well-integrated and -tested components (for instance Apache httpd and the libraries on which it depends). It can be deployed into production in various ways, for instance as "classic" RPM packages or a container image, and is updated as a whole. Different modules can emphasize new features, stability, security, etc. differently.

Background

The modularization objective was introduced to Fedora as a next step to the Fedora Rings proposal. We have been through many discussions on this subject and false starts on development work. We are trying to capture what the project is and will be in this wiki page. By way of introduction it is probably useful to try to define what we mean by "modularization."

First of all, why do we call it "modularization?" Well, we are trying to use an agnostic term to not indicate the specific nature of a module or a component. The modules could be big or small, low or high-level, brand new or very mature. As a result, please try not to ascribe too much meaning to the word module or component aside from being a slightly prettier and shorter version of "chunk of some stuff".

Next, why are we pursuing this goal? Well, there a a lot of reasons but, I think, the simplest is to try to disconnect the lifecycle of major components from each other so that they can grow and change at the speed that is appropriate to the component. Why does that matter? Well, that is a significantly more complex conversation and somewhat beyond the scope of this document.

The first step in this project was the Fedora Rings proposal. The original proposal being adopted has resulted in a number of tangible changes with varying levels of success. The split in to the three editions, the creation of several Working Groups to further some of these goals, namely Envs & Stacks and Base, and the creation and implementation of COPR.

However, looking at the original Fedora Rings proposal, it seems that the simple metaphor of concentric rings doesn't actually work very well for our increasingly messy open source software world. For example, there ends up being a huge number of rings to represent the whole spectrum of "quality" for modules. Also, the rings have problems representing orthogonal concerns like build dependencies, not to mention docs and tests. As a result, much of this discussion has stagnated trying to force fit the solution to the problem.

Motivations

• Fedora Rings talk: for slides or Video
• 2014 Fedora Rings update: slides
• A recent talk at FOSDEM 2016 and reprised at devconf.cz 2016 and slides
• Concrete example: can we update docker at different speeds in, say, Fedora Atomic vs. Fedora Server? Mailing list thread at projectatomic.io
• A talk by Colin Charles at FOSDEM 2016: "Distributions from the view of a package"; slides, unfortunately, no video.

Context

The Modularity topic has a number of perspectives and goals, as a result, we will be posting a series of articles to the community blog outlining the problems and benefits. Hopefully, separating the perspectives in to dedicated posts will make the overall story clearer. As the articles are published, we will provide links to them below.

Conversations

• Mo Duffy wrote up a recap of a server-wg meeting
• Matthew Miller posted to devel asking for a minimum viable demo of security scanning for a container-based module

Blog Posts

• Modularity Use Case: Application Independence
• Why modularization matters to Sys Admins

LWN Article

• Modularizing Fedora (March 16, 2016) — Good overview from an outside-the-project perspective. (n.b. paywalled until March 31st)

Summary of Work

• The Aleph proposal made in the E&S Working Group allows for different levels of package quality.
• A playground proposal which would combine certain COPR repositories in to one unified repository for packages that are considered more production quality than COPR but not strong enough for inclusion in the main repos. A Change was also proposed.
• See Modularity_Working_Group#Tooling Prototypes for some existing tooling prototypes.

Fedora Objectives

Fedora has focused efforts related to the Modularization project through a number of Objectives.

• Objectives/Fedora Editions, Phase 2
  • Summary: Take the initial Server/Workstation/Cloud split from Fedora 21 from an experiment into solid production. Increase autonomy from FESCo and improve targeted outreach.
  • Objective Lead: Stephen Gallagher
  • Timeframe: F22 and F23 releases, ending shortly after Flock 2015.

• Objectives/Fedora Modularization, Requirements Phase
  • Summary: Requirements-gathering from stakeholders for "Rings" aspect of Fedora.next
  • Objective Lead: Langdon White
  • Timeframe: F23 release - Flock; will be updated with subsequent phase

• Objectives/Fedora Modularization, Prototype Phase
  • Summary: The goal of this phase is to deliver a functional implementation of modular Fedora.
  • Objective Lead: Langdon White
  • Timeframe: F25 release, with demo at conferences in early 2017.

What is a module (esoteric)?

Now, regarding the "what," we think we have some proposed characteristics of a module but we really need feedback to call this "true":

• A dotted line we draw around a set of components that we declare a "thing." I hesitate to give examples of a "thing" because we aren't sure yet if "apache-httpd" is a module, "webserver" is a module, or both are. However, I think it makes sense as "a larger unit of measure than a traditional RPM".
• A module is a unit of delivery, that is always tested together and released together. That said, we would like to provide a short-circuiting mechanism for a sys-admin/ops person whereby they can knowingly "break" a module by applying a library update because the module is not being released in a timely manner.
• A module as a whole has its own lifecycle that is independent of any other module. Maintainers may decide to release multiple modules together on a common release schedule, but it is always possible to release modules independently when desired.
• A module may include different versions of components than other modules.
• A module comes with associated metadata: this may include such things as lifecycle information (when does the module go end of life), who maintains it and to what support level, etc.

What is a module (tactical)?

OK, so that was the "esoteric what." How about an "implementation what"? We want to use this definition for the sake of prototyping and early implementation. We may find this changes based on the prototypes. For now, a module is:

• A repository. Yeah, just a plain old RPM repository (for now). A module definition declares what RPMs it includes (both hard requires and optionally requires). All of these RPMs are included in the repo that “is” the module. A module may also specify in its definition that it depends on one or more other modules, but it may not specify any of the RPMs in that "remote" module.
• A module has a unique name
• A module can have multiple distinct versions, likely corresponding to distinct functionality or ABI versions; and multiple versions may be available at the same time.
• Each version of a module has its own independent update stream associated with it. We avoid changing ABI or intentionally breaking forwards compatibility in any way within the update stream of a single version.
• Has a well known set of non-runtime dependencies which are not available in the same "repo" as the module itself. While it seems like this could be easily supported in the "for now" case, having this requirement makes sure we don't paint ourselves into a corner.
• A module has an API. In essence, the API is what "makes" the module. For example, if we had a "Web server" module, its "api" might be HTTP/2, we could provide that using httpd or nginx, and, next week, swap it, because the api is king, not the binaries inside. However, while we need to consider the API model, full support of this may not be necessary for the MVP.
• Alluded to earlier, but, a module is *not* self-hosted. That doesn't mean Fedora doesn't know how to build it, or that the information and steps to build it aren't available, just that the consumer has to take some extra steps to find this information. We don't, necessarily, want to consider the build deps the same level of quality as the module itself. In the future, an “edition” or a “spin” would be composed of a set of modules (vs a set of RPMs).
• Not directly installable and/or may be installers themselves. In other words, a module does not have to carry tooling to get itself to the end user.

How to Get Involved

See Modularity WG and Getting involved.

FAQ

If you identify anything that you would like the answer to, but aren’t sure actually belongs here, please use the FAQ.