It is still under construction and content may change. Do not rely on the information on this page.
Proposal Overview
Proposal author: Ncoghlan (talk) 21:49, 21 August 2015 (UTC)
The overarching concept behind this proposal is as follows: To ensure Fedora (and its derivatives) provide a superb platform for constructing a software application from open source components, regardless of the open source programming language used to develop those components, or the open source packaging system used by the developers to publish them
To achieve this, it is proposed to explicitly split the software assembly process into the following stages:
- Upstream development & publication
- Upstream release tracking
- Downstream filtering
- Downstream integration
The first three stages are new and collectively referred to as the "Software Component Pipeline". The final stage refers to what Fedora (and other Linux distributions) have long handled on behalf of their users (with commercial Linux distributions being able to sustainably fund more rigorous review and QA processes).
For each of these new stages, we aim to identify clear responsibilities and expectations for the Environments & Stacks working group and individual language SIGs in the provision of relevant software components to Fedora users. We also identify any online services specifically relevant to that stage.
This page also aims to provide the broader context for some of the ideas discussed in:
- Env_and_Stacks/Projects/DeveloperPortal
- Env_and_Stacks/Projects/PackageReviewProcessRedesign
- Env_and_Stacks/Projects/UserLevelPackageManagement
- Env_and_Stacks/Projects/LanguageSpecificRepositories
Initial software component pipeline
The initial iteration of the software component pipeline would be focused on the existing "upstream source format -> SRPM -> RPM" redistribution pipeline. Any components consumed using language specific tools or from cross platform ecosystems like Nix or Conda would still be consumed directly from upstream repositories, even if the package management tools themselves are provided as Fedora packages.
A possible sketch of that version of the pipeline is:
See https://github.com/ncoghlan/repofunnel/blob/master/docs/InitialFedoraPipeline.svg for the latest version of that sketch.
Projects to be added to the diagram:
- Rebase Helper (likely between Anitya and COPR)
Future software component pipeline
Some possible future enhancements to the software component pipeline include:
- support automatic conversion of upstream package formats to SRPM in COPR
- support building of upstream binary formats in COPR
- supporting direct filtering of upstream package repositories in RepoFunnel
A possible sketch of that version of the pipeline is:
See https://github.com/ncoghlan/repofunnel/blob/master/docs/FedoraPipeline.svg for the latest version of that sketch.
Upstream development & publication
Environments & Stacks responsibilities
For this stage, the primary responsibility of the Environments & Stacks working group would be to set appropriate expectations for the individual language SIGs.
Environments & Stacks would also be directly responsible for any language agnostic tools which Fedora may choose to provide to developers for use in this stage of development (e.g. Nix, Conda, Conary)
Individual language SIG responsibilities
For upstream development and publication, it would be the responsibility of individual language SIGs to provide the following for at least Fedora, and potentially for EPEL (if SIG members are interested in that):
- regularly updated language runtimes
- popular development frameworks for their ecosystem (e.g. pygtk, pyqt, Django, Flask, Pyramid, Jupyter for Python)
- ecosystem specific tools for software testing (e.g. tox, nose, pytest, coverage.py for Python)
- ecosystem specific tools for static analysis (e.g. flake8, pylint, mypy for Python)
- ecosystem specific tools for local development environment management (e.g. virtualenv, vex, pyvenv for Python)
- ecosystem specific tools for application debugging and profiling (e.g. pdb++, Meliaie, RunSnakeRun, SnakeViz for Python)
- tools for consuming software directly from upstream software repositories (e.g. Maven Central, pypi.python.org, rubygems.org, npm.org)
- tools for publishing software to upstream software repositories
- if available, tools for maintaining local repositories of software in upstream formats (e.g. Nexus for Java, devpi for Python)
- if available, recommending a specific IDE for new developers that prefer not to write in a plain text editor, and aren't interested in building their own custom IDE by installing and configuring a range of editor plugins
In addition to providing and maintaining these kinds of software components, individual language SIGs would be expected to oversee the following activities:
- publication and maintenance of DevAssistant assistants for creation and modification of projects using the language on dapi.devassistant.org
- collaborating with softwarecollections.org and the CentOS SCL SIG on providing regularly updated language runtimes in that format
- collaborating with the Project Atomic community on Source-to-image builders to create container applications using the relevant language runtimes
- providing content for the Developer Portal ensuring that developers using their language on Fedora are readily able to:
- publish software to upstream software repositories
- publish software using COPR
- publish Docker container images to the upstream DockerHub
- publish Vagrant machine images to the upstream Atlas service
- providing content for the Fedora Release Notes that is tailored to the interests of their language community
Optionally, individual language SIGs may also choose to provide alternate runtimes that seem particularly interesting to their communities (such as the PyPy interpreter for Python), and tools for choosing a "default" runtime on a per-user basis (without affecting operating system components written in that language)
NOTE: JavaScript/HTML/CSS (including Node.js) don't have their own language SIGs, but rather are bundled together into the web-devel sig: https://admin.fedoraproject.org/mailman/listinfo/web-devel
Relevant online services
For this stage of development, the only relevant Fedora specific online service is the Developer Portal. Other Fedora services would be involved only as part of the package maintenance process.
Upstream release tracking
The key goal of this step would be to get upstream software components readily available through COPR in a way which requires minimal ongoing effort on the part of package maintainers.
Environments & Stacks responsibilities
For this stage, the primary responsibilities of the Environments & Stacks working group would be to:
- set appropriate expectations for the individual language SIGs
- drive changes to the package review process and related tooling to facilitate this model
- in particular, moving reviews out of Bugzilla to a dedicated review tool
- provide tooling that makes it trivial for folks that have signed the Fedora Contributor Agreement to:
- select an upstream project
- register it for tracking on release-monitoring.org
- assert that it complies with Fedora's licensing policies
- (optionally) create a COPR repo for it with an automatically generated spec file, and link that repo to release-monitoring.org
- provide tooling that automatically updates a linked COPR repo whenever the upstream project issues a new release
Individual language SIG responsibilities
For this stage, the primary responsibilities of the individual language SIGs would be to:
- ensure that anitya (the software powering release-monitoring.org) can track updates from relevant upstream repositories
- ensure that tools are available to automate generation of SRPM files from upstream package definitions and sources (the results may or may not be compliant with Fedora packaging policies)
- ensure that source-to-image builder images are available to create container images directly from a source repo and an appropriate language runtime image
Relevant online services
Achieving the goals laid out for this stage would involve four key online services:
- release-monitoring.org (powered by Anitya)
- the Fedora Review Service (powered by Fresque, not yet proposed for deployment)
- a new rebasing service based on Rebase Helper
- the COPR build service
Anitya itself shouldn't need any changes, but COPR would need updates to track the links to Anitya monitored packages, and a new supporting service to trigger automated updates and rebuilds when upstream issues a new release.
The Fedora Review Service isn't a live project yet - Fresque exists, but I'm not aware of any currently active effort to get it into production. E&S would need to drive the process of getting that ready for use and deployed.
Rebase Helper is currently available as a command line tool, this would be about setting up a service to run that automatically and feed the results into COPR.
Other projects of potential interest
There are a number of things that could be done to improve Fedora's tracking of upstream projects just by finding ways to adopt and integrate tools already available elsewhere. In particular, ROSA Linux is an RPM-based distribution with a number of interesting automated analysis tools developed by Andrey Ponomarenko.
- ROSA Labs upstream monitoring
- Andrey Ponomarenko's ABI Laboratory (the open source components)
Downstream filtering
The key goal of this step would be to provide self-service filtered subsets of the COPR smorgasboard that meet additional criteria beyond "a Fedora Contributor decided to publish a COPR for them"
Environments & Stacks responsibilities
For this stage, the primary responsibilities of the Environments & Stacks working group would be to:
- drive the addition of a repo tagging and filtering system to COPR
- manage the definition of a set of "standard tags" identifying various attributes about a repo
- work with Fedora QA to enable Taskotron managed CI tasks to set appropriate tags to indicate when repo updates have passed their CI testing
- work with the Fresque developers to enable Fresque to set appropriate tags to indicate when a repo has passed through additional review gates (e.g. packaging policy compliance)
- create tooling to make it easy to define custom repositories based on COPR filtering criteria
Individual language SIG responsibilities
For this stage, the primary responsibilities of the individual language SIGs would be to provide guidance to developers on how to effectively use COPR and Taskotron for automated software CI.
Relevant online services
Achieving the goals laid out for this stage would involve four key online services:
- the Fedora Review service
- the COPR build service
- the Taskotron CI service
- a new filtered repo service called RepoFunnel
The impact on the first three was described in the E&S subsection above. RepoFunnel would be a new service that provided the ability to select a filtered subset of COPR repositories, create a corresponding downstream Pulp repo for them, and automate the process of keeping that repo updated when the query results changed. (I've started working on building a proof of concept for this part, since it's the key to making the whole idea practical)
Downstream integration
This is the point where longstanding Fedora processes take over.
Environments & Stacks responsibilities
At this point, E&S hands over the reins to FESCo, Release Engineering and the Base WG to define policies and procedures. E&S involvement covers ensuring that the handover from RepoFunnel to dist-git is smooth, and preferably automated via Fresque.
Individual language SIG responsibilities
This proposal doesn't change anything here - language SIGs work with FESCo to define appropriate packaging policies, maintain RPM macro packages, etc.
Relevant online services
This proposal doesn't change anything here - dist-git, koji, bodhi, Taskotron, etc