Env and Stacks/Product Requirements Document

Fedora Environment and Stacks Product Requirements Document.

Document Purpose and Overview

What this document describes

This document lists tasks and goals, which must be finished for other products and for making development in Fedora easier.

The purpose of this document is to encompass the following goals:

• What can be done for developers and users in environment (development tools, documentation, examples, latest tools, stable tools)?
• What is needed by other products from the Env and Stacks WG?

Document Structure

The document provides a list of tasks and goals, which are to be completed next year. Other products or users can ask for other requirements, e.g. to package something into an image and mount it somewhere. Complete features can be added into Fedora products, e.g. an SCL into a Fedora product.

Tasks and Goals

The following is a list of goals with a short description.

Testing/additional repositories

• Many features from this WG will need an additional repository, which has to be hosted somewhere (on copr, koji tags). Also, a policy for enabling the repository must be defined.
• A repo with packages only for building. It might help some developers to just build their package without maintaining the build dependencies.
• Repositories with a looser policy than the current Fedora Packaging Guidelines. Examples of what a looser policy would allow would be bundling libraries, overriding things in Base/Core. It is important to note that these repositories would still need to follow Legal/Licensing Policy as the goal for these repositories is to eventually end up being officially distributed by Fedora.

Automation

• Additional repositories with automatically updated SPEC files and RPM packages for packages that are already included in Fedora.
  • Maintainers will benefit from having less work with updating the packages, since they would just need to review the automatically updated package and import it into rawhide.
  • Users will benefit from having the latest versions of software available. Clearly, the users will need to be informed that the provided packages are a preview available AS IS.
  • In a way, such repositories would follow the upstream release schedule:
    • As soon as the upstream makes it available on e.g. CPAN, PyPI, it would be available in the repository.
    • We would need to handle major updates with an API/ABI bump. Checking for ABI/API breakage could be done automatically with api-checker.

• Automated packaging
  • The general idea is to enable easier/quicker packaging of upstream software by generating SPEC files for the packager automatically.
  • Various tools for automatic packaging already exist (e.g. the *2spec and *2rpm tools), so we will use them as a starting point.
  • The critical non-automated part of the packaging and review process that will remain manual will be the licensing checks.
  • By leveraging copr, we could have automatically updated repositories generated automatically from upstream sources.
  • jzeleny is working on other automatic packaging tools.

• Automated package review tools
  • Development of a new service where package reviews would happen from start to finish [1]:
    • No Bugzilla
    • Automatic FAS integration (one could automatically mark reviews that need sponsors)
    • Guided (self-)review
    • Inline comments (like gerrit)
    • FedoraReview integration
    • pingou plans to work on such a tool [2]

• QA automation
  • Join Fedora QA and help them with the Taskotron project concerning the future of QA automation in Fedora.
  • Develop new tools for Fedora QA that will help achieving high standards of software in Fedora.
    • An example of such a tool is rpmgrill, a set of analysis tests that run against a particular RPM build. Its main difference to rpmlint is that rpmgrill handles *builds*, that is, the entire sets of RPMs built from one source RPM file instead of single RPM files. As soon as it is ready for deployment, it should be introduced to Fedora QA.

Integration of Fedora services/tools

• We want to lessen/automate the work of packagers by integrating the Fedora services/tools such as fedpkg, git, Bodhi, Bugzilla.
  • An example of a git-Bugzilla integration: packager's git commit referencing a particular bug number in Bugzilla triggers automatic generation of a comment that includes the commit link and commit message and changes the state of the referenced bug to MODIFIED.
  • An example of a koji-git integration: a successful build of a package in koji triggers automatic generation of a tag in package's git repository so that the packager can easily access the content of the particular build using pure git.
• This will be done in co-operation with the Fedora Infrastructure Team.

Build systems

• COPR & koji
  • COPR able to build an SCL
  • koji - a number of improvements for developers (buildlogs, etc.), in co-operation with the koji upstream.

SCL

• SCL in Fedora.
  • Pending in FPC. Most probably each SCL will be added into Fedora as a system-wide change.
• SCL in SCL Upstream from COPR.
  • Possible to use 3rd party repos for various projects.
  • Almost ready.
• scl-utils v2
  • New features out of scope for this version of the PRD.

CI

Continuous integration (CI) allows early detection of bugs or potential issues, thus it is beneficial for improving and maintaining the high quality of software available in Fedora. A number of tools for CI are available such as Travis CI or Jenkins.

• We will encourage upstream projects to move toward a CI model. Projects based on GitHub can use the hosted Travis CI instance at https://travis-ci.org/ free of charge. Projects related to Fedora can use the Fedora Infrastructure hosted Jenkins instance at http://jenkins.cloud.fedoraproject.org/.

• For CI of RPMs built by koji/COPR and Fedora updates submitted through bodhi, we will cooperate with Fedora QA on the AutoQA tool. Future plans for AutoQA are described in the Taskotron project.

Documentation, guidelines

• Wiki pages.
  • Currently lacking any usable structure, lots of abandoned and outdated content, thus not being very helpful.
  • Look into creating a better structure, improve the related wiki page categorization.
  • Keep the Packaging: namespace, the official Packaging Guidelines are and will be maintained by the Packaging Committee members.
  • Archive the outdated/duplicated/unneeded wiki pages (outside the Packaging: namespace).
  • Motivate people to create new content (with badges, swags,...).
  • Improve the search on Fedora wiki.
  • Add more references to formal documentation (see below).
• Formal documentation.
  • Some people prefer to use a single document to learn about concepts and tasks rather that browsing through a number of wiki pages. Wiki content may not feel qualified and may not invoke the same trust as formal documentation.
  • pkovar will be working on a new Packager's Guide, to help people getting started with RPM packaging for Fedora and EPEL. The goal is to share as much content as possible with downstream RHEL/EPEL documentation, using the same format and toolchain (DocBook, Publican).
  • The guide will be mostly based on the Fedora wiki pages/HOWTOs. It will reference additional resources on the wiki (esp. in the Packaging: namespace) and/or other content where appropriate.
  • An outdated draft is here: http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/Packagers_Guide/index.html
• SCL
  • Continue to improve the SCL packager guide.
  • It currently lives at http://docs.fedoraproject.org/en-US/Fedora_Contributor_Documentation/1/html/Software_Collections_Guide/index.html
  • Using the same format and toolchain as the Packager's Guide and other Fedora guides, it currently shares content with the RH version of the same book at access.redhat.com.
  • Look into upstreaming the guide to SCL Upstream if it turns out not to be possible to share most of the content with the version at SCL Upstream or access.redhat.com (due to limited resources).

DevAssistant

The aim of DevAssistant project (http://devassistant.org) is to help developers with repetitive everyday tasks, such as kickstarting/scaffolding new projects, installing dependencies, working with SCM, setting up development environments and so on.

DevAssistant is a "best effort" service - we can't solve all the corner cases, but we can do good in vast majority of cases.

In current DevAssistant release, 0.8.0, core is starting to be mature and stable and for upstream release 0.9.0 we will be concentrating on providing as much end-user functionality as possible. This contains:

• Providing more assistants for end-users, improving the functionality of current assistant set
• Support for generating (and perhaps modifying) Docker.io images
• Dependency versioning support

User stories

This section lists a couple of concrete user stories highlighting the problems they are facing with the current Fedora and what they would like to see improved in the future.

Persona #1: Alan the Big Data analyst

Alan is a Big Data analyst and member of the Fedora Big Data SIG. He uses a number of applications written in different languages to perform the data analysis. He wants to focus his time and effort in the application and the actual data analysis. He want to minimize the time and hassle spent obtaining, compiling, packaging and maintaining the applications that he needs. The form of packaging (rpm, deb, npm, other) isn't as important to him.

Problem statement: Currently, there is a lot of hassle and pain in dealing with non-primary (i.e. C/C++, Python, Perl) language stacks in Fedora. Although Alan wants to focus his time in the application and the actual data analysis, he too often finds himself spending time managing the language-specific toolchain needed by the application.

Cause #1: The upstream application authors usually assume that any developer would just be using the language-specific packaging ecosystem rather that also taking into account the downstream distribution-based packaging and dependency management systems. This is a problem since there are many differences between language-specific packaging ecosystems and the Fedora way of packaging software.

Cause #2: Many upstream applications use very brittle versioning. In other words, each application expects the user will be able to have its particular versions of the runtime, compiler and libraries available.

Example #1: Applications written in Scala require different versions of Scala, some need v2.9 and some need v2.10. Although Fedora could provide both versions in the same release (via SCLs or by manually maintaining two Scala packages), they would both need to be resolvable via Apache Ivy, rather than just providing both binaries, "scalac29" and "scalac210".

Example #2: The version of Jetty in Fedora does not work with Java 6 and the Apache Hadoop upstream isn't ready to abandon Java 6 yet. So the Big Data SIG ends up maintaining a patch to Apache Hadoop for Jetty 9 that will live purely in Fedora for quite a while to come. Although this could be worked-around by using compat packages for Jetty for a couple of Fedora releases, it just hides the fact that there is a mismatch in expectations between Fedora and upstream application authors.

Example #3: Node.js has its package/dep management tool available in Fedora, but very little of the language ecosystem / base libraries are packaged and available in Fedora.

Possible solution: Fedora would need a way to provide language-specific ecosystems in a way that aligns with how these language itself are used and applications written in them are developed and distributed.

Persona #2: Student or Corporate developer needing multiple development environments

Billy is a CS student with multiple assignments that need different development environments/stacks to build. Bob is a corporate enterprise developer who is both developing new applications using the latest technology stacks while also maintaining older software releases that uses older libraries and toolchains.

Problem statement: While they can setup different OS's or environments in separate virtual machines, it is a lot of work and wastes a lot of diskspace and time. Being able to install multiple environments or versions of stacks in the same Fedora instance and switching between them on a per-project basis would be much easier and more efficient.

About this Document

This PRD (Product Requirements Document) is an evolving document, created by the Env and Stacks Working Group as part of the process for designing the Fedora.next. The PRD is not real PRD, because this WG is not creating a product, but merely list of task and goals of various technologies, which could be once used by Fedora products.

Authors

Contributors to this document include:

• Marcela Maslanova
• Tadej Janež
• Honza Horák
• Bohuslav Slávek Kabrda
• Petr Kovář
• Jens Petersen

Reviewers & Contributors

The following people have contributed to the development of this document, through feedback on IRC, mailing lists, and other points of contact.

Community Information

The Env and Stacks WG mailing list is located here.

Minutes and logs from IRC meetings related to the development of this document should be listed here as the document evolves.

Approval History

There will be added new requirements by other products in the future. The first version should contain what our WG believe we should do.

• version 1

Tracking of Progress

While Fedora uses the Changes Process to track changes in the distribution, those changes are typically described as details of changes to a specific package, or the introduction of a specific package, rather than as a piece of a larger feature set.

This document could possibly be used to do any or a number of the following things:

• Provide a secondary location where changes are tracked (which seems like a lot of overhead to me)
• Provide a location where overall Feature Progress is tracked, via periodic cross-checking against Change pages; this could be either in a standalone section, or simply attached to each Feature description.
• Scope out how features are expected to progress over a number of releases.
• None of these things.

When we more fully determine how to most efficiently track progress, the pointer to where that tracking is done, and/or the description of or process by which we do the tracking is formalized, should be documented in this section in lieu of what is currently written here.

Document Conventions

Definitions and Acronyms

• PRD: Product Requirements Document
• EPEL: Extra Packages for Enterprise Linux
• CI: Continuous Integration
• SCL: Software Collections
• NTH: Nice-to-have
• BZ: Bugzilla
• WG: Working Group
• FAS: Fedora Account System
• API: Application programming interface
• ABI: Application binary interface
• CPAN: Comprehensive Perl Archive Network
• PyPI: Python Package Index