Document Purpose and Overview
This is the Product Requirements Document for the Fedora Atomic Working Group. It:
- Provides a high-level market of the cloud computing market as it pertains to Fedora Atomic; this includes overviews of things which may not be within our actual scope/ability to accomplish at the current time.
- Provides deeper understanding of the types of users who could use Fedora for their application deployment needs. This includes describing their main day-to-day tasks, common problems, and so on. The perspective here is not necessarily limited to system administrators, or developers, but a combination of many types of users and roles.
- Ties common issues and needs of potential users and consumers of the Fedora Atomic product to high-level product needs, from a "functional" standpoint
- Provides solutions, in the form of "changes" or "features" which will provide the functionality described as needs for the potential users.
Release/Product Overview
Fedora Atomic provides a lightweight, immutable platform, designed with the sole purpose of deploying and scaling containerized applications on public and private clouds, as well as on virtual and physical infrastructure.
Market Opportunity
The market around Linux containers and "serverless" architecture is growing rapidly, as are the technologies and processes that enable it.
From public and private clouds to traditional virtualization systems and bare metal environments, there's an opportunity for Fedora to unseat incumbent distributions and strengthen its overall adoption by better serving these new application deployment models.
Fedora, being a leading-edge Linux distribution, is the ideal place to collaborate on fast-moving components such as the Linux kernel, the docker runtime and the kubernetes orchestration system, particularly when combined with image-based system updates via rpm-ostree.
Longer term, the technologies included in Fedora Atomic for enabling atomic system updates and containerized application hosting will have relevance to other Fedora products, there's an opportunity for this work group to help develop and spread these technologies to other parts of the Fedora community.
Product Objectives
The primary deliverable of the WG is the Fedora Atomic Host, a lightweight container OS image composed from standard Fedora rpms using rpm-ostree. Initially, the Atomic Host is focused on the docker runtime and on kubernetes orchestration, but will allow for the use of other runtimes and orchestration technologies.
Fedora Atomic Host can be used at various scales. For example, it could be used in the following example environments:
- Running as a VM on a developer workstation.
- Running on bare metal or virtual machines in a data center.
- Running as a cloud image on a public or private Infrastructure-as-a-Service (IaaS) cloud.
- Running as the embedded OS in an IoT device
Secondary Objectives
1. Production, promotion, and distribution of Fedora-based container images. Initially this is via the Fedora Docker Layered Image Build Service. This includes:
* encouraging production of a broad and diverse library of container images * qualifying images and ensuring that guidelines for quality images are followed * promoting Fedora-based images to application developers and operators
2. Development, maintenance and distribution of a Fedora-based full-stack container cluster environment
* based on OpenShift Origin * includes work on development of installation and deployment tools for easy user start-up
Target Market / Audience
Developers and operators creating containerized workloads, and organizations and users running those applications. Fedora is particularly interested in organizations that might want to contribute back to Fedora and be involved in helping find/report/fix bugs, and develop new features. Organizations with a strong interest in an immutable OS even without containers, such as "Internet of Things" system builders.
Delivery Mechanisms
Atomic Host images must be easy to consume as part of a public or private cloud, as virtual machines, or on bare metal. Because we target these environments, we'll produce ISO images, downloadable VM or cloud images, and vagrant images, as well as work to make Atomic Host images available within popular public cloud services.
Where to obtain
Users will be able to obtain the images for public clouds via download or via the usual marketplace for those images. For instance, we publish AMIs directly to the AWS Marketplace. Users are able to launch new instances with Fedora without having to obtain the images directly from the Fedora Project and then upload to AWS.
Users will be able to download appropriate images for OpenStack, as well as ones usable on other public clouds where we don't have direct support.
Delivery Format
Images will be delivered as AMIs on Amazon EC2, as downloadable images in qcow2 and raw.xz formats, as vagrant images, and as installable ISO images. We may add other public cloud images and other downloadable formats to meet demand or anticipated need.
Architectures
We are targeting 64-bit x86. As new community members join who are willing to maintain other architectures, we will add ARM and/or POWER images.
Updates
Fedora Cloud images can be updated using rpm-ostree using the command "atomic host upgrade." This is somewhat like a yum update, except it updates all of the content as an atomic unit. We also intend to produce periodically respun images with security updates, every two weeks. Further, Atomic will offer a "developer" stream which distributes daily or more frequent builds based on project CI.
Measuring Success
Success looks like:
- Increase in adoption.
- Third party support / targeting of Fedora Atomic as a platform.
- Increased contribution and participation in the Fedora Atomic WG and Fedora Project in general.
- Creation of new stacks by third parties based on Atomic and OSTree.
- Increase in the quality and quantity of Fedora containers using the Layered Build Service
User Profiles, Goals, and Primary Use Cases
User Profiles
There are three cloud user roles which describe the tasks of the people who interact with any cloud-based system:
- Cloud Service Creator: Develops the technical and business aspects of a (simple or high-level) cloud service
- Cloud Service Provider: Provides all types of services to a Service Consumer
- Cloud Service Consumer: Consumes all types of services offered by a Service Provider
We will use a set of personas to describe our target users and their respective needs. These users are primarily in the cloud service creator and cloud service provider roles. This document lists the personas in summary form, with detailed explanations of each one available in a separate document. https://fedoraproject.org/wiki/Cloud_Personas. These roles include aspects of cloud service creators, providers, and consumer.
- AWS enthusiast and Early Adopter
- Writes and maintains a number of AWS-deployed applications including staging and production.
- Works for a small start-up, primarily on his own. Automates as much as possible.
- DevOps Team Member
- Uses the cloud to do Ruby on Rails development utilizing virtual machines.
- Works as part of a DevOps team responsible for all aspects of a set of applications.
- Development, QA, and Production environments need to be identical.
- Web Developer
- Concentrates on app development, not architecture, deployment, or operations.
- Unlikely to care deeply about OS internals, cares that they can easily deploy the frameworks they want to use.
- Large-Environment Sysadmin
- Interested in deploying self-service PaaS to lighten operational load
- Will work closely with development team on deployment and development.
- Data Scientist
- Works with large data sets
- Interested in big data tools and dedicated scripting languages (R, Octave, etc.)
- HPC Scientist
- Moving from traditional batch and grid technology to the cloud.
- Interested in hybrid cloud (infrastructure overflow) for some massive computation campaigns.
Primary Use Cases
Web Application Deployment in a Public Cloud
Modern web applications are deployed as a collection of interconnected services, including parts like web servers, application servers, databases, and caching layers. Fault tolerance is handled at the overall orchestration level rather than by individual instances. Fedora Atomic can be the base of each of these parts, providing recent libraries, server software, and language toolchains. Each system will be managed using the public cloud's own management tools plus an orchestration system, such as OpenShift or Kubernetes. Code may be developed against different language stacks; if these are readily available to users, that's a win. In the real world, libraries and other code we don't include may also be required and we should make that as painless as possible. Fedora Cloud will provide access to the following stacks (not exhaustive):
- Ruby: Rails, Sinatra
- Python 2 & 3: Django, Pyramid, Flask
- PHP 5.x: Symfony2, ZendFramework 2.x
- Node.js
- Java
- Perl 5.x
- Go
Web Application Deployment in Private Cloud
As above, but in a locally-deployed and managed private cloud system.
Web Application Deployment in Hybrid Cloud
As above, but rather than a single cloud provider, the application seamlessly takes advantage of resources in both a private and public cloud.
Hybrid Cloud as Staging Area
Application development and testing happens in-house on a private cloud, while production goes to a public cloud. Possible scenarios: Using an open source IaaS stack internally and externally (e.g. OpenStack internally, Rackspace externally; Eucalyptus internally, AWS externally; CloudStack internally, any of the CloudStack providers or AWS externally; or using a library like jClouds to smooth out the bumps.) This may include identical software stacks which run in different clouds, or cloud-specific tailored images onto which an application is deployed, or even very different environments under which the same Docker container is run.
Simple Deployment of Code from Development to Production
As a developer, I want to ensure that my code is easily deployable from my development environment to a QA environment and finally a production environment, without encountering compatibility issues or surprises from differences in the environment. A solution where an identical customized image or container (such as Docker) is used in all three environments is ideal.
Target IaaS environments
The Fedora Atomic product can be used as a guest/VM/image under many IaaS services and providers. For projects that are not currently packaged within Fedora/EPEL, we may need to locate a kind person to ensure testing. These include: Open source IaaS systems:
- OpenStack
- Eucalyptus
- Apache Cloudstack
- OpenNebula
- oVirt
Public clouds:
- Amazon EC2
- Google Compute Engine
- HP Cloud
- Rackspace
- Digital Ocean
- Linode
Features
Feature: Ready to run in Public and Private Clouds
The Fedora cloud image is ready to go out of the box in the public and private cloud environments we target. It includes cloud-init, the defacto standard for boot-time configuration for cloud instances, and heat_cfgtools package, which is used for handling guest initialization with OpenStack Heat.
Feature: Ready Access to Fedora Collection of Packages
- Access to the standard Fedora package repositories
- Standard language stacks (like Python, Ruby, NodeJS, PHP)
- Modernized language and application-specific stacks, perhaps in the form of SCLs, for running userspace applications
Feature: Atomic Host
The Atomic Host provides an easy means to keep your container infrastructure up to date. Atomic provides git-like updates and roll backs of your host packages so your containers can keep running, secure and stable. The Atomic spin will provide updates every two weeks, ensuring that your host receives the latest system updates.
Feature: Dockerfiles
Fedora Dockerfiles provides a curated list of Dockerfiles designed to work with Fedora - making both development and deployment of your Docker infrastructure easier. By maintaining a list of supported Dockerfiles, we're able to better saturate the container ecosystem, making it easier for developers to use Fedora in their stack from end to end.
Requirements
Requirement #1: Reducing Image Footprint
Making the Fedora Cloud image fit in a smaller space. Storage is a significant cost element on cloud platforms.
Feature Addressed
Ready to run in Public and Private Clouds.
Priority
Nice to Have. The current cloud image is reasonably sized when compared to our competitors. However, smaller footprint has several advantages. Fewer packages means fewer updates and a smaller target for security problems. It makes it faster to download and deploy images across networks where the image storage and compute nodes run separately. And, reducing things our users don't really need gives more room to include things that they do.
Specific Areas
This requirement has several areas where effort will yield meaningful results. Each has its own level of effort, stakeholders, and dependencies.
Cloud-Focused Base Kernel
The cloud-focused base kernel will be a minimal kernel without many of the drivers traditionally necessary on hardware, like most PCI devices. The goal of having a custom kernel package is ultimately to reduce the size of the images which will reduce storage costs, make the images easier to move across networks between compute and image storage nodes.
Note that the kernel binaries will be shared across all of Fedora; the difference will be that not all modules will be packaged and included by default in the cloud image. This also means that it will be simple to add the other drivers if they are needed for special cases.
Effort is moderate; it includes defining the exact requirements of the smaller kernel, and creation and maintenance of that separation by the Fedora kernel team.
Internationalization / Localization
Fedora Cloud base images should only include a single default locale since it is meant to be used as an deployment target to save space. Other locales should be available through repositories. The idea is to rely on langpacks rather than the current RPM kludges which leave out internationalization, because that approach does not allow one to easily add language support later. Effort required is fairly high, because the packaging tools do not currently have such support. It will also likely have some impact on all Fedora packagers, depending on the implementation, and may also involve changes to the packaging guidelines.
Included Documentation
Documentation has the same issue as internationalization: it is often not needed on each image but takes significant space, and cannot be easily added back if RPM is configured to exclude it initially. Effort is similar to that for internationalization, although significant impact could be made by breaking out documentation from key packages included in the base image.
Refactoring Cloud-Init
Cloud Init was initially designed for a different distribution and is only loosely tailored for our needs. As it stands, it pulls in a rather large set of packages not used for other things. It is also written in Python, itself a large subsystem which it would eventually be nice to leave out of the base. Effort is moderate, with some low-hanging fruit which may be addressed easily.
Requirement #2: Improved Docker Integration
Docker is a popular tool for launching and managing application containers. This runs on bare metal, in virtualization on developers' machines, or in cloud guests. Right now, Docker can be added to the cloud base image but isn't included by default. Docker itself is small, but infrastructure to support it includes Libvirt, which is not small. In order to make it easier to get started and to speed up use in production, we will produce a Docker-specific image. Additionally, we want to create a library of Dockerfiles and possibly our own registry of images.
Feature Addressed
Docker support.
Priority
Should. This feature is an important change to support a variety of different teams working on PaaS (like OpenShift) and end-user delivery issues. That being said, since there is very little usage of Docker in those tools right now and the package is available in the repositories already, it won't be terribly detrimental to not have the image complete.
Effort required
High. We can produce the basic image with minor work, but SELinux is a key feature which will make us stand out from other systems on which Docker can be run. Additionally, producing official Docker container images depends on changes to the build system.
Stakeholders / Owners / Major Dependencies
The Cloud SIG and our users are the major stakeholders. The SELinux team is instrumental in delivering the functionality we would like to see. Fedora Infrastructure and Release Engineering are involved in updating the build system. And the official images will be produced through Release Engineering. Major dependencies include:
- Better SElinux MCS separation incorporated in the policies shipped in Fedora.
- Creation of an agreed upon set of Dockerfiles for different tasks.
- ImageFactory support for creating Docker images, and that support integrated into Koji.
Requirement #3: Automatic Production and Upload of Images
We will develop Fedora Cloud images using text-based descriptive files and use an automated toolchain that generates images for our targets and upload them where appropriate. Currently, this is done by hand. Automation is necessary to scale to multiple supported images and a growing set of targets. Fedora images must pass automated QA before publication to ensure a minimum level of quality. In order to release updated images with confidence, automated testing is required, because we cannot expect the level of human attention which can be brought to a twice-yearly release.
Feature Addressed
Updated Images.
Priority
Must. We simply need this to be useful to our users, and therefore to be competitive.
Effort required
High. Work is currently ongoing with Release Engineering and QA to facilitate this process.
Stakeholders / Owners / Major Dependencies
The Fedora Cloud Working Group will work with Release Engineering in implementing and supporting the needed functionality.
Requirement #4: Software Stacks for Various Languages
People developing and deploying code on top of Fedora need access to the relevant language stacks. Fedora already provides the latest versions of many of these, but often real-world version dependencies do not match the Fedora cycle. We need some way of offering multiple versions of languages and software stacks. Additionally, it is ideal if support for a given version lasts longer than the 13-month Fedora lifecycle, so that moving from one release of the base to the next can be less painful. The Cloud SIG plan to support the major languages:
- Ruby
- Python 2 & 3
- PHP 5.x
- Java
- Node.js
- Perl 5.x
- Go
We will provide infrastructure and support to volunteers who want to complete the supported stacks or bring new ones.
Feature Addressed
Ready Access to Fedora Collection of Packages.
Priority
Must. This addresses a key need of userbase.
Effort required
High. Access to existing Fedora packages is simple (we simply won't break what currently works), but Fedora does not currently have a good infrastructure for multiple versions of stacks or for decoupled lifecycles.
Stakeholders / Owners / Major Dependencies
Software in this area will be the responsibility of the Environments and Stacks working group.
Requirement #5: Integration with Fedora Server Roles
We simply want to make sure that the tools to convert a Fedora Cloud base image to a Fedora Server role are available and function.
Feature Addressed
Cloud -> Server
Priority
Should. This helps tie the Fedora products together and makes the blurry lines between the Cloud and Server product more understandable.
Effort required
Medium. Effort is primarily in coordination and testing.
Stakeholders / Owners / Major Dependencies
This effort will be done in collaboration with the Fedora Server Working Group.
Requirement #6: Tools for User Image Creation and Image / Template Library
We do not have a standardized process for end-user image creation, although we have several tools which can be used. We plan to develop this process and document it. This may also include an image library or a library of image creation templates, or both.
Features Addressed
Image Generation Tools, Image or Image Template Library
Priority
Nice to have. This is not required for our initial launch.
Effort required
High.
Stakeholders / Owners / Major Dependencies
Image creation tools (Oz/ImageFactory, etc.), Fedora Websites, Fedora Documentation, Cloud SIG users in general.
Requirement #7: Updated Web Site for Obtaining Cloud Images
As we provides images for various use cases (ie: Big Data, PaaS hosting, scale-out apps, etc.) and various target platforms, we need an updated website. Fedora provides its build toolchain and encourage the community to use it to build new products. The updated website should also reflect that by allowing folks to share and review the work done by their peers.
Feature(s) Addressed
Easy access to Fedora Cloud products and encouraging sharing in the community.
Priority
Must. A simple access to the cloud products is also a key argument to their success. Should. Providing a sharing platform to the community (as in not supported by the Cloud SIG) and peer-review based system will allow to service better our users and foster innovation.
Effort required
High. We need to define precisely what we want to show, and maintain the website.
Stakeholders / Owners / Major Dependencies
The Cloud SIG will work closely with Fedora Websites/Infrastructure and possibly Fedora Engineering if we need to develop specific apps and/or integrate with the existing Fedora infrastructure (like fedmsg).
Documentation
We aim to provide a comprehensive documentation covering users to developers. Should. Provide narrative documentation to make it easier to use cloud images and use them as foundations to new products. NTH. On a volunteer basis, we will maintain a knowledge base.
Fedora Project Documentation
The Cloud SIG will cooperate with the Documentation Project to provide documentation to users, and developers of Fedora Cloud Products.
Open Source Projects documentation
Ensuring that Fedora is well-represented, up-to-date in other open source project documentation...
About this Document
This PRD (Product Requirements Document) is an evolving document, created by the Fedora Cloud SIG Working Group as part of the process for designing the Fedora Cloud product. The framework for the PRD itself is currently in a draft state. This document will evolve over time as the purpose of the SIG continues to be determined as the working group process gets under way and initial products start to get produced.
Authors
Contributors to this document include:
Reviewers & Contributors
The following people have contributed to the development of this document, through feedback on IRC, mailing lists, and other points of contact.
Credits
Some of the Cloud User Roles are based on “Description and Application of Core Cloud User Roles” ACM CHIMIT 2011, December 4 2011. Some aspects of Fedora Cloud personas are based on OpenStack personas (licensed under CC-By 3.0)
Community Information
The Fedora Cloud SIG is one of many teams within the Fedora Project. The Cloud SIG mailing list is located here. Minutes and logs from IRC meetings related to the development of this document should be listed here as the document evolves.
Approval History
Over time, it is expected that this document will undergo various rounds of review, approval, and editing; in the future, it may be rewritten for different releases of Fedora. While one can review the history of a wiki document (by clicking the "history" tab), it is useful to provide explicit indicators of any major format changes, approvals, or indications of it being in a “final” state, in a list that can allow someone to quickly see that all of the prescribed layers of approval have occured.
- January 22, 2014: Approval by FESCo
- January 8, 2014: Collaborative hackfest day.
- October 28, 2013: Initial Draft of template.
- FutureDate: Approval by SomeGroup; link to any pertinent mail announcement and/or meeting minutes
Document Conventions
Definitions and Acronyms
- AWS: Amazon Web Services
- Amazon EC2: Amazon Elastic Compute Cloud, a popular public IaaS, part of AWS
- IaaS: Infrastructure as a Service
- PaaS: Platform as a Service
- SaaS: Software as a Service
- PRD: Product Requirements Document
- EPEL: Extra Packages for Enterprise Linux
- CI: Continuous Integration
- CD: Continuous Delivery or Continuous Deployment
- SCL: Software Collections
- teams in charge of some aspects of Fedora Project
- NTH: Nice-to-have
- BZ: Bugzilla
- GUI: Graphical User Interface
- CLI: Command Line Interface
- API: Application Programming Interface
- RDBMS: Relational DataBase Management System
- FESCo: Fedora Engineering Steering Committee