| (13 intermediate revisions by the same user not shown) | |||
| Line 10: | Line 10: | ||
''On-Demand build service'' seeks to build | ''On-Demand build service'' seeks to build Live and installation images for developers, testers and consumers of Fedora Linux. | ||
During the testing of Fedora releases, test images are often useful as smoke tests before full TC/RC composes, as baselines for specific test days or for automated installation testing in AutoQA. The idea is to make an on-demand Web-based build service which users/developers can use to make custom Fedora based distributions. | During the testing of Fedora releases, test images are often useful as smoke tests before full TC/RC composes, as baselines for specific test days or for automated installation testing in AutoQA. The idea is to make an on-demand Web-based build service which users/developers can use to make custom Fedora based distributions. | ||
The service would be capable of building and hosting images (boot iso, installation DVDs and live images) made up of builds from stable repositories in addition to side repos containing specific builds from both updates-testing and koji builds that have yet to be pushed to any repos. The service will also also have a RESTful API which will make it accessible from command-line clients as well. | The service would be capable of building and hosting images (boot iso, installation DVDs and live images) made up of builds from stable repositories in addition to side repos containing specific builds from both updates-testing and koji builds that have yet to be pushed to any repos. The service will also also have a RESTful API which will make it accessible from command-line clients as well. | ||
The toolchain for the installer creation will be changing from Fedora 18. While working on this project, I shall keep this in mind and discuss with the mentor how to best keep a provision for incorporating the usage of the newer/updated tools causing a minimum disruption of the code base of the build service. | |||
== Any relevant experience you have == | == Any relevant experience you have == | ||
As the creator and maintainer of the Fedora Scientific Spin, I am well aware of the LiveCD creation tool chain | As the creator and maintainer of the Fedora Scientific Spin, I am well aware of the LiveCD/DVD creation tool chain. | ||
I am also knowledgeable about creating local repositories (side repositories) and pulling in packages from there into the installer/Live CD. I am currently exploring building installation images. | |||
== How do you intend to implement your proposal == | == How do you intend to implement your proposal == | ||
'''Creation of side-repositories''' | *'''Creation of side-repositories''' | ||
As a first step, I am experimenting with creating a local repository (side repo) with builds of packages, which are not yet available in the main (either updates/updates-testing or Koji). So far, I have experimented with creating a Live CD having packages from this repository. The next step is to attempt to create installer images. | As a first step, I am experimenting with creating a local repository (side repo) with builds of packages, which are not yet available in the main (either updates/updates-testing or Koji). So far, I have experimented with creating a Live CD having packages from this repository. The next step is to attempt to create installer images. | ||
'''Web-based Interface and Job delegation''' | *'''Web-based Interface and Job delegation''' | ||
The next step would be to implement an interface (web-based) whereby the user can select the packages he/she wants to be in the installer/live CD. If these packages are available in the main, they are taken from there, else a side repository is built using the packages from updates/updates-testing or Koji. Once the packages are all available, a kickstart file (for Live images) and other necessary information will be generated at the '''master''' node. The master node then delegates this job to a job queue. The job queue management system will be responsible for handing over the jobs to a '''free build node'''. | |||
*'''Notification''' | |||
On successful/unsuccessful build, an email is sent to the user with appropriate information. If the build was successful, a link is sent to the user to download the image and logs. In case of a build failure, the build logs are sent to the user so that the cause of the error is known. | |||
''' | *'''Image Hosting ''' | ||
Once the image is built successfully, the image is pushed by the slave to an image hosting provider along with the logs. | |||
* '''Build Request/Image download Regulation''' | |||
The proposed service will have the facility to regulate users by the maximum number of builds per day or similar and also for the number of times a particular image can be downloaded. This will enable fair use of the system for multiple users. This will depend on the number of build nodes (client nodes) available to the system. | |||
== Final deliverable of the proposal at the end of the period == | == Final deliverable of the proposal at the end of the period == | ||
* | * The build service running as Python web application on the '''master node'''. | ||
* Client build services running on designated '''build nodes''' | * Client build services running on designated '''build nodes''' | ||
* A REST API to the build service | * A REST API to the build service | ||
* '''Fedora RPMs:''' The master node build service and client side services will be made available as RPMs. | |||
== A rough timeline for your progress == | == A rough timeline for your progress == | ||
* '''Current - May 21:''' Implementing an automated script which if given a list of packages, retrieves the ones not available in the main and creates a side-repository, and building the web interface along with a REST API implementation to the build service. | |||
* '''May 22 - June 10:''' Finalize the image creation scripts and have a system containing a master node and a single build node in place. Work on implementing the client service which will run on the build node. | |||
* '''June 11 - Mid-term evaluation:''' Formalize the Image hosting node specifications - FTP/HTTP interface. Evaluation and initial implemention of a job queueing system. | |||
* '''Mid-term Evaluation Deliverable''': Build service running with the capability of building images and storing it in the image host(s). Multiple build nodes may not be supported at this stage. | |||
*''' July 20 - August 10:''' Formalize a way to distribute build jobs. Final testing and Documentation | |||
==Any other details you feel we should consider== | ==Any other details you feel we should consider== | ||
=== Have you communicated with a potential mentor? If so, who? === | === Have you communicated with a potential mentor? If so, who? === | ||
Yes, I have been in touch with [https://fedoraproject.org/wiki/User:Tflink Tim Flink]. Tim contacted me upon seeing my proposal on the Fedora GSoC wiki and he has helped me develop my initial idea to make it into the current proposal, which will definitely help the Fedora QA and testing community among others. | |||
[[Category:Summer_coding_2012]] | [[Category:Summer_coding_2012]] | ||
Latest revision as of 14:26, 4 April 2012
On-Demand Fedora Build Service
Contact Information
- Name: Amit Saha
- Email Address: amitksaha@fedoraproject.org
- Blog URL: http://echorand.me/category/fedora/
- Freenode IRC Nick: amitksaha
Why do you want to work with the Fedora Project?
My second stint as a Fedora user started about an year back, and liked it much more than I had about 10 years ago when I first started using Linux. I was nursing the idea of a custom Linux distribution for scientific and numerical computing for quite some time. I started exploring the options of creating this spin based on Fedora, and the Fedora project's easy and transparent way to propose a custom spin attracted me. In no time I had the Scientific Spin's process rolling. The Scientific Spin was released along with Fedora 16, thanks to the welcoming and co-operative nature of the Spins SIG, release engineering and the Fedora design and websites team.
This first involvement and my growing understanding of the Fedora community makes it a first choice for me for GSoC 2012. Needless to say, my involvement won't start or cease with GSoC.
Past involvement with Fedora Project:
- Creator and Maintainer of the Fedora Scientific Spin
- Mailing list moderator of the SciTech SIG
- Packager (Review awaited)
Past involvement with other Open Source projects:
Did you participate with the past GSoC programs, if so which years, which organizations?
Not yet
Why should we choose you over the other applicants?
There are a few things which should definitely give me a head-start over other applicants
- My experience of creating and maintaining the Fedora Scientific Spin
- My experience packaging for Fedora
- Existing experience with the toolchain (creating/building Fedora live CDs, kickstart files, installers) desired for the proposed GSoC 2012 project
- Possession of Generic toolset knowledge - community and mailing list interaction, version control system and programming knowhow - Shell scripting, Python scrpting and working with the Fedora tools such as Koji, Pungi, Lorax, etc.
- Documentation is important. My articles have appered in Linux Journal, Linux For You, Linux Gazette and elsewhere. So, I actually like documentation.
Proposal Description
This page shall describe the proposed project.
Overview and The Need
On-Demand build service seeks to build Live and installation images for developers, testers and consumers of Fedora Linux.
During the testing of Fedora releases, test images are often useful as smoke tests before full TC/RC composes, as baselines for specific test days or for automated installation testing in AutoQA. The idea is to make an on-demand Web-based build service which users/developers can use to make custom Fedora based distributions.
The service would be capable of building and hosting images (boot iso, installation DVDs and live images) made up of builds from stable repositories in addition to side repos containing specific builds from both updates-testing and koji builds that have yet to be pushed to any repos. The service will also also have a RESTful API which will make it accessible from command-line clients as well.
The toolchain for the installer creation will be changing from Fedora 18. While working on this project, I shall keep this in mind and discuss with the mentor how to best keep a provision for incorporating the usage of the newer/updated tools causing a minimum disruption of the code base of the build service.
Any relevant experience you have
As the creator and maintainer of the Fedora Scientific Spin, I am well aware of the LiveCD/DVD creation tool chain.
I am also knowledgeable about creating local repositories (side repositories) and pulling in packages from there into the installer/Live CD. I am currently exploring building installation images.
How do you intend to implement your proposal
- Creation of side-repositories
As a first step, I am experimenting with creating a local repository (side repo) with builds of packages, which are not yet available in the main (either updates/updates-testing or Koji). So far, I have experimented with creating a Live CD having packages from this repository. The next step is to attempt to create installer images.
- Web-based Interface and Job delegation
The next step would be to implement an interface (web-based) whereby the user can select the packages he/she wants to be in the installer/live CD. If these packages are available in the main, they are taken from there, else a side repository is built using the packages from updates/updates-testing or Koji. Once the packages are all available, a kickstart file (for Live images) and other necessary information will be generated at the master node. The master node then delegates this job to a job queue. The job queue management system will be responsible for handing over the jobs to a free build node.
- Notification
On successful/unsuccessful build, an email is sent to the user with appropriate information. If the build was successful, a link is sent to the user to download the image and logs. In case of a build failure, the build logs are sent to the user so that the cause of the error is known.
- Image Hosting
Once the image is built successfully, the image is pushed by the slave to an image hosting provider along with the logs.
- Build Request/Image download Regulation
The proposed service will have the facility to regulate users by the maximum number of builds per day or similar and also for the number of times a particular image can be downloaded. This will enable fair use of the system for multiple users. This will depend on the number of build nodes (client nodes) available to the system.
Final deliverable of the proposal at the end of the period
- The build service running as Python web application on the master node.
- Client build services running on designated build nodes
- A REST API to the build service
- Fedora RPMs: The master node build service and client side services will be made available as RPMs.
A rough timeline for your progress
- Current - May 21: Implementing an automated script which if given a list of packages, retrieves the ones not available in the main and creates a side-repository, and building the web interface along with a REST API implementation to the build service.
- May 22 - June 10: Finalize the image creation scripts and have a system containing a master node and a single build node in place. Work on implementing the client service which will run on the build node.
- June 11 - Mid-term evaluation: Formalize the Image hosting node specifications - FTP/HTTP interface. Evaluation and initial implemention of a job queueing system.
- Mid-term Evaluation Deliverable: Build service running with the capability of building images and storing it in the image host(s). Multiple build nodes may not be supported at this stage.
- July 20 - August 10: Formalize a way to distribute build jobs. Final testing and Documentation
Any other details you feel we should consider
Have you communicated with a potential mentor? If so, who?
Yes, I have been in touch with Tim Flink. Tim contacted me upon seeing my proposal on the Fedora GSoC wiki and he has helped me develop my initial idea to make it into the current proposal, which will definitely help the Fedora QA and testing community among others.