Category:Copr: Difference between revisions

Copr (Cool Other Package Repo) is a Fedora project to help make building and managing third party package repositories easy. The instance to be installed within Fedora Infrastructure provides Fedora maintainers with the ability to create repos of packages to build against and share with others.

Some Design Requirements

• Copr need their own builders to protect against badness when installing pkgs for builddeps into buildroots
• the system hosting this becomes more critical to infrastructure so it's probably best not to reuse fedorapeople for the hosting

Design Overview

• We're creating a new build system that's more lightweight than koji to host this. the build system will also be named Copr. It will have a small web UI and a command line client. We'll create this using TG2 as it's something we're familiar with.
• Copr will send tasks to a generic task scheduler that will drive the actual builds.
• For now, the task scheduler will communicate with the builders using func. (toshio working on this)
• The builders need to have a vm based mock to do the actual builds to protect against malicious packages compromising the builder for future builds (skvidal working on this)
• People consuming from the Copr repos need to have a WebUI for this
• Yum plugin to make configuring copr repos (with inter-repo deps) work
• Hardware to test this on (mmcgrath has provided us with test hardware) and hardware to run it in production (spot when we're ready)
  • host for builders
  • host for task scheduler
  • host for web ui (app servers?)
  • shared storage for repositories, uploaded srpms, and other things all builders need access to
  • host for exposing the repositories to people downloading packages from copr (app servers?)

Packager workflow

• tool to submit an srpm to the above (using their fedora certs/account) and specify any targets/repos it should use
• user is prompted to sign the built packages (or not)
• puts them into the repo they are being built for

Scripts to start on

• sync-to-repo: takes a dir of localpkgs and remote read and write path
  • syncs down the repodata from the remote read path and merges the localpkgs into that path and pushes the data back up via the write path.

http://skvidal.fedorapeople.org/misc/sync_to_repo.py

Design work

Usage Cases

• Bob would like to try out a new version of GTK2. Bob does a copr build of the new GTK2 package, and then builds a number of other packages against this. Bob shares his gtk2newness copr with others for validation of the new GTK2

• Jill would like to experiment with rpm using a different compression method for packages. Jill builds a modified version of rpm for her rpm copr and builds a number of other packages with this rpm. Jill shares her rpm copr for further testing of the rpm change.

• Dave would like to experiment with kernel patches and have the repo creation handled for him. Dave continues to add patches to kernel source rpms and builds them for his copr(s). Dave shares his various kernel coprs with the community for early testing and validation. Dave spends more time working on kernel and less time dealing with repo management.

• Fedora Infrastructure would like to create a repo with updates to RHEL packages that it needs in order to manage its systems. It would like other people to be able to use the repos so that they can replicate their environment.

• The KDE SIG wants to have a repository where they can perform major upgrades that their users want but don't follow the Fedora update guidelines. By publishing to a copr repo the packages don't have to go into the main Fedora repositories but they are able to point to it for people who want to run later versions.

• The KDE SIG wants to test their updates for a Fedora release as a set so they can find regressions between the components and the release. They publish the packages to a copr repo and have their testers pull updates from there.

KDE RedHat

kde-redhat is a large external repo, asked rdieter how he manages it so far.

All I'm doing really is building stuff, either in koji or my own mock instance. Grabbing those rpms, signing them, stuffing it into repos. Whatever kopers end up being, can't be much worse than what I'm doing now. honestly.

• repos managable by > 1 person (ie, a sig)
• custom buildroots (ie, to use itself or other kopers). we've had repo heirarchies do something like that already kde-unstable depends on kde-testing depends on kde-stable
• easy mirroring, I suppose, would be nice too
• not sure what kind of workflow to get packages into kopers would look like. something bodhi like, or maybe doesn't even have to be that complicated
• maybe we could get some inspiration from how the opensuse buildservice works though I've only spent a couple of hours looking at it.

imo kde redhat is beyond the scope of Copr - it's WAY beyond 1GB per user - skv

Resources

A Fedora maintainer would be alloted up to 1GiB of storage for package repos. Packages can be built for any arch Fedora builds for (but doesn't have to be all of them). Copr would have repoview content for easy browsing and subscribing to changes. Copr's could be increased by request.

Work Flow

1. Bob requests a Copr repo via the Copr WebUI -- defines what other repos are dependencies
   • The Copr service creates an empty repo and generates an rpm that has yum repository definitions for the repo and dependencies on other repo-configuring rpms
2. Bob constructs an srpm of gtk2
   • Not a Copr issue
3. Bob issues a command (cu build "gtk2newness" gtk2.src.rpm ?) to build the gtk2 package.
   • first this checks the pkg and looks at the License tag to make sure it's not something we absolutely cannot build.
   • this just delivers the srpm to the shared filesystem and then kicks off a build task for it in headhunter.
   • Notes about building from SCM rather than SRPM; SCM server needs to be accessible from builders (currently, within Fedora's network)
   • Building from SCM is not necessarily as important as being able to share changes via the SCM. If I can clone the SCM repo and you can as well to make changes but we create an SRPM as an intermediate step for build that's not much of a burden.
4. Copr creates a buildroot based upon the rawhide collection and any packages in Bob's "gtknewness" copr.
   • two issues here: getting koji to use arbitrary repos
   • not having the above beat the crap out of fedorapeople with koji constantly hitting it for pkgs/repodata
   • These are some of the reasons we need a different build system and different storage.
5. Copr/headhunter builds Bob's gtk2
6. headhunter takes the koji built package and adds it to Bob's "gtknewness" copr, recreating yum repodata and repoview content.
   • this means having a task to build repodata
   • need something in here for SIGNING those pkgs, too.
     • Probably want a separate key for each Copr repo.
     • why a separate key? if I'm building these for ME and mine there is no reason why I shouldn't use the same key
     • Repos can have multiple people adding to them so there needs to be one key for it.
     • you could use the same key for all repositories that only you contribute to but then we'd need to keep track of two different types of repository
7. Bob proceeds to create srpms for various other packages and builds them in koji against his "gtknewness" koper.
   • having koji know the repo exists or to have it know when the repo no longer exists.
8. Bob shares his "gtknewness" koper with the world
   • this is just advertising a url
9. Bob creates a Desktop Live image based on rawhide and his "gtknewness" koper and shares that with the world too.
10. Bob fixes bugs with his gtk2 and continues to do builds until everybody is happy
11. Bob commits his gtk2 changes to package SCM and does a normal koji build.
12. Bob profits.
    • These last 4 items don't matter at all they are outside of scope, imo.

Content

Content within one's Copr(s) must be content acceptable for distribution within Fedora.

• Manual testing, reaction is one possibility
• Review process for not-yet-in-Fedora-items (just license review probably all that's needed)

Changes Needed

To create a koper system various tools in the Fedora infrastructure would need to have changes made to them.

Koji

In order to build packages against a koper, koji would need to gain the ability to reference external yum repos for (at least) scratch builds.

fedorapeople.org

In order to support kopers, fedorapeople.org would need to be able to:

• Track a separate quota for kopers
• Accept a signal to recreate repodata and repoview content in kopers

Trademark Guidelines

We'd like to enable people to make development like spins of Fedora that include content from kopers. Since all content in kopers has to adhere to Fedora's guidelines it should be safe. We just need terms in the trademark that allow for this kind of spin creation and sharing so that these people don't have to spend time re-branding and can focus more on development.

New Tools

A few new tools might be needed for dealing with kopers.

Tool to build

A script or make target would be needed to initiate a koper build. This is necessary to tell koji what koper to build against, as well as to get the results of the built added to said koper.

Tool to add to koper

A tool would be needed to take the koji scratch build output and add it to ones koper, signaling fedorapeople.org to create repodata and repoview content. It should be able to replace older builds of the same package to save quota space. This tool could also be used to add already built packages to an existing koper without having to go through a koji build.

Version 2.0

Here are some thoughts for future enhancements once we prove that we can walk.

• koper groups. Be able to have a group koper that multiple people work on. This should be made as easy as possible to get the most people as possible working together. Start with the idea of open. Extremely useful for feature driven development.
• koper directory. Have a page off of fedorapeople.org that lists all the (active?) kopers. Easy to walk the filesystem looking for */koper/*. Read from .description files in koper directories for text to display in the directory
• Autogeneration of koper-repo packages for end users to easily install to use said koper. (or maybe register .repo files with pirut to process?)

Comments

If you wish to leave a comment about this concept, you can do here. I'll try to address them as I can.

• MatthiasClasen: The examples make it appear that this is mostly geared towards replacing Fedora packages with own builds. I'd like to see some explanation or guidance on how this will work wrt to versioning (do we require a mandatory suffix for the release, similar to disttags ?). How do we address the concern that bugs reported against such private gtk2 builds will end up in my bugzilla queue ?
  • JesseKeating: The same things we have in place today for these kinds of things... not much (to begin with). However it really boils down to a consumer education kind of thing, educating the consumers of kopers that they're keeping all the pieces if it breaks and whatnot. The packages obviously wouldn't be signed by a Fedora key so that could be one method of discovery.
    • User:till: Which key will be used to sign the packages or is it planned to not sign them at all?
      • User:jkeating: They could be signed with the user's key. Since a user might be able to stuff anything they want into their own repo, I wouldn't feel all that comfortable signing it with a "koji" key. That should be reserved for builds that are produced from repos that koji has ultimate control over.
• RussellHarrison: This would be very nice for new packages or packagers. The review process could be streamlined by running some automatic tests (such as rpmlint) on the packages as soon as they're built. Also the ability to use external yum repos would make it much easier to setup a private koji system for making derivative Fedora derived distributions. Say a company internal release or a new live CD.