Category:Copr: Difference between revisions

Copr (Cool Other Package Repo) is a Fedora project to help make building and managing third party package repositories easy. The instance to be installed within Fedora Infrastructure provides Fedora maintainers with the ability to create repos of packages to build against and share with others.

Some Design Requirements

• Copr need their own builders to protect against badness when installing pkgs for builddeps into buildroots
• the system hosting this becomes more critical to infrastructure so it's probably best not to reuse fedorapeople for the hosting

Design Overview

• We're creating a new build system that's more lightweight than koji to host this. the build system will also be named Copr. It will have a small web UI and a command line client. We'll create this using TG2 as it's something we're familiar with.
• Copr will send tasks to a generic task scheduler that will drive the actual builds.
• For now, the task scheduler will communicate with the builders using func. (toshio working on this)
• The builders need to have a vm based mock to do the actual builds to protect against malicious packages compromising the builder for future builds (skvidal working on this)
• People consuming from the Copr repos need to have a WebUI for this (frankieonuonga working on this)
• Yum plugin to make configuring copr repos (with inter-repo deps) work
• Hardware to test this on (mmcgrath has provided us with test hardware) and hardware to run it in production (spot when we're ready)
  • host for builders
  • host for task scheduler
  • host for web ui (app servers?)
  • shared storage for repositories, uploaded srpms, and other things all builders need access to
  • host for exposing the repositories to people downloading packages from copr (app servers?)

Packager workflow

• tool to submit an srpm to the above (using their fedora certs/account) and specify any targets/repos it should use
• user is prompted to sign the built packages (or not)
• puts them into the repo they are being built for

Scripts to start on

• sync-to-repo: takes a dir of localpkgs and remote read and write path
  • syncs down the repodata from the remote read path and merges the localpkgs into that path and pushes the data back up via the write path.

http://skvidal.fedorapeople.org/misc/sync_to_repo.py

Design work

Usage Cases

• Bob would like to try out a new version of GTK2. Bob does a copr build of the new GTK2 package, and then builds a number of other packages against this. Bob shares his gtk2newness copr with others for validation of the new GTK2

• Jill would like to experiment with rpm using a different compression method for packages. Jill builds a modified version of rpm for her rpm copr and builds a number of other packages with this rpm. Jill shares her rpm copr for further testing of the rpm change.

• Dave would like to experiment with kernel patches and have the repo creation handled for him. Dave continues to add patches to kernel source rpms and builds them for his copr(s). Dave shares his various kernel coprs with the community for early testing and validation. Dave spends more time working on kernel and less time dealing with repo management.

• Fedora Infrastructure would like to create a repo with updates to RHEL packages that it needs in order to manage its systems. It would like other people to be able to use the repos so that they can replicate their environment.

• The KDE SIG wants to have a repository where they can perform major upgrades that their users want but don't follow the Fedora update guidelines. By publishing to a copr repo the packages don't have to go into the main Fedora repositories but they are able to point to it for people who want to run later versions.

• The KDE SIG wants to test their updates for a Fedora release as a set so they can find regressions between the components and the release. They publish the packages to a copr repo and have their testers pull updates from there.

KDE RedHat

kde-redhat is a large external repo, asked rdieter how he manages it so far.

All I'm doing really is building stuff, either in koji or my own mock instance. Grabbing those rpms, signing them, stuffing it into repos. Whatever kopers end up being, can't be much worse than what I'm doing now. honestly.

• repos managable by > 1 person (ie, a sig)
• custom buildroots (ie, to use itself or other kopers). we've had repo heirarchies do something like that already kde-unstable depends on kde-testing depends on kde-stable
• easy mirroring, I suppose, would be nice too
• not sure what kind of workflow to get packages into kopers would look like. something bodhi like, or maybe doesn't even have to be that complicated
• maybe we could get some inspiration from how the opensuse buildservice works though I've only spent a couple of hours looking at it.

imo kde redhat is beyond the scope of Copr - it's WAY beyond 1GB per user - skv

Resources

A Fedora maintainer would be alloted up to 1GiB of storage for package repos. Packages can be built for any arch Fedora builds for (but doesn't have to be all of them). Copr would have repoview content for easy browsing and subscribing to changes. Copr's could be increased by request.

Work Flow

1. Bob requests a Copr repo via the Copr WebUI -- defines what other repos are dependencies
   • The Copr service creates an empty repo and generates an rpm that has yum repository definitions for the repo and dependencies on other repo-configuring rpms
2. Bob constructs an srpm of gtk2
   • Not a Copr issue
3. Bob issues a command (cu build "gtk2newness" gtk2.src.rpm ?) to build the gtk2 package.
   • first this checks the pkg and looks at the License tag to make sure it's not something we absolutely cannot build.
   • this just delivers the srpm to the shared filesystem and then kicks off a build task for it in headhunter.
   • Notes about building from SCM rather than SRPM; SCM server needs to be accessible from builders (currently, within Fedora's network)
   • Building from SCM is not necessarily as important as being able to share changes via the SCM. If I can clone the SCM repo and you can as well to make changes but we create an SRPM as an intermediate step for build that's not much of a burden.
4. Copr creates a buildroot based upon the rawhide collection and any packages in Bob's "gtknewness" copr.
   • two issues here: getting koji to use arbitrary repos
   • not having the above beat the crap out of fedorapeople with koji constantly hitting it for pkgs/repodata
   • These are some of the reasons we need a different build system and different storage.
5. Copr/headhunter builds Bob's gtk2
6. headhunter takes the koji built package and adds it to Bob's "gtknewness" copr, recreating yum repodata and repoview content.
   • this means having a task to build repodata
   • need something in here for SIGNING those pkgs, too.
     • Probably want a separate key for each Copr repo.
     • why a separate key? if I'm building these for ME and mine there is no reason why I shouldn't use the same key
     • Repos can have multiple people adding to them so there needs to be one key for it.
     • you could use the same key for all repositories that only you contribute to but then we'd need to keep track of two different types of repository
7. Bob proceeds to create srpms for various other packages and builds them in koji against his "gtknewness" koper.
   • having koji know the repo exists or to have it know when the repo no longer exists.
8. Bob shares his "gtknewness" koper with the world
   • this is just advertising a url
9. Bob creates a Desktop Live image based on rawhide and his "gtknewness" koper and shares that with the world too.
10. Bob fixes bugs with his gtk2 and continues to do builds until everybody is happy
11. Bob commits his gtk2 changes to package SCM and does a normal koji build.
12. Bob profits.
    • These last 4 items don't matter at all they are outside of scope, imo.

Content

Content within one's Copr(s) must be content acceptable for distribution within Fedora. (The packaging doesn't need to be, just the content).

• Manual testing, reaction is one possibility
• Review process for not-yet-in-Fedora-items (just license review probably all that's needed)

Trademark Guidelines

We'd like to enable people to make development like spins of Fedora that include content from Coprs. Since all content in Coprs has to adhere to Fedora's guidelines it should be safe (?What do we mean by safe and what guidelines make it so? TK). We just need terms in the trademark that allow for this kind of spin creation and sharing so that these people don't have to spend time re-branding and can focus more on development.

New Tools

A few new tools might be needed for dealing with Coprs.

Tool to build

A script would be needed to initiate a Copr build. This will submit the build to the Copr buildsystem.

Tool to add package to a copr

The build system needs to be able to add the package to the copr on the shared filesystem and create new repodata for it. We want to be able to remove older versions of packages. But we also want to have former packages so that people can downgrade around problems if they have to.

This tool could also be used to add already built packages to an existing koper without having to go through a koji build (?Do we want this? TK).

Version 2.0

Here are some thoughts for future enhancements once we prove that we can walk. Some of these may go into the first version since we have to write a new buildsystem to deal with this.

• Copr groups. Be able to have a group copr that multiple people work on. This should be made as easy as possible to get the most people as possible working together. Start with the idea of open. Extremely useful for feature driven development.
• Copr directory. Have a page off of the copr web ui that lists all the (active?) coprs. Easy to walk the filesystem looking for */copr/*. Read from .description files in copr directories for text to display in the directory
• Autogeneration of copr-repo packages for end users to easily install to use said copr. (or maybe register .repo files with pirut to process?)

Notes from former comments

• How do we manage versioning?
  • We don't. We can mandate a repotag suffix so we have something like fc13.copr but the packages in coprs can replace packages in Fedora so they can have versions that are more recent than Fedora. The repotag is there to help differentiate between copr builds and Fedora builds.
  • As a side note, Coprs can very easily lead to dependency hell. PPAs have done that to Ubuntu users. Messaging about what Coprs is for and how it can lead to dependency problems is very important otherwise we'll have end users complaining about dep hell.

• How do we address the concern that bugzilla bugs will be reported against copr builds rather than Fedora builds?
  • Nothing we can do about it really except ask people for their rpm -q output. We can try to message that Coprs will break your system but that's about it.

• What keys will we sign with?
  • Something that isn't used for any Fedora release. However, it needs to be something that the new build system can sign with if we're going to enable group repositories.

• This would be very nice for new packages or packagers. The review process could be streamlined by running some automatic tests (such as rpmlint) on the packages as soon as they're built. Also the ability to use external yum repos would make it much easier to setup a private koji system for making derivative Fedora derived distributions. Say a company internal release or a new live CD.
  • It may be possible to add an rpmlint check for packages in the build system but it wouldn't be a first implementation.