From Fedora Project Wiki
(Created page with "Raw notes from composedb brainstorming session (still to be formatted) August 5th, 16:00UTC https://lists.fedoraproject.org/pipermail/rel-eng/2015-August/020562.html We shoul...")
 
No edit summary
Line 1: Line 1:
Raw notes from composedb brainstorming session (still to be formatted)
These are notes from a [https://lists.fedoraproject.org/pipermail/rel-eng/2015-August/020562.html "composedb" brainstorming session] on August 5th, 16:00UTC.
August 5th, 16:00UTC
https://lists.fedoraproject.org/pipermail/rel-eng/2015-August/020562.html


We should post these notes on the wiki afterwards for posterity.
The initial idea for "composedb" was to have something that knows what goes into every compose and what comes out of it: the atomic repos, the live cds, etc.. what's in them.  what's in cloud, server, workstation, etc.  We need such a thing so that we have a place where we can go and ask what changed between this compose and this compose, so we can easily visualize what's different between primary arch composes and secondary arch composes, etc.


The idea is to have something that knows what goes into every compose and what comes out of it.
Furthermore, it would be a more robust solution than the [https://apps.fedoraproject.org/releng-dash releng dash] to be able to show (for instance) when was the last nightly compose that worked and to give you a way to visualize when the last updates push was done.
the atomic repos, the live cds, etc.. what's in them.  what's in cloud, server, workstation, etc.
so that we have a place where we can go and say what changed between this compose and this compose.
so we can easily visualize what's different between primary this and s390 that.


Once upon a time we were thinking about keeping track of what's in development what's EOL, when was the last nightly compose that worked.
'''Action''' We're going to look at PDC (Production Definition Center) for this.  It's a Django app that does almost all of that described above.


Give you a way to visualize when the last updates push was done.
It may not currently be able to support the notion of "rings" (a la Fedora.NEXT).  We need a way to say what's in the different rings (so they can have different policies and processes)
And there's lots of things that can be built using this information that we can't do today.


Should really look at PDC since sharing tools is great.
----


it would be cool if when we're doing the rawhide compose,
Beyond having a system that ''knows'' what inputs go into which releng artifacts, it would be great to then develop tooling around that data source. For instance:
we can look at see that nothing has changed in XFCE
  so we don't rebuild that, but we do rebuild other things where things actually changed.


with that we can do things as they're needed, instead of once a night or only when release time happens.
* it would be cool if when we're doing the rawhide compose we can look at see that nothing has changed in XFCE so we don't rebuild that livecd, but we do rebuild other artifacts where things actually changed.
* furthermore, with that kind of knowledge we can rebuild artifacts as their inputs change (fedmsg) instead of doing things on a nightly or semi-annual basis like we do now.
* it would be cool to produce reports on the different editions and their artifacts over time.  i.e., show how the size of the workstation image is growing (so we can fix it) or show how the size of the cloud image is shrinking (so we can celebrate).
* it would be cool to automatically impose gating via taskotron for some artifacts, depending on what "rings" (Fedora.NEXT) the inputs are in and what policies we have associated with those rings.
* leverage taskotron QA checks to create side-tags where we automatically rebuild stuff in the event of soname bumps.  We could then also auto-gate artifacts and keep them from reaching the next step in the process if (for instance) things fail depcheck.  Say, stuff in ring 0 and ring 1 require tests X, Y, and Z, but ring 2 requires less.  we could make sure that "rawhide is never broken".
* it could be auspicious to build artifacts immediately (as their inputs change) but to gate publication to the mirrors on some sort of human sign-off from releng.


for CI for rpm dependencies, what about koschei?
If PDC is the system that '''knows''' what goes into what, then this second section is about a system (or group of systems) that knows '''how''' to build those things -- and does it.  We're calling that system [https://twitter.com/TheMaxamillion/status/608040785829871616 Outhouse].  Think of it as a rewrite of the collection of shell scripts in the releng repo into a continuously-running daemon.
  well, it's a CI at the RPM level, but not one at the compose level.


it would be cool to produce reporting on the different editions over time:
However this ends up shaping up, a note about builders:
  - show how the rpm size of workstation is growing (so we can fix it)
  - show how the rpm size of the cloud image is shrinking (so we can cheer it on)


supporting rings stuff properly
* We can't use jenkins for building - its unsupported.
  we need a way to say what's in the different rings (so they can have different policies and processes)
* We can't use taskotron for building - nodes in the QA network are of an insufficient security grade.
  (anyway, there's lots of things that can come from having this information that we can't do today)
* We could roll our own thing.
 
* Or we could kick off ''everything'' as koji tasksThis is preferable and we can lay the groundwork for it now by porting our existing compose processes to koji tasks.
let's open source PDC -- they want to do it and we want to do it.
 
beyond that, let's think about a system that runs continuously to rebuild things as needed.
internally, it seems unlikely that anything like this exists already.  they have a mostly manual process now involving sign-off, etc.
 
maybe - build things to as-complete as they can be, but require human signoff to make things public
 
leverage taskotron to create side-tags to rebuild stuff (if soname bumps) also to auto-gate things and keep them from reaching the next step in the process. say, stuff in ring 0 and ring 1 require tests X, Y, and Z, but ring 2 requires less. we could make sure that "rawhide is never broken".
 
publish fedmsg messages about failures, etc..
 
have all actual build processes running in koji (other options are less secure or less supported, jenkins, taskotron?, tunir?)
 
outhouse could be a place where the policy glue (gating) comes into play by figuring out what goes into artifacts and what "ring" things are inthen we could block things appropriately if such and such input doesn't pass depcheck (for instance).
 
https://twitter.com/TheMaxamillion/status/608040785829871616
 
 
 
 
Requirements
------------
 
- Solve all the problems
-
 
Design/implementation notes
---------------------------
 
- Written in python
-

Revision as of 19:01, 5 August 2015

These are notes from a "composedb" brainstorming session on August 5th, 16:00UTC.

The initial idea for "composedb" was to have something that knows what goes into every compose and what comes out of it: the atomic repos, the live cds, etc.. what's in them. what's in cloud, server, workstation, etc. We need such a thing so that we have a place where we can go and ask what changed between this compose and this compose, so we can easily visualize what's different between primary arch composes and secondary arch composes, etc.

Furthermore, it would be a more robust solution than the releng dash to be able to show (for instance) when was the last nightly compose that worked and to give you a way to visualize when the last updates push was done.

Action We're going to look at PDC (Production Definition Center) for this. It's a Django app that does almost all of that described above.

It may not currently be able to support the notion of "rings" (a la Fedora.NEXT). We need a way to say what's in the different rings (so they can have different policies and processes) And there's lots of things that can be built using this information that we can't do today.


Beyond having a system that knows what inputs go into which releng artifacts, it would be great to then develop tooling around that data source. For instance:

  • it would be cool if when we're doing the rawhide compose we can look at see that nothing has changed in XFCE so we don't rebuild that livecd, but we do rebuild other artifacts where things actually changed.
  • furthermore, with that kind of knowledge we can rebuild artifacts as their inputs change (fedmsg) instead of doing things on a nightly or semi-annual basis like we do now.
  • it would be cool to produce reports on the different editions and their artifacts over time. i.e., show how the size of the workstation image is growing (so we can fix it) or show how the size of the cloud image is shrinking (so we can celebrate).
  • it would be cool to automatically impose gating via taskotron for some artifacts, depending on what "rings" (Fedora.NEXT) the inputs are in and what policies we have associated with those rings.
  • leverage taskotron QA checks to create side-tags where we automatically rebuild stuff in the event of soname bumps. We could then also auto-gate artifacts and keep them from reaching the next step in the process if (for instance) things fail depcheck. Say, stuff in ring 0 and ring 1 require tests X, Y, and Z, but ring 2 requires less. we could make sure that "rawhide is never broken".
  • it could be auspicious to build artifacts immediately (as their inputs change) but to gate publication to the mirrors on some sort of human sign-off from releng.

If PDC is the system that knows what goes into what, then this second section is about a system (or group of systems) that knows how to build those things -- and does it. We're calling that system Outhouse. Think of it as a rewrite of the collection of shell scripts in the releng repo into a continuously-running daemon.

However this ends up shaping up, a note about builders:

  • We can't use jenkins for building - its unsupported.
  • We can't use taskotron for building - nodes in the QA network are of an insufficient security grade.
  • We could roll our own thing.
  • Or we could kick off everything as koji tasks. This is preferable and we can lay the groundwork for it now by porting our existing compose processes to koji tasks.