From Fedora Project Wiki
Fedora Release Engineering Meeting :: Monday 2007-07-09
Composes in the colo
- downsides are that it will:
1. probably take longer to complete the compose 1. certainly take longer to get the compose into the hands of our testing lab in Raleigh 1. take longer to get onto the machines in Westford
- other benefits most likely outweigh these issues.
- f13 is single point of failure for doing composes--everything is dependent on him
Decision: Releng will continue investigations into composing trees within the colo; potentially composing Test1 there.
Fedora Signing Server
- f13 has not received any feedback since last meeting (some gave it before the meeting)--moving proposal to the ReleaseEngineering/ space as it's new home
Open Discussion
- jwb--have we documented the "you need to email rel-eng to get packages in the buildroot" thing?
- jeremy--sounds like something worth adding to the update doc. feel free to add :)
- notting--we were going to switch that based on bodhi changes with an alert for packages built against unpushed things
- f13--but until we switch it, we need it documented somewhere. Once bodhi can prevent an update from going out that was built against an unreleased update, we can make the buildroot selfupdate.
- jeremy--note that may need to be overridable, otherwise, I have a _great_ way to keep security updates from being able to go out :)
- f13--we do need to be able to override it.
Decision: No decision made
IRC Transcript
 f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Composes in the colo - JesseKeating | <a href="#t12:58" class="time">12:58</a> | |
| * spot warns that anyone caught near my chicago apt will be forced to help pack | <a href="#t12:58" class="time">12:58</a> | |
| f13 | rdieter: jwb: wwoods: ping | <a href="#t12:59" class="time">12:59</a> |
|---|---|---|
| jwb | here | <a href="#t13:00" class="time">13:00</a> |
| rdieter | here (for awhille anyway) | <a href="#t13:00" class="time">13:00</a> |
| f13 | ok, so lets talk a bit about doing composes in the colo. | <a href="#t13:00" class="time">13:00</a> |
| f13 | as opposed to my desk | <a href="#t13:00" class="time">13:00</a> |
| warren | f13, I like relying upon services under our desks. | <a href="#t13:01" class="time">13:01</a> |
| jwb | ew | <a href="#t13:01" class="time">13:01</a> |
| f13 | this has some obvious benifits, others can get to the compose bits a lot quicker (or at all), theoretically somebody other than me can do a compose, etc... | <a href="#t13:01" class="time">13:01</a> |
| f13 | poelcat: ping; | <a href="#t13:01" class="time">13:01</a> |
| f13 | The downsides are that A) it will probably take longer to complete the compose, B) it will certainly take longer to get the compose into the hands of our testing lab in Raleigh, C) it'll take longer to get onto the machines in Westford | <a href="#t13:02" class="time">13:02</a> |
| * wwoods here | <a href="#t13:02" class="time">13:02</a> | |
| jwb | f13, why can't composes be done on your machine and then uploaded somewhere? | <a href="#t13:02" class="time">13:02</a> |
| f13 | but the other benifits most likely outweigh these issues. | <a href="#t13:02" class="time">13:02</a> |
| f13 | jwb: upload slow, and I'm less likely to upload right away and I end up doing lots of testing locally before incurring the cost of an upload | <a href="#t13:03" class="time">13:03</a> |
| f13 | jwb: I may end up doing composes in two locations though, at the same time. | <a href="#t13:03" class="time">13:03</a> |
| wwoods | jwb: it also means that nobody but jesse can do the composes, unless he syncs his config etc. | <a href="#t13:03" class="time">13:03</a> |
| f13 | at least until the trees stop changing as much and then rsync will just work. | <a href="#t13:03" class="time">13:03</a> |
| jwb | you have a script, yes? | <a href="#t13:03" class="time">13:03</a> |
| f13 | not exactly | <a href="#t13:03" class="time">13:03</a> |
| jwb | write a script and have it sync the compose and config at the end | <a href="#t13:04" class="time">13:04</a> |
| jwb | in the background | <a href="#t13:04" class="time">13:04</a> |
| f13 | welll | <a href="#t13:04" class="time">13:04</a> |
| f13 | given that then everybody is completely dependant upon me to produce things I think that's a bad route to go down. | <a href="#t13:05" class="time">13:05</a> |
| jwb | depends i guess | <a href="#t13:05" class="time">13:05</a> |
| jwb | if <upload location> were a username based location, then others could do composes and upload them as well, no? | <a href="#t13:05" class="time">13:05</a> |
| poelcat | f13: here | <a href="#t13:05" class="time">13:05</a> |
| jeremy | jwb: but why should "I'm a composer" be dependent on "I have tons of bandwidth" | <a href="#t13:06" class="time">13:06</a> |
| jeremy | (and tons of upload bandwidth at that) | <a href="#t13:06" class="time">13:06</a> |
| jwb | jeremy, shouldn't. this can all be done _in addition to_ composing in the colo | <a href="#t13:06" class="time">13:06</a> |
| jeremy | jwb: plus, it places a lot of trust in the "how is the machine set up and the composes being done" | <a href="#t13:06" class="time">13:06</a> |
| f13 | The process to compose a tree for releases is as such: tag newly needed build -> sign unsigned packages -> mash a tag -> adjust pungi + mock configs to point to new mash output -> update mock chroot -> run pungi -> make tree accessable. | <a href="#t13:06" class="time">13:06</a> |
| rdieter | f13: you're right, of course, the pros outweigh the cons. How much work/pain is involved to implement this? | <a href="#t13:06" class="time">13:06</a> |
| f13 | rdieter: probably not a whole lot of work. We already have one x86_64 box that we use for various compose like things that can be used to compose i386/x86_64 (once it's given more disk space/ram). | <a href="#t13:07" class="time">13:07</a> |
| jwb | so maybe i'm misunderstanding things | <a href="#t13:07" class="time">13:07</a> |
| f13 | finding a ppc box may be more difficult, and in teh short term we may just have to (ab)use a koji builder to od the compose. | <a href="#t13:07" class="time">13:07</a> |
| jwb | we have dedicated boxen to do composes on? | <a href="#t13:07" class="time">13:07</a> |
| jwb | f13, that's what i dislike | <a href="#t13:08" class="time">13:08</a> |
| f13 | jwb: right now? no. | <a href="#t13:08" class="time">13:08</a> |
| jwb | ppc builds already take too long for some reason | <a href="#t13:08" class="time">13:08</a> |
| f13 | jwb: we have machines in my desk that are used for various development stuff that I use to do the composes. | <a href="#t13:08" class="time">13:08</a> |
| jeremy | longer term, I think we want "compose" type tasks to be done in koji much like runroots in brew | <a href="#t13:08" class="time">13:08</a> |
| jwb | running composes on a koji ppc builder will just make them take longer | <a href="#t13:08" class="time">13:08</a> |
| f13 | jwb: but we have no rackspace for a dedicated box that will just sit there for large amounts of time. | <a href="#t13:09" class="time">13:09</a> |
| jwb | f13, right and you're now getting to my other point | <a href="#t13:09" class="time">13:09</a> |
| f13 | in the $FUTURE when we can ship the unused blade center to phx or elsewhere we can perhaps dedicate one of the blades to this task | <a href="#t13:09" class="time">13:09</a> |
| jwb | f13, secondary arches... | <a href="#t13:09" class="time">13:09</a> |
| f13 | jwb: secondary arches are responsible for doing and hosting their own composes. | <a href="#t13:10" class="time">13:10</a> |
| jwb | right. which is why composing in the colo is bad imho | <a href="#t13:10" class="time">13:10</a> |
| jeremy | jwb: don't think of it as "composing in the colo" | <a href="#t13:10" class="time">13:10</a> |
| jeremy | jwb: think of it as "composing with the same sets of machines used for building" | <a href="#t13:10" class="time">13:10</a> |
| notting | jeremy: if we start doing multilib differently, that becomes a whole lot easier | <a href="#t13:10" class="time">13:10</a> |
| jwb | jeremy, fine. my point is, we need to be careful that doing that doesn't hard code "colo-isms" | <a href="#t13:11" class="time">13:11</a> |
| * f13 is a bit confused why secondary arches matter here. We're talking about primary content for our primary arches. | <a href="#t13:11" class="time">13:11</a> | |
| warren | notting, will we do multilib differently before test1? | <a href="#t13:11" class="time">13:11</a> |
| jwb | f13, because it's important? | <a href="#t13:11" class="time">13:11</a> |
| notting | warren: i suppose it needs to be on the feature list | <a href="#t13:11" class="time">13:11</a> |
| jeremy | jwb: I think that you're getting hung up on the term "colo". the bigger thing here is that doing the tree composes on the box in jesse's cube and the livecds on the box in my cube is wrong, wrong, wrong, wrong, wrong :) | <a href="#t13:12" class="time">13:12</a> |
| f13 | jwb: wait, so you're worried about coloisms leaking into say pungi? | <a href="#t13:12" class="time">13:12</a> |
| jwb | f13, pungi, koji, etc. yes. 2ndary arches have to compose with the exact same tools as primary arches, so i just want to make sure those tools continue to work outside the colo | <a href="#t13:12" class="time">13:12</a> |
| f13 | yes, that is a continual goal, although a bit 'fun' wrt things like mash | <a href="#t13:13" class="time">13:13</a> |
| f13 | but mostly it's about config file changes, not code level changes. | <a href="#t13:13" class="time">13:13</a> |
| jwb | fair enough. not a show stopper, just something i want us to keep in mind | <a href="#t13:13" class="time">13:13</a> |
| jwb | sorry, likely making entirely too much noise about it | <a href="#t13:13" class="time">13:13</a> |
| f13 | jwb: indeed. For the first phase of this, all we're really doing is composing the same way, just on a different machine. | <a href="#t13:13" class="time">13:13</a> |
| notting | mash doesn't work outside of local koji access, for example | <a href="#t13:14" class="time">13:14</a> |
| f13 | instead of a machine on my desk, i'ts a machine on phx | <a href="#t13:14" class="time">13:14</a> |
| jwb | yeah, and that's all fine. though i still don't like the "steal ppc koji builder" part. but it's minor i suppose | <a href="#t13:14" class="time">13:14</a> |
| f13 | notting: would work with /a/ koji though right? Not necessarily /our/ koji? | <a href="#t13:14" class="time">13:14</a> |
| notting | f13: right.. | <a href="#t13:14" class="time">13:14</a> |
| notting | f13: technically, you could mount the koji storage over a wan. be a tad painful. | <a href="#t13:14" class="time">13:14</a> |
| f13 | jwb: yes. And we're mostly talking about the composes for test/final releases, not doing 20 composes a day every day. | <a href="#t13:15" class="time">13:15</a> |
| f13 | jwb: I'll still be doing the random test compose stuff on my workstations | <a href="#t13:15" class="time">13:15</a> |
| f13 | (which also ensures that it still works this way) | <a href="#t13:15" class="time">13:15</a> |
| f13 | what's going to be difficult is defining who can do what parts of composes, and where. | <a href="#t13:16" class="time">13:16</a> |
| f13 | especially since it's a rather disconnected process | <a href="#t13:17" class="time">13:17</a> |
| f13 | each step is done differently potentially in different places with different tools | <a href="#t13:17" class="time">13:17</a> |
| warren | f13, we have the current pain due to a lack of dedicated compose hardware in PHX? | <a href="#t13:18" class="time">13:18</a> |
| warren | Or due to design issues? | <a href="#t13:18" class="time">13:18</a> |
| f13 | which current pain? | <a href="#t13:18" class="time">13:18</a> |
| warren | err | <a href="#t13:19" class="time">13:19</a> |
| rdieter | pain = uneasiness of depending solely on f13 and the machine on his desk? or is there more to it than that? | <a href="#t13:20" class="time">13:20</a> |
| wwoods | the problem is high turnaround time (and reduced reproducability) on composes, because it's being done on a machine under f13's desk | <a href="#t13:20" class="time">13:20</a> |
| wwoods | yes, unease as well, I suppose | <a href="#t13:20" class="time">13:20</a> |
| f13 | warren: we have it for multiple reasons. Some of it is design yes, pungi was designed to be easily used on a user's workstation, and as such its a tad bit harder to work directly into koji runroot like tasks | <a href="#t13:21" class="time">13:21</a> |
| f13 | warren: and secondly that we just don't have dedicated boxes in the colo to do things | <a href="#t13:21" class="time">13:21</a> |
| f13 | but to have dedicated boxes, we need to know what we need so that mmcgrath can allocate it for us | <a href="#t13:22" class="time">13:22</a> |
| f13 | which for right now, we need a place to run sign_unsigned (until we get a signing server), a place to run mash from, a place to run pungi from, and a place to drop the results on the netapp taking advantage of hardlinks. | <a href="#t13:24" class="time">13:24</a> |
| f13 | we currently have app5 (which also runs bodhi) that we can use for signing, mashing, and pungi of i386/x86_64. We just don't have a place to pungi ppc, nor a good method of getting the results up to the netapp | <a href="#t13:24" class="time">13:24</a> |
| wwoods | can we say "this is a good idea" and wait until we have hardware to actually make changes? | <a href="#t13:25" class="time">13:25</a> |
| wwoods | although now that I mention it, "wait until we have hardware to actually make changes" got us in an assload of trouble for F7 | <a href="#t13:25" class="time">13:25</a> |
| f13 | yeah | <a href="#t13:25" class="time">13:25</a> |
| f13 | warren: we can work around the ppc by making temporary use of koji builders at compose time | <a href="#t13:26" class="time">13:26</a> |
| f13 | (which really isn't all that far off from when we make koji do the pungi parts itself) | <a href="#t13:26" class="time">13:26</a> |
| jwb | are we starting to repeat ourselves? | <a href="#t13:27" class="time">13:27</a> |
| f13 | I think so. | <a href="#t13:27" class="time">13:27</a> |
| jeremy | jwb: yes | <a href="#t13:27" class="time">13:27</a> |
| f13 | basically we have an RFR, I'm going to work with mmcgrath to get that fine tuned, and start attempting to do some composes in the colo so we get a better idea of what's needed. | <a href="#t13:27" class="time">13:27</a> |
| f13 | I may end up having to write some of pungi to work better when composing directly to nfs | <a href="#t13:27" class="time">13:27</a> |
| jeremy | f13: we also should make sure that livecd composes are happy with what we do | <a href="#t13:28" class="time">13:28</a> |
| f13 | or to be smarter with gathering packages from a file:/// url and making hardlinks when possible. | <a href="#t13:28" class="time">13:28</a> |
| jeremy | because I'd like to off-load some of the livecd composing as a first step :-) | <a href="#t13:28" class="time">13:28</a> |
| f13 | jeremy: indeed. Does LiveCD composing work in mock yet? | <a href="#t13:28" class="time">13:28</a> |
| * warren sees "composting" whenever he sees "composing" | <a href="#t13:28" class="time">13:28</a> | |
| jeremy | f13: I haven't tried in a while... if I can actually get a livecd to work at all today, then I'll give it a try | <a href="#t13:28" class="time">13:28</a> |
| wwoods | so: 1. composing in colo = good. 2. start doing it now. 3. get hardware ASAP so we can continue doing so without messing up current infrastructure. | <a href="#t13:29" class="time">13:29</a> |
| f13 | k | <a href="#t13:29" class="time">13:29</a> |
| f13 | jeremy: that would be an important first step to being able to do it in the colo | <a href="#t13:29" class="time">13:29</a> |
| wwoods | 4. make sure we keep tools / configs public so it's reproducable elsewhere. and of course: 5. profit | <a href="#t13:29" class="time">13:29</a> |
| f13 | warren: heh sounds good. | <a href="#t13:29" class="time">13:29</a> |
| jeremy | f13: indeed. although that might require some ... interesting questions for mock + selinux | <a href="#t13:29" class="time">13:29</a> |
| f13 | poelcat: Decision: Releng will continue investigations into composing trees within the colo; potentially composing Test1 there. | <a href="#t13:30" class="time">13:30</a> |
| f13 | jeremy: well, don't we already do some interesting things w/ selinux for buildinstall? | <a href="#t13:30" class="time">13:30</a> |
| jeremy | f13: buildinstall is an entirely different beast | <a href="#t13:30" class="time">13:30</a> |
| f13 | yes, but there is some selinux context there | <a href="#t13:31" class="time">13:31</a> |
| jeremy | f13: the images aren't labeled at all, though | <a href="#t13:31" class="time">13:31</a> |
| jeremy | f13: we just generate a policy module in buildinstall | <a href="#t13:31" class="time">13:31</a> |
| f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Fedora Signing Server - JesseKeating | <a href="#t13:31" class="time">13:31</a> | |
| jeremy | f13: which is far far far different from wanting to build an image with contexts | <a href="#t13:31" class="time">13:31</a> |
| f13 | jeremy: nod. | <a href="#t13:31" class="time">13:31</a> |
| f13 | I haven't really gotten any feedback since last meeting (some gave it before the meeting) so I'm going to move this into the ReleaseEngineering/ space as it's new home | <a href="#t13:32" class="time">13:32</a> |
| f13 | I also haven't spent much time looking at code either. | <a href="#t13:32" class="time">13:32</a> |
| f13 | but that's all I ahve to say about this subject. Anybody else ? | <a href="#t13:32" class="time">13:32</a> |
| f13 | Guess not. | <a href="#t13:34" class="time">13:34</a> |
| f13 changed the topic of #fedora-meeting to: RELENG-Meeting - Open Discussion | <a href="#t13:34" class="time">13:34</a> | |
| f13 | anything else anybody would like to discuss? | <a href="#t13:34" class="time">13:34</a> |
| jwb | have we documented the "you need to email rel-eng to get packages in the buildroot" thing? | <a href="#t13:34" class="time">13:34</a> |
| jwb | i went looking the other day and couldn't find it | <a href="#t13:34" class="time">13:34</a> |
| jeremy | jwb: sounds like something worth adding to the update doc. feel free to add :) | <a href="#t13:34" class="time">13:34</a> |
| notting | we were going to switch that based on bodhi changes | <a href="#t13:35" class="time">13:35</a> |
| jwb | notting, which bodhi changes? | <a href="#t13:35" class="time">13:35</a> |
| notting | some sort of alert for packages built against unpushed things | <a href="#t13:35" class="time">13:35</a> |
| f13 | but until we switch it, we need it documented somewhere. | <a href="#t13:35" class="time">13:35</a> |
| f13 | jwb: once bodhi can prevent an update from going out that was built against an unreleased update, we can make the buildroot selfupdate. | <a href="#t13:36" class="time">13:36</a> |
| jeremy | f13: note -- that may need to be overridable | <a href="#t13:36" class="time">13:36</a> |
| jeremy | f13: otherwise, I have a _great_ way to keep security updates from being able to go out :) | <a href="#t13:36" class="time">13:36</a> |
| f13 | jeremy: yes, we do need to be able to override it. | <a href="#t13:37" class="time">13:37</a> |
| jwb | f13, jeremy: ok well until that gets done, then i'll update the update doc | <a href="#t13:37" class="time">13:37</a> |
| f13 | jwb: muchas gracias. | <a href="#t13:37" class="time">13:37</a> |
| f13 | anything else? | <a href="#t13:38" class="time">13:38</a> |
| f13 | alright, calling the meeting. | <a href="#t13:39" class="time">13:39</a> |
| spot | hello? meeting? collect call from f13. | <a href="#t13:39" class="time">13:39</a> |
| f13 changed the topic of #fedora-meeting to: <a href="http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel">http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel</a> -- Meetings often get logged -- see schedule in the wiki for next meeting | <a href="#t13:40" class="time">13:40</a> | |
Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!