From Fedora Project Wiki
Fedora Release Engineering Meeting :: Monday 2008-04-14
Preview Release (aka PR) Status
- Not ready--scheduled for availability on 2008-04-11
- Problems with yum and other compose tools
- Snafu with package signing which caused rawhide to revert back to all unsigned copies of most packages--100G worth of churn
- Spend rest of today untangling rawhide situation
- Attempt Preview Release on 2008-04-15 with no change to GA date of 2008-04-29
Mass Branching
- Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.
- Should probably have a CVS outage to get the branching done--usually takes a full work day
- Discusison left unfinished
What is the Preview Release?
- Just another snapshot or equivalent to Alpha and Beta which requires more formal testing before release?
- Discussion about confusion over what preview release is
- Need clearer definition for what Preview Release and Release Candidate mean
- DECISIONS:
1. Poelcat to take a shot at documenting purpose and criteria for snapshots and test releases 1. Preview Release will be as installable and functional as other test releases (Alpha and Beta)
IRC Transcript
| f13 | ping: notting jeremy spot lmacken warren rdieter wwoods poelcat jwb | <a href="#t13:00" class="time">13:00</a> |
|---|---|---|
| * notting is here | <a href="#t13:00" class="time">13:00</a> | |
| rdieter | yo | <a href="#t13:00" class="time">13:00</a> |
| notting | although still drowning in mail | <a href="#t13:00" class="time">13:00</a> |
| skvidal | notting: I've got a filter that helps with that - search for @ in the sender's address and delete the mail if it is found | <a href="#t13:01" class="time">13:01</a> |
| wwoods | ooh, was about to reboot for preupgrade testing | <a href="#t13:03" class="time">13:03</a> |
| * wwoods stays his hand for a bit | <a href="#t13:03" class="time">13:03</a> | |
| f13 | alright, lets kick this pig. | <a href="#t13:05" class="time">13:05</a> |
| -!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Preview Release | <a href="#t13:05" class="time">13:05</a> | |
| f13 | I"d really like to get PR out the door, but the universe hates me. | <a href="#t13:05" class="time">13:05</a> |
| f13 | I'm convinced. | <a href="#t13:05" class="time">13:05</a> |
| f13 | After all the battling with yum and other compose tool fun, I was all set to get PR composed and out the door today. | <a href="#t13:05" class="time">13:05</a> |
| * jeremy is sort of here | <a href="#t13:05" class="time">13:05</a> | |
| f13 | Unfortunately, that's not to be. | <a href="#t13:05" class="time">13:05</a> |
| f13 | Due to a snafu with package signing, we caused rawhide to revert back to all unsigned copies of most packages, some 100G worth of churn | <a href="#t13:06" class="time">13:06</a> |
| f13 | probably not the best to compose PR against. | <a href="#t13:06" class="time">13:06</a> |
| wwoods | ack! | <a href="#t13:06" class="time">13:06</a> |
| * poelcat here | <a href="#t13:06" class="time">13:06</a> | |
| jeremy | f13: doh. I was wondering why I saw so much churn in my sync but thought it was going to signed not the unsigned packages :( | <a href="#t13:06" class="time">13:06</a> |
| f13 | I'm in the middle of trying to recover the situation, and we've turned off snapmirror tos that PHX/TPA don't get more of the bad bits | <a href="#t13:06" class="time">13:06</a> |
| f13 | jeremy: sadly, mash has a "if I don't find the preferred signed copy, I fall back to unsigned" method that I wasn't fully aware of. | <a href="#t13:07" class="time">13:07</a> |
| * rdieter was wondering why his rawhide mirror sync was taking so long to finish today, rats. | <a href="#t13:07" class="time">13:07</a> | |
| notting | f13: well, the alternative is to not fall back and fail | <a href="#t13:07" class="time">13:07</a> |
| f13 | so I'll likely spend the entire day cleaning up from that, and investigating why sign_unsigned suddenly is getting md5 mismatches on the things it signs. | <a href="#t13:08" class="time">13:08</a> |
| notting | which also won't help you | <a href="#t13:08" class="time">13:08</a> |
| f13 | notting: or stop switching gpg sigs needlessly | <a href="#t13:08" class="time">13:08</a> |
| f13 | at any rate, I may end up composing a PR against a recovered snapshot from /yesterday's/ rawhide | <a href="#t13:09" class="time">13:09</a> |
| f13 | if it looks like recovering todays rawhide bits is going to take too long | <a href="#t13:09" class="time">13:09</a> |
| f13 | but I haven't looked at what all got signed last night that would be good to have in the PR | <a href="#t13:10" class="time">13:10</a> |
| f13 | Looking beyond PR... | <a href="#t13:11" class="time">13:11</a> |
| f13 | I think we'll still be taking in tag requests until early next week | <a href="#t13:11" class="time">13:11</a> |
| f13 | then we need to crack down and start considering release candidates, and what are actual release blockers | <a href="#t13:11" class="time">13:11</a> |
| f13 | anything more on Preview Release? | <a href="#t13:13" class="time">13:13</a> |
| notting | do we know of any PR blockers aside from the 'getting it composed issues'? | <a href="#t13:14" class="time">13:14</a> |
| f13 | jeremy said there was one bug left on the PR blocker list | <a href="#t13:14" class="time">13:14</a> |
| jeremy | the x keyboard config one | <a href="#t13:14" class="time">13:14</a> |
| f13 | but that's the only thing I'm aware of, I thought we were in pretty damn good shape for PR | <a href="#t13:15" class="time">13:15</a> |
| jeremy | and I really really really don't know if I feel comfortable with the RC being the first thing that has a fix for that | <a href="#t13:15" class="time">13:15</a> |
| jeremy | I know ajax was working on it on friday, but I didn't quite catch what he was saying the problem was on the bus (since I was in the python unicode discussion instead ;-) | <a href="#t13:15" class="time">13:15</a> |
| jeremy | and I've been in class all morning | <a href="#t13:15" class="time">13:15</a> |
| notting | what's wrong with keyboards today? | <a href="#t13:16" class="time">13:16</a> |
| f13 | if you set a layout at install, it doesn't follow through to X once installed I think | <a href="#t13:17" class="time">13:17</a> |
| notting | oh, because we're only writing to sysconfig, not to X | <a href="#t13:19" class="time">13:19</a> |
| notting | unless you use s-c-keyboard, which writes to both | <a href="#t13:19" class="time">13:19</a> |
| f13 | I think so? I'm not clear on the problem space | <a href="#t13:19" class="time">13:19</a> |
| f13 | so for the keyboard thing, we may be putning the PR to tomorrow anyway | <a href="#t13:21" class="time">13:21</a> |
| -!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Mass Branching | <a href="#t13:22" class="time">13:22</a> | |
| rdieter | punt++ , will the package signing be resolved by then? | <a href="#t13:22" class="time">13:22</a> |
| wwoods | since we're now a week from RC I don't think we should slip the PR any further for fixes | <a href="#t13:22" class="time">13:22</a> |
| f13 | Toshio says that the tools (pkgdb, etc..) are ready for the mass branching. | <a href="#t13:23" class="time">13:23</a> |
| f13 | unfortunately unless we want to mailbomb everybody, we should probably have a CVS outage to get the branching done, and that usually takes a full work day | <a href="#t13:23" class="time">13:23</a> |
| f13 | and I have 0 time to focus on it right now | <a href="#t13:23" class="time">13:23</a> |
| f13 | hrm, | <a href="#t13:23" class="time">13:23</a> |
| f13 | lets put CVS branchingon hold for a moment. | <a href="#t13:23" class="time">13:23</a> |
| f13 | wwoods: You'd be OK with PR going out with the current blocker unfixed? | <a href="#t13:24" class="time">13:24</a> |
| f13 | (if so, mind moving that blocker off of the PR blocker list) | <a href="#t13:24" class="time">13:24</a> |
| wwoods | lemme recheck the blocker list, but seriously if we're releasing another snapshot in a week *and* we have a fix approaching the proverbial runway | <a href="#t13:25" class="time">13:25</a> |
| wwoods | then, yeah, get the dang thing into people's hands and release note it | <a href="#t13:25" class="time">13:25</a> |
| wwoods | "we know, it'll be fixed next week, just ssshh" | <a href="#t13:25" class="time">13:25</a> |
| f13 | wwoods: I don't think there is really time in the schedule to "release" RCs | <a href="#t13:25" class="time">13:25</a> |
| f13 | wwoods: so we'd be relying on rawhide users to confirm if it's fixed or not | <a href="#t13:26" class="time">13:26</a> |
| wwoods | wait, what? I must have misunderstood something | <a href="#t13:26" class="time">13:26</a> |
| wwoods | 22 April 2008 | <a href="#t13:26" class="time">13:26</a> |
| wwoods | Release Candidate 1 | <a href="#t13:26" class="time">13:26</a> |
| f13 | RCs are seriously going to be compose as fast as you can, multiple times a day, and drop it on the mirrors ASAP | <a href="#t13:26" class="time">13:26</a> |
| wwoods | There's no schedule for *subsequent* RCs - those are at our option | <a href="#t13:26" class="time">13:26</a> |
| wwoods | hm, okay | <a href="#t13:27" class="time">13:27</a> |
| wwoods | so we're not planning on putting RC1 on the mirrors? | <a href="#t13:27" class="time">13:27</a> |
| f13 | I have to have something going to the mirrors by the 24th really for a successful 29th release | <a href="#t13:27" class="time">13:27</a> |
| f13 | wwoods: there is not enough time. | <a href="#t13:27" class="time">13:27</a> |
| wwoods | forgive me if I'm getting confused, I think I'm coming down with something | <a href="#t13:27" class="time">13:27</a> |
| f13 | mirrors == 3~ days of sync lead time | <a href="#t13:27" class="time">13:27</a> |
| f13 | often 3 business days | <a href="#t13:28" class="time">13:28</a> |
| f13 | that's for a "minor" release. For a major release, we want a lot more time to have a lot more mirrors ready for the bit flip | <a href="#t13:28" class="time">13:28</a> |
| notting | well, i think the issues is just that the schedule says 'Preview Release' and 'Release Candidate 1' | <a href="#t13:28" class="time">13:28</a> |
| notting | are those defined? | <a href="#t13:28" class="time">13:28</a> |
| f13 | notting: not well. PR is new for this round, and "RC" is just something we used to call attempts to do the final compose | <a href="#t13:28" class="time">13:28</a> |
| f13 | I don't think there was any promise that "RC" would see the light of day in the public space | <a href="#t13:29" class="time">13:29</a> |
| notting | no, but we should probably add something somewhere that defines what they are, or we'll be answering this question again | <a href="#t13:29" class="time">13:29</a> |
| wwoods | Yeah, my memory's hazy but I think f13's right and I've just confused myself | <a href="#t13:30" class="time">13:30</a> |
| wwoods | so yeah, that needs a more detailed explanation somewhere, I guess | <a href="#t13:30" class="time">13:30</a> |
| wwoods | and if we're not planning on doing another fullblown worldwide mirrored release until final | <a href="#t13:30" class="time">13:30</a> |
| wwoods | then yes, I guess we'll need to continue to block on bug 438246 | <a href="#t13:31" class="time">13:31</a> |
| buggbot | Bug <a href="https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=438246">https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=438246</a> low, low, ---, Kristian Høgsberg, ASSIGNED , Keyboard layout selected during install is forgotten at post-install boot | <a href="#t13:31" class="time">13:31</a> |
| f13 | I know we've already got a potential schedule for F10, but yeah, I think we seriously need to discuss what we're trying to accomplish with PR, and RC, etc... | <a href="#t13:32" class="time">13:32</a> |
| f13 | I originally thought that PR was just going to be the regularly scheduled snapshot of the week attempt | <a href="#t13:32" class="time">13:32</a> |
| f13 | just under a new name | <a href="#t13:32" class="time">13:32</a> |
| f13 | not that we'd be holding it up for bugs, or treating it all that special, or mirroring it, just bog standard snapshot on torrent | <a href="#t13:33" class="time">13:33</a> |
| f13 | and that we'd continue to use rawhide for testing things we want to get fixed before we say "Ok, we're good enough, start attempting the release spin" and if the spin is good, as in nothing fell over during compose, that was it, we're shipped | <a href="#t13:33" class="time">13:33</a> |
| f13 | trying to do public releases of candidate trees is going to put a serious hurt on things | <a href="#t13:34" class="time">13:34</a> |
| f13 | anyway.... back to /topic | <a href="#t13:35" class="time">13:35</a> |
| wwoods | No, we wanted a third fullblown test release before final | <a href="#t13:35" class="time">13:35</a> |
| f13 | if anybody wants to tackle cvs branching, by all means, I'd be very happy | <a href="#t13:35" class="time">13:35</a> |
| f13 | I have some notes from last time, and toshio started an SOP page to outline calls to pkgdb and such | <a href="#t13:36" class="time">13:36</a> |
| * poelcat wonders how much we reduce our potential pool of testers when we reduce availability to rawhide | <a href="#t13:36" class="time">13:36</a> | |
| notting | wwoods: f13: so, how do we resolve this dichotomy | <a href="#t13:36" class="time">13:36</a> |
| wwoods | but RCs don't need to be fullblown mirrored releases | <a href="#t13:36" class="time">13:36</a> |
| f13 | by the time we get to RC state, it should seriously be an RC, as in all the bugs are fixed, we're just putting the tested builds into a tree, and making sure the tree compose itself doesn't fall over | <a href="#t13:37" class="time">13:37</a> |
| wwoods | right | <a href="#t13:38" class="time">13:38</a> |
| f13 | I'm not sure what the value is of trying to put that tree out to the public and waiting for somebody to give it thumbs up | <a href="#t13:38" class="time">13:38</a> |
| wwoods | that's what the PR is for | <a href="#t13:38" class="time">13:38</a> |
| poelcat | wwoods: +1 | <a href="#t13:39" class="time">13:39</a> |
| f13 | wwoods: so what do we do with the time between cuttin the PR, having it mirrored, getting people to download and test, and report things, and cutting RC? | <a href="#t13:39" class="time">13:39</a> |
| f13 | do we take in other bugfixes people find? | <a href="#t13:39" class="time">13:39</a> |
| wwoods | yes? | <a href="#t13:39" class="time">13:39</a> |
| f13 | do we just say "nope, sorry, th'at snot a blocker, fix it in an update?" | <a href="#t13:39" class="time">13:39</a> |
| wwoods | the reason it's called "Preview Release" and not "Public Release Candidate" is that we *knew* there'd be unfixed blockers at this point | <a href="#t13:40" class="time">13:40</a> |
| f13 | and if we're going to take fixes in, what's the criteria for what we wait for to be fixed for PR, vs what we'll allow to be fixed after PR goes out? | <a href="#t13:40" class="time">13:40</a> |
| wwoods | so calling it an RC would have just been an obvious lie | <a href="#t13:40" class="time">13:40</a> |
| wwoods | the criteria for PR are the same as for Alpha and Beta | <a href="#t13:40" class="time">13:40</a> |
| f13 | uh.. | <a href="#t13:40" class="time">13:40</a> |
| wwoods | seriously, (alpha, beta, preview_release) = (test1, test2, test3) | <a href="#t13:41" class="time">13:41</a> |
| f13 | if that were the case, we wouldn't be blocking on the X thing | <a href="#t13:41" class="time">13:41</a> |
| wwoods | we changed the names and shifted dates but that was the original plan | <a href="#t13:41" class="time">13:41</a> |
| wwoods | is that right? yeah I guess so | <a href="#t13:41" class="time">13:41</a> |
| f13 | wwoods: I think that really fell apart when we scheduled the PR for 3 days after the final freeze | <a href="#t13:42" class="time">13:42</a> |
| wwoods | I'm not sure what criteria made the X thing a blocker in the first place, actually | <a href="#t13:42" class="time">13:42</a> |
| f13 | and didn't give it any compose/stage/sync time | <a href="#t13:42" class="time">13:42</a> |
| f13 | I was under the (mistaken?) impression that PR was just going to be a weekly snapshot, tossed onto torrent | <a href="#t13:43" class="time">13:43</a> |
| wwoods | no, PR was supposed to map to test3 | <a href="#t13:43" class="time">13:43</a> |
| f13 | and would service as a post-freeze media set to be the basis of further testing | <a href="#t13:43" class="time">13:43</a> |
| wwoods | *that* part is true | <a href="#t13:43" class="time">13:43</a> |
| f13 | ok, our signals were quite crossed then | <a href="#t13:43" class="time">13:43</a> |
| f13 | the schedule certainly doesn't have room for a full blown test3 like release | <a href="#t13:44" class="time">13:44</a> |
| wwoods | okay then | <a href="#t13:44" class="time">13:44</a> |
| f13 | I thought we obviated the need for a 3rd major developmental milestone | <a href="#t13:44" class="time">13:44</a> |
| wwoods | let's go with your plan | <a href="#t13:44" class="time">13:44</a> |
| wwoods | because it reduces our workload | <a href="#t13:44" class="time">13:44</a> |
| f13 | 'cause it just make things take too long | <a href="#t13:44" class="time">13:44</a> |
| wwoods | and if anyone actually flips out | <a href="#t13:44" class="time">13:44</a> |
| f13 | we'll have me to blame (: | <a href="#t13:44" class="time">13:44</a> |
| wwoods | or if it actually has an adverse effect | <a href="#t13:44" class="time">13:44</a> |
| notting | so, 1) do we need to document this 2) do we need to adjust the f10 schedule to have PR be a 'release'? | <a href="#t13:45" class="time">13:45</a> |
| wwoods | then we'll rejigger the F10 schedule around the assumption that PR == test3 | <a href="#t13:45" class="time">13:45</a> |
| wwoods | 1) yes, 2) no, unless this goes horribly awry | <a href="#t13:45" class="time">13:45</a> |
| wwoods | since we're doing (mostly) weekly snapshot releases, we're putting a lot more testable bits in people's hands | <a href="#t13:46" class="time">13:46</a> |
| wwoods | than just test1/test2/test3 | <a href="#t13:46" class="time">13:46</a> |
| wwoods | Alpha/Beta/snapshot,snapshot,snapshot,snapshot...pr,rc,rc,rc,rc.. | <a href="#t13:46" class="time">13:46</a> |
| wwoods | obviously that's More, Fresher Bits | <a href="#t13:47" class="time">13:47</a> |
| wwoods | practially farm-fresh | <a href="#t13:47" class="time">13:47</a> |
| poelcat | wwoods: except that the snaps have to install... they haven't for me | <a href="#t13:47" class="time">13:47</a> |
| wwoods | that has nothing to do with the schedule. | <a href="#t13:47" class="time">13:47</a> |
| f13 | and yet they have for a very large set of users | <a href="#t13:47" class="time">13:47</a> |
| wwoods | that's a development problem particular to this release and your setup | <a href="#t13:47" class="time">13:47</a> |
| wwoods | so: sorry | <a href="#t13:47" class="time">13:47</a> |
| poelcat | wwoods: your point about people having more bits to test | <a href="#t13:47" class="time">13:47</a> |
| f13 | please, lets not get trapped in the "it doesn't work for me, it can't possibly work for anybody else" syndrome | <a href="#t13:47" class="time">13:47</a> |
| wwoods | I think I just showed that there's more, fresher stuff available for test | <a href="#t13:48" class="time">13:48</a> |
| notting | ok, so who is going to document/announce the PR/RC cycle? | <a href="#t13:48" class="time">13:48</a> |
| notting | and we can table the 'how do we do F10' until we're less in AAAUGGH mode | <a href="#t13:48" class="time">13:48</a> |
| f13 | I suppose it falls to me, since I own <a href="http://fedoraproject.org/wiki/ReleaseEngineering/Overview">http://fedoraproject.org/wiki/ReleaseEngineering/Overview</a> and it's a bit vague on the definitions of these releases | <a href="#t13:48" class="time">13:48</a> |
| f13 | but I'll welcome anybody to edit that page in my stead. | <a href="#t13:49" class="time">13:49</a> |
| wwoods | so what's the difference between PR and previous snaps? full test-release QA effort? | <a href="#t13:49" class="time">13:49</a> |
| f13 | poelcat: snapshots are just that, snapshots. If they're broken, that means more things need to be fixed. waiting until things are perfect to snapshot for testing.... seems odd | <a href="#t13:49" class="time">13:49</a> |
| f13 | wwoods: From my point of view, PR has some extra attention put on it from releng and QA and developers | <a href="#t13:50" class="time">13:50</a> |
| wwoods | right | <a href="#t13:50" class="time">13:50</a> |
| poelcat | f13: i'm not advocating that :) | <a href="#t13:50" class="time">13:50</a> |
| wwoods | I mean, it's getting the test release treatment from QA | <a href="#t13:50" class="time">13:50</a> |
| f13 | wwoods: the fact that we'd actually slip PR for things is one difference | <a href="#t13:50" class="time">13:50</a> |
| wwoods | right. but the release process is going to skip the normal test-release mirroring junk | <a href="#t13:51" class="time">13:51</a> |
| wwoods | 'cuz there's not time in the cycle | <a href="#t13:51" class="time">13:51</a> |
| wwoods | is that the plan? | <a href="#t13:51" class="time">13:51</a> |
| f13 | wwoods: we're /going/ to have a PR, I don't think there was any guarentee that there would be a working snapshot every week | <a href="#t13:51" class="time">13:51</a> |
| f13 | we'd just /try/ for weekly snapshots | <a href="#t13:51" class="time">13:51</a> |
| f13 | wwoods: yes, that sounds right. | <a href="#t13:51" class="time">13:51</a> |
| f13 | bittorrent will be our primary release path for PR, as we can get the bits up and public within a few hours (5~?) | <a href="#t13:52" class="time">13:52</a> |
| f13 | We've talked about adding mirrors / jigdo to this in parallel | <a href="#t13:52" class="time">13:52</a> |
| wwoods | I think the plan was for a best effort to make working weekly snapshots, but you're correct: no guarantee | <a href="#t13:52" class="time">13:52</a> |
| f13 | and changes to our mirror system may make that more of a reality | <a href="#t13:52" class="time">13:52</a> |
| wwoods | so the PR is the final snapshot, and it's guaranteed to be as installable/functional as any other test release | <a href="#t13:53" class="time">13:53</a> |
| f13 | sounds fair | <a href="#t13:53" class="time">13:53</a> |
| notting | well, if you don't want mirrored jigdo, you can always point the jigdo files at koji *runs away* | <a href="#t13:54" class="time">13:54</a> |
| f13 | well, we're coming up on one hour here, and I'm no closer to fixing the rawhide snafu and the sudden inability to sign any packages. | <a href="#t13:54" class="time">13:54</a> |
| f13 | I'll commit to documenting this somewhere, but it won't happen today, and I'd really rather somebody else do it, I can provide input/corrections as needed | <a href="#t13:55" class="time">13:55</a> |
| poelcat | f13: i'll take a shot at | <a href="#t13:56" class="time">13:56</a> |
| poelcat | it | <a href="#t13:56" class="time">13:56</a> |
| f13 | thanks a ton! | <a href="#t13:56" class="time">13:56</a> |
| notting | is there a good way to post a draft wiki edit so other people can see it? | <a href="#t13:56" class="time">13:56</a> |
| poelcat | what is the exsiting page to update? | <a href="#t13:56" class="time">13:56</a> |
| f13 | poelcat: <a href="http://fedoraproject.org/wiki/ReleaseEngineering/Overview">http://fedoraproject.org/wiki/ReleaseEngineering/Overview</a> is the likely place for it, as it can be referenced by other things | <a href="#t13:56" class="time">13:56</a> |
| f13 | notting: other than coping the source into a new page, I don't think so | <a href="#t13:56" class="time">13:56</a> |
| poelcat | f13: okay; i'll commit to something by mid-week | <a href="#t13:56" class="time">13:56</a> |
| GH-VAIO | web address will be <a href="https://cripperz.org/~username/">https://cripperz.org/~username/</a> | <a href="#t13:57" class="time">13:57</a> |
| GH-VAIO | ops | <a href="#t13:57" class="time">13:57</a> |
| f13 | poelcat: cheers | <a href="#t13:57" class="time">13:57</a> |
| GH-VAIO | hello, antbody here wanna trade shell accout? | <a href="#t13:57" class="time">13:57</a> |
| f13 | GH-VAIO: please go away, extremely off topic for this channel. | <a href="#t13:57" class="time">13:57</a> |
| f13 | ok, anything else for this week? | <a href="#t13:59" class="time">13:59</a> |
Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!