ReleaseEngineering/Meetings/2008-apr-14

Fedora Release Engineering Meeting :: Monday 2008-04-14

Preview Release (aka PR) Status

• Not ready--scheduled for availability on 2008-04-11
• Problems with yum and other compose tools
• Snafu with package signing which caused rawhide to revert back to all unsigned copies of most packages--100G worth of churn
• Spend rest of today untangling rawhide situation
• Attempt Preview Release on 2008-04-15 with no change to GA date of 2008-04-29

Mass Branching

• Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.
• Should probably have a CVS outage to get the branching done--usually takes a full work day
• Discusison left unfinished

What is the Preview Release?

• Just another snapshot or equivalent to Alpha and Beta which requires more formal testing before release?
• Discussion about confusion over what preview release is
• Need clearer definition for what Preview Release and Release Candidate mean
• DECISIONS:

1. Poelcat to take a shot at documenting purpose and criteria for snapshots and test releases 1. Preview Release will be as installable and functional as other test releases (Alpha and Beta)

• http://fedoraproject.org/wiki/QA/TestResults/Fedora9Install/PreviewRelease

IRC Transcript

f13	ping: notting jeremy spot lmacken warren rdieter wwoods poelcat jwb	13:00
* notting is here		13:00
rdieter	yo	13:00
notting	although still drowning in mail	13:00
skvidal	notting: I've got a filter that helps with that - search for @ in the sender's address and delete the mail if it is found	13:01
wwoods	ooh, was about to reboot for preupgrade testing	13:03
* wwoods stays his hand for a bit		13:03
f13	alright, lets kick this pig.	13:05
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Preview Release		13:05
f13	I"d really like to get PR out the door, but the universe hates me.	13:05
f13	I'm convinced.	13:05
f13	After all the battling with yum and other compose tool fun, I was all set to get PR composed and out the door today.	13:05
* jeremy is sort of here		13:05
f13	Unfortunately, that's not to be.	13:05
f13	Due to a snafu with package signing, we caused rawhide to revert back to all unsigned copies of most packages, some 100G worth of churn	13:06
f13	probably not the best to compose PR against.	13:06
wwoods	ack!	13:06
* poelcat here		13:06
jeremy	f13: doh.  I was wondering why I saw so much churn in my sync but thought it was going to signed not the unsigned packages :(	13:06
f13	I'm in the middle of trying to recover the situation, and we've turned off snapmirror tos that PHX/TPA don't get more of the bad bits	13:06
f13	jeremy: sadly, mash has a "if I don't find the preferred signed copy, I fall back to unsigned" method that I wasn't fully aware of.	13:07
* rdieter was wondering why his rawhide mirror sync was taking so long to finish today, rats.		13:07
notting	f13: well, the alternative is to not fall back and fail	13:07
f13	so I'll likely spend the entire day cleaning up from that, and investigating why sign_unsigned suddenly is getting md5 mismatches on the things it signs.	13:08
notting	which also won't help you	13:08
f13	notting: or stop switching gpg sigs needlessly	13:08
f13	at any rate, I may end up composing a PR against a recovered snapshot from /yesterday's/ rawhide	13:09
f13	if it looks like recovering todays rawhide bits is going to take too long	13:09
f13	but I haven't looked at what all got signed last night that would be good to have in the PR	13:10
f13	Looking beyond PR...	13:11
f13	I think we'll still be taking in tag requests until early next week	13:11
f13	then we need to crack down and start considering release candidates, and what are actual release blockers	13:11
f13	anything more on Preview Release?	13:13
notting	do we know of any PR blockers aside from the 'getting it composed issues'?	13:14
f13	jeremy said there was one bug left on the PR blocker list	13:14
jeremy	the x keyboard config one	13:14
f13	but that's the only thing I'm aware of, I thought we were in pretty damn good shape for PR	13:15
jeremy	and I really really really don't know if I feel comfortable with the RC being the first thing that has a fix for that	13:15
jeremy	I know ajax was working on it on friday, but I didn't quite catch what he was saying the problem was on the bus (since I was in the python unicode discussion instead ;-)	13:15
jeremy	and I've been in class all morning	13:15
notting	what's wrong with keyboards today?	13:16
f13	if you set a layout at install, it doesn't follow through to X once installed I think	13:17
notting	oh, because we're only writing to sysconfig, not to X	13:19
notting	unless you use s-c-keyboard, which writes to both	13:19
f13	I think so?  I'm not clear on the problem space	13:19
f13	so for the keyboard thing, we may be putning the PR to tomorrow anyway	13:21
-!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Mass Branching		13:22
rdieter	punt++ , will the package signing be resolved by then?	13:22
wwoods	since we're now a week from RC I don't think we should slip the PR any further for fixes	13:22
f13	Toshio says that the tools (pkgdb, etc..) are ready for the mass branching.	13:23
f13	unfortunately unless we want to mailbomb everybody, we should probably have a CVS outage to get the branching done, and that usually takes a full work day	13:23
f13	and I have 0 time to focus on it right now	13:23
f13	hrm,	13:23
f13	lets put CVS branchingon hold for a moment.	13:23
f13	wwoods: You'd be OK with PR going out with the current blocker unfixed?	13:24
f13	(if so, mind moving that blocker off of the PR blocker list)	13:24
wwoods	lemme recheck the blocker list, but seriously if we're releasing another snapshot in a week *and* we have a fix approaching the proverbial runway	13:25
wwoods	then, yeah, get the dang thing into people's hands and release note it	13:25
wwoods	"we know, it'll be fixed next week, just ssshh"	13:25
f13	wwoods: I don't think there is really time in the schedule to "release" RCs	13:25
f13	wwoods: so we'd be relying on rawhide users to confirm if it's fixed or not	13:26
wwoods	wait, what? I must have misunderstood something	13:26
wwoods	22 April 2008	13:26
wwoods	Release Candidate 1	13:26
f13	RCs are seriously going to be compose as fast as you can, multiple times a day, and drop it on the mirrors ASAP	13:26
wwoods	There's no schedule for *subsequent* RCs - those are at our option	13:26
wwoods	hm, okay	13:27
wwoods	so we're not planning on putting RC1 on the mirrors?	13:27
f13	I have to have something going to the mirrors by the 24th really for a successful 29th release	13:27
f13	wwoods: there is not enough time.	13:27
wwoods	forgive me if I'm getting confused, I think I'm coming down with something	13:27
f13	mirrors == 3~ days of sync lead time	13:27
f13	often 3 business days	13:28
f13	that's for a "minor" release.  For a major release, we want a lot more time to have a lot more mirrors ready for the bit flip	13:28
notting	well, i think the issues is just that the schedule says 'Preview Release' and 'Release Candidate 1'	13:28
notting	are those defined?	13:28
f13	notting: not well.  PR is new for this round, and "RC" is just something we used to call attempts to do the final compose	13:28
f13	I don't think there was any promise that "RC" would see the light of day in the public space	13:29
notting	no, but we should probably add something somewhere that defines what they are, or we'll be answering this question again	13:29
wwoods	Yeah, my memory's hazy but I think f13's right and I've just confused myself	13:30
wwoods	so yeah, that needs a more detailed explanation somewhere, I guess	13:30
wwoods	and if we're not planning on doing another fullblown worldwide mirrored release until final	13:30
wwoods	then yes, I guess we'll need to continue to block on bug 438246	13:31
buggbot	Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=438246 low, low, ---, Kristian Høgsberg, ASSIGNED , Keyboard layout selected during install is forgotten at post-install boot	13:31
f13	I know we've already got a potential schedule for F10, but yeah, I think we seriously need to discuss what we're trying to accomplish with PR, and RC, etc...	13:32
f13	I originally thought that PR was just going to be the regularly scheduled snapshot of the week attempt	13:32
f13	just under a new name	13:32
f13	not that we'd be holding it up for bugs, or treating it all that special, or mirroring it, just bog standard snapshot on torrent	13:33
f13	and that we'd continue to use rawhide for testing things we want to get fixed before we say "Ok, we're good enough, start attempting the release spin" and if the spin is good, as in nothing fell over during compose, that was it, we're shipped	13:33
f13	trying to do public releases of candidate trees is going to put a serious hurt on things	13:34
f13	anyway.... back to /topic	13:35
wwoods	No, we wanted a third fullblown test release before final	13:35
f13	if anybody wants to tackle cvs branching, by all means, I'd be very happy	13:35
f13	I have some notes from last time, and toshio started an SOP page to outline calls to pkgdb and such	13:36
* poelcat wonders how much we reduce our potential pool of testers when we reduce availability to rawhide		13:36
notting	wwoods: f13: so, how do we resolve this dichotomy	13:36
wwoods	but RCs don't need to be fullblown mirrored releases	13:36
f13	by the time we get to RC state, it should seriously be an RC, as in all the bugs are fixed, we're just putting the tested builds into a tree, and making sure the tree compose itself doesn't fall over	13:37
wwoods	right	13:38
f13	I'm not sure what the value is of trying to put that tree out to the public and waiting for somebody to give it thumbs up	13:38
wwoods	that's what the PR is for	13:38
poelcat	wwoods: +1	13:39
f13	wwoods: so what do we do with the time between cuttin the PR, having it mirrored, getting people to download and test, and report things, and cutting RC?	13:39
f13	do we take in other bugfixes people find?	13:39
wwoods	yes?	13:39
f13	do we just say "nope, sorry, th'at snot a blocker, fix it in an update?"	13:39
wwoods	the reason it's called "Preview Release" and not "Public Release Candidate" is that we *knew* there'd be unfixed blockers at this point	13:40
f13	and if we're going to take fixes in, what's the criteria for what we wait for to be fixed for PR, vs what we'll allow to be fixed after PR goes out?	13:40
wwoods	so calling it an RC would have just been an obvious lie	13:40
wwoods	the criteria for PR are the same as for Alpha and Beta	13:40
f13	uh..	13:40
wwoods	seriously, (alpha, beta, preview_release) = (test1, test2, test3)	13:41
f13	if that were the case, we wouldn't be blocking on the X thing	13:41
wwoods	we changed the names and shifted dates but that was the original plan	13:41
wwoods	is that right? yeah I guess so	13:41
f13	wwoods: I think that really fell apart when we scheduled the PR for 3 days after the final freeze	13:42
wwoods	I'm not sure what criteria made the X thing a blocker in the first place, actually	13:42
f13	and didn't give it any compose/stage/sync time	13:42
f13	I was under the (mistaken?) impression that PR was just going to be a weekly snapshot, tossed onto torrent	13:43
wwoods	no, PR was supposed to map to test3	13:43
f13	and would service as a post-freeze media set to be the basis of further testing	13:43
wwoods	*that* part is true	13:43
f13	ok, our signals were quite crossed then	13:43
f13	the schedule certainly doesn't have room for a full blown test3 like release	13:44
wwoods	okay then	13:44
f13	I thought we obviated the need for a 3rd major developmental milestone	13:44
wwoods	let's go with your plan	13:44
wwoods	because it reduces our workload	13:44
f13	'cause it just make things take too long	13:44
wwoods	and if anyone actually flips out	13:44
f13	we'll have me to blame (:	13:44
wwoods	or if it actually has an adverse effect	13:44
notting	so, 1) do we need to document this 2) do we need to adjust the f10 schedule to have PR be a 'release'?	13:45
wwoods	then we'll rejigger the F10 schedule around the assumption that PR == test3	13:45
wwoods	1) yes, 2) no, unless this goes horribly awry	13:45
wwoods	since we're doing (mostly) weekly snapshot releases, we're putting a lot more testable bits in people's hands	13:46
wwoods	than just test1/test2/test3	13:46
wwoods	Alpha/Beta/snapshot,snapshot,snapshot,snapshot...pr,rc,rc,rc,rc..	13:46
wwoods	obviously that's More, Fresher Bits	13:47
wwoods	practially farm-fresh	13:47
poelcat	wwoods: except that the snaps have to install... they haven't for me	13:47
wwoods	that has nothing to do with the schedule.	13:47
f13	and yet they have for a very large set of users	13:47
wwoods	that's a development problem particular to this release and your setup	13:47
wwoods	so: sorry	13:47
poelcat	wwoods: your point about people having more bits to test	13:47
f13	please, lets not get trapped in the "it doesn't work for me, it can't possibly work for anybody else" syndrome	13:47
wwoods	I think I just showed that there's more, fresher stuff available for test	13:48
notting	ok, so who is going to document/announce the PR/RC cycle?	13:48
notting	and we can table the 'how do we do F10' until we're less in AAAUGGH mode	13:48
f13	I suppose it falls to me, since I own http://fedoraproject.org/wiki/ReleaseEngineering/Overview and it's a bit vague on the definitions of these releases	13:48
f13	but I'll welcome anybody to edit that page in my stead.	13:49
wwoods	so what's the difference between PR and previous snaps? full test-release QA effort?	13:49
f13	poelcat: snapshots are just that, snapshots.  If they're broken, that means more things need to be fixed.  waiting until things are perfect to snapshot for testing.... seems odd	13:49
f13	wwoods: From my point of view, PR has some extra attention put on it from releng and QA and developers	13:50
wwoods	right	13:50
poelcat	f13: i'm not advocating that :)	13:50
wwoods	I mean, it's getting the test release treatment from QA	13:50
f13	wwoods: the fact that we'd actually slip PR for things is one difference	13:50
wwoods	right. but the release process is going to skip the normal test-release mirroring junk	13:51
wwoods	'cuz there's not time in the cycle	13:51
wwoods	is that the plan?	13:51
f13	wwoods: we're /going/ to have a PR, I don't think there was any guarentee that there would be a working snapshot every week	13:51
f13	we'd just /try/ for weekly snapshots	13:51
f13	wwoods: yes, that sounds right.	13:51
f13	bittorrent will be our primary release path for PR, as we can get the bits up and public within a few hours (5~?)	13:52
f13	We've talked about adding mirrors / jigdo to this in parallel	13:52
wwoods	I think the plan was for a best effort to make working weekly snapshots, but you're correct: no guarantee	13:52
f13	and changes to our mirror system may make that more of a reality	13:52
wwoods	so the PR is the final snapshot, and it's guaranteed to be as installable/functional as any other test release	13:53
f13	sounds fair	13:53
notting	well, if you don't want mirrored jigdo, you can always point the jigdo files at koji *runs away*	13:54
f13	well, we're coming up on one hour here, and I'm no closer to fixing the rawhide snafu and the sudden inability to sign any packages.	13:54
f13	I'll commit to documenting this somewhere, but it won't happen today, and I'd really rather somebody else do it, I can provide input/corrections as needed	13:55
poelcat	f13: i'll take a shot at	13:56
poelcat	it	13:56
f13	thanks a ton!	13:56
notting	is there a good way to post a draft wiki edit so other people can see it?	13:56
poelcat	what is the exsiting page to update?	13:56
f13	poelcat: http://fedoraproject.org/wiki/ReleaseEngineering/Overview is the likely place for it, as it can be referenced by other things	13:56
f13	notting: other than coping the source into a new page, I don't think so	13:56
poelcat	f13: okay; i'll commit to something by mid-week	13:56
GH-VAIO	web address will be https://cripperz.org/~username/	13:57
GH-VAIO	ops	13:57
f13	poelcat: cheers	13:57
GH-VAIO	hello, antbody here wanna trade shell accout?	13:57
f13	GH-VAIO: please go away, extremely off topic for this channel.	13:57
f13	ok, anything else for this week?	13:59

Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!