From Fedora Project Wiki
Fedora Release Engineering Meeting :: Monday 2008-03-03
Beta Freeze
- unresolved issues in X, perl, and mkinitrd
- koji slowness also contributing to slip
- delaying freeze and corresponding beta release by one week
- no proposed change to GA date at this time--discuss next week
IRC Transcript
2008-03-03-14:07-meeting-snapshot
| -!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Roll Call | 13:00 | |
| f13 | ping: notting jeremy rdieter wwoods lmacken warren jwb poelcat spot | 13:00 |
|---|---|---|
| warren | meh | 13:00 |
| * spot is here | 13:01 | |
| * lmacken is here | 13:01 | |
| * jeremy | 13:01 | |
| * rdieter here kinda sorta | 13:01 | |
| -!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Fedora 9 Beta | 13:01 | |
| f13 | Tonight, we freeze. | 13:01 |
| * poelcat here | 13:01 | |
| notting | f13: tomorrow, we dine in hell? | 13:02 |
| f13 | I already have f9-beta being populated from dist-f9. I'll commit a change tonight so that dist-rawhide pulls from it | 13:02 |
| f13 | notting: pretty much. | 13:02 |
| f13 | tomorrow's rawhide will be the frozen content | 13:02 |
| * wwoods here | 13:02 | |
| f13 | I'm also going to start signing later today (once the updates push is done) | 13:02 |
| f13 | since people have been slamming planes into the runway the last few days (and today) the next couple days are likely going to be sweeping the decks of all the wreckage. | 13:03 |
| f13 | we're keeping our eyes on the X stuff, as we may just maybe have to delay the release a bit for good X bits to land | 13:03 |
| f13 | but delaying the release won't be the same as keeping rawhide frozen. If we get to the point where we're just waiting for X, we'll let rawhide move on and just wait for X to land to do the final compose+push | 13:04 |
| f13 | We'll ask wwoods to hook us up with a wiki page to start tracking testing and issues | 13:05 |
| f13 | and we already have an F9Beta bug | 13:05 |
| wwoods | yup | 13:05 |
| jeremy | with a fair number of open things on it already | 13:05 |
| f13 | right | 13:05 |
| f13 | we should be adding more as we see them. | 13:05 |
| f13 | I'm asking jwb to do test composes of ppc, but I will take care of pushing the final buttons to make the content show up | 13:07 |
| f13 | we may use machines in PHX for it, if mmcgrath has the resources, otherwise we'll do it in BOS again. | 13:07 |
| f13 | but since the netapp drop point is in PHX, PHX would be preferred. | 13:07 |
| f13 | that's about all I have initially for Beta stuff, are there any questions/issues that you'd like discussed? | 13:08 |
| notting | X bad | 13:09 |
| f13 | yes | 13:09 |
| spot | f13: any chance of perl 5.10.0 slipping in? | 13:09 |
| * notting looks at the clock | 13:09 | |
| f13 | spot: how close are you? | 13:09 |
| wwoods | Oh uh. Someone should probably mention that boot.iso is gone baby gone | 13:09 |
| f13 | oh yeah | 13:09 |
| jeremy | wwoods: yeah, need to do some release note stuff | 13:09 |
| wwoods | and that netinst.iso and/or SNAKE are the easy ways to do rawhide installs | 13:10 |
| f13 | I need to throw some code at pungi to move/copy netinst.iso from images/ out to iso/ | 13:10 |
| notting | need to add relnotes about upstart | 13:10 |
| spot | f13: maybe 330 packages done at this point | 13:10 |
| spot | out of about 800 | 13:10 |
| spot | slowdown is koji, not me. | 13:11 |
| stickster | notting: http://fedoraproject.org/wiki/Docs/Beats/SystemDaemons | 13:11 |
| stickster | I added a short note there, but it might be able to use clarification. | 13:11 |
| spot | koji just can't build fast enough, its doing one an hour, on average | 13:11 |
| notting | f13: i may be of surprisingly little use this week, i appear to have caught the flu | 13:11 |
| warren | notting, Sinucleanse! | 13:12 |
| f13 | spot: there aren't any places you can make better chain groups instead of single links? | 13:12 |
| notting | warren: um, yeah. i'm sure that will do wonders for aches and pains ;) | 13:12 |
| spot | f13: not until i get about 400 done. | 13:12 |
| notting | stickster: oh dear. someone changed a script to be started from upstart? must beat them ;) | 13:12 |
| stickster | notting: I thought someone mentioned zeroconf in some idle chitchat here or f-devel-l | 13:13 |
| stickster | It's not firsthand knowledge, feel free to delete. | 13:13 |
| wwoods | I wonder if initscripts can send out upstart events when they do their thing | 13:13 |
| notting | wwoods: see the linked bug from the 'upstart' tracker about cross-system deps | 13:14 |
| * stickster & # derailing meeting-- | 13:14 | |
| notting | wwoods: sysv -> upstart is doable. upstart -> sysv is hard | 13:14 |
| f13 | spot: ok, it's going to be close. I don't like our options here. Slip beta, land it during the freeze and cross fingers, land it after beta and cross fingers, delay until F10 | 13:14 |
| wwoods | yeah, sorry, abandoning tangent | 13:14 |
| * jeremy is strongly against "land it after beta" | 13:14 | |
| spot | i would really hate to delay | 13:14 |
| spot | if i had a few more days, i could get it done | 13:14 |
| f13 | that leaves us either with slipping beta freeze, or landing it after the freeze and hoping beta will be ok | 13:15 |
| poelcat | f13: do we know any more about the state of X which could affect beta? | 13:15 |
| * jwb is here now | 13:15 | |
| f13 | poelcat: many drivers aren't working right now, ajax is working on it as fast as he can | 13:15 |
| spot | X is having rather annoying keymapping issues | 13:16 |
| f13 | poelcat: but ajax/jrb will know more details. | 13:16 |
| warren | Have we had installable rawhide at all in the past weeks? | 13:16 |
| spot | (hint: don't select multiple items in evolution) | 13:16 |
| f13 | warren: yes | 13:16 |
| jeremy | spot: the keymap problems are worse on upgrades than new installs -- and you can change your config to make it work. but yeah, still not pretty | 13:16 |
| wwoods | today's is installable AFAIK | 13:16 |
| spot | jeremy: any workarounds would be good to document | 13:16 |
| wwoods | 650/1050 packages installed; I'll let you know if it reboots OK | 13:17 |
| jeremy | wwoods: still not rebootable due to the mkinitrd thing. but I guess I'm going to end up looking at that this afternoon since peter is hosed by rhel | 13:17 |
| f13 | provided you don't use lvm or label= mounting methods | 13:17 |
| wwoods | fucking seriously? still no progress on that mkinitrd bug? | 13:17 |
| f13 | to be fair, it was the weekend | 13:18 |
| warren | jeremy, what needs to be done? | 13:18 |
| warren | bug #? | 13:18 |
| notting | wwoods: iscsi and 5.2 happened | 13:19 |
| wwoods | warren: bug 435228 | 13:19 |
| bzbot | Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=435228 low, low, ---, Peter Jones, NEW , mkinitrd doesn't grab dm modules for LVs listed by LABEL or UUID | 13:19 |
| warren | is the nash and dhcp libs dep problem fixed? | 13:19 |
| f13 | wasn't aware of such a problem. | 13:19 |
| wwoods | the second patch, by the way, does *not* fix the problem | 13:20 |
| jeremy | warren: yes. | 13:20 |
| f13 | right, so we clearly need work on this issue | 13:21 |
| f13 | that's not all that surprising | 13:21 |
| warren | Isn't it pretty clear that we have to slip the freeze? | 13:21 |
| wwoods | so: a bug I submitted a fix for last week has made all rawhide installs non-bootable and therefore we have no pre-freeze testing. | 13:21 |
| warren | perl is a huge change | 13:21 |
| f13 | warren: slipping the freeze just means more shit gets broken. | 13:22 |
| f13 | this happens /every/ /single/ /freeze/ | 13:22 |
| jwb | when is freeze date again? | 13:22 |
| wwoods | I don't think slipping the freeze is a good idea. Lengthening it might be necessary. | 13:22 |
| f13 | jwb: tonight | 13:22 |
| jwb | i agree with wwoods | 13:22 |
| f13 | right, we always have the option of lengthening the freeze | 13:23 |
| f13 | and that doesn't seem unreasonable in this case. | 13:23 |
| f13 | especially for perl to land | 13:23 |
| f13 | bah | 13:23 |
| * f13 wipes wagamamas off his laptop | 13:24 | |
| wwoods | that's my gut feeling on the matter, anyway | 13:24 |
| wwoods | anyway I spent a couple of days hacking on the mkinitrd thing so feel free to ask me if you want logs / debugging info /etc | 13:24 |
| notting | if we want perl 5.10 in f9, we should land it now | 13:24 |
| f13 | so here is my take on it | 13:25 |
| f13 | I don't like announcing ahead of time that we're slipping a release date, or lengthening the freeze | 13:25 |
| f13 | this just invites people to take their time and land stuff at the last possible moment, not giving us any time to verify integration and test. | 13:25 |
| f13 | so I'd rather announce that we're going ahead as scheduled, and only announce changes if they become absolutely necessary | 13:26 |
| f13 | notting: I think we all agree on that | 13:26 |
| * spot wants to land it, its just a matter of getting the builds done | 13:26 | |
| wwoods | is it just a matter of rebuilding, or are there lots of fixes necessary? | 13:26 |
| jeremy | f13: the flip side is that people pull of heroics only to get told "haha, just kidding. you had more time" | 13:27 |
| f13 | Proposal: Maintain schedule as is. Keep eye on Perl builds for timing. Also let ajax know if our contengency plan of delaying the release ( he should already be aware of this via jrb ). Delay the release of the beta for Perl/X/Whatever else only when it becomes necessary. | 13:27 |
| warren | jeremy, +1 | 13:28 |
| warren | jeremy, that also means that some people might give up when they really could have fixed something in time. | 13:28 |
| f13 | it is a balancing act. | 13:28 |
| spot | wwoods: at this point, almost no fixes | 13:29 |
| spot | just rebuilds | 13:29 |
| jwb | spot, is this being done on the side still? | 13:30 |
| spot | jwb: in dist-f9-perl | 13:30 |
| jwb | and the plan is to mass tag those builds to dist-f9? | 13:30 |
| * spot nods | 13:30 | |
| jwb | k... | 13:30 |
| jeremy | f13: -1. | 13:30 |
| warren | f13, -1 | 13:31 |
| warren | jeremy, what do you suggest instead though? | 13:31 |
| jwb | i'll refrain from voting at the moment | 13:31 |
| jeremy | I think that we need to take a real look at where we are and actually adjust the schedule based on what we really think rather than saying "lalalalala" when we know that there's really _zero_ chance of us having a real freeze of any kind tonight | 13:31 |
| poelcat | jeremy: +1 | 13:32 |
| f13 | jeremy: my only problem with that is the schedule has been known well in advance. Without any sort of freeze, I have very little confidence that anything will actually improve | 13:32 |
| f13 | because, this shit has been /bad/ for a long time | 13:32 |
| jwb | f13 has a point | 13:33 |
| f13 | we've really only had two relatively stable installable points. Just after F8 release, and the Alpha freeze~ | 13:33 |
| jwb | would it be possible to grant tag privileges to ajax for X stuff? | 13:34 |
| jeremy | and being unrealistic changes anything how exactly? | 13:34 |
| f13 | jeremy: I'm not sure of any other way to get the message across to fix shit and stop breaking things | 13:34 |
| wwoods | doesn't a freeze do that? | 13:35 |
| f13 | perhaps we need to move the feature freeze back before the beta freeze | 13:35 |
| f13 | because beta just tends to be the crap people land last minute | 13:35 |
| jwb | this isn't a beta | 13:35 |
| f13 | but I'm game for jeremy's suggestion | 13:35 |
| * warren notes that there are different kinds of "features", many don't threaten stability. | 13:35 | |
| f13 | jeremy: Looking at what we have now, what would you consider realistic to do? | 13:35 |
| * f13 notes that feature freezes aren't just for stability either. | 13:36 | |
| wwoods | yes, but many of the things that *are* causing instability are feature-related (xserver 1.5, encrypted filesystems, etc) | 13:36 |
| f13 | It seems that the two main pain points are X and mkinitrd | 13:37 |
| jeremy | wwoods: what instability is encrypted filesystems causing? | 13:37 |
| f13 | and then the desire to see perl land. | 13:37 |
| f13 | do we have a good idea of what other than those things are threatening beta? | 13:37 |
| lmacken | 13:30 spot> jwb: in dist-f9-perl | 13:37 |
| lmacken | erm | 13:37 |
| warren | the size of perl's changes doesn't worry you? | 13:37 |
| wwoods | jeremy: it kept people from booting for a while and broke rescue. current mkinitrd junk isn't encryption-related | 13:38 |
| warren | it might build but how well does it work? | 13:38 |
| spot | they really only affect perl, and about 10 other packages. | 13:38 |
| wwoods | so I guess I should have said "*have been* causing instability" | 13:38 |
| f13 | warren: that's what beta is /for/ | 13:38 |
| wwoods | sorry for confusion! | 13:38 |
| spot | in my testing, those other packages work fine with 5.10 | 13:38 |
| warren | nice | 13:38 |
| f13 | warren: to get testing on these features, not to say "everything is fine, have fun!" | 13:38 |
| wwoods | also encryption in rescue mode totally works now, so yeah.. encryption is totally A-OK fine | 13:40 |
| f13 | so what we're really talking about is X, perl, and mkinitrd. | 13:41 |
| f13 | I'd be willing to bet we have a working mkinitrd before the end of the day, if people don't bug jeremy for the rest of the day | 13:41 |
| f13 | perl is going as fast as it can, and ajax is working on X, we need to leave him alone too | 13:41 |
| f13 | what else is going on? | 13:42 |
| * spot could be done in a week, assuming the builders keep going | 13:42 | |
| spot | (worst case) | 13:42 |
| jeremy | perl needing a week sounds like we either need to a) drop perl 5.10 or b) slip a week. I don't really see how we're having any other conversation | 13:42 |
| spot | it only needs a week because the builders only spit out a package an hour | 13:43 |
| jwb | if we drop perl, is it dropped for RC/Final as well? | 13:43 |
| wwoods | can't we freeze now and just lengthen the freeze for perl and xserver? | 13:44 |
| f13 | jwb: yes, if it doesn't go into beta, it doesn't go in. | 13:44 |
| wwoods | I'm trying to find some way to express how desperately I want to spend some time stabilizing rawhide so people can actually test it | 13:44 |
| jwb | wwoods, that's what i was getting at with giving ajax tagging privileges | 13:44 |
| jeremy | wwoods: I'd rather let us be honest. there are some other things which are almost certainly on the fence or people pushing hard at the last minute | 13:44 |
| wwoods | Which is kind of the point of the freeze | 13:44 |
| f13 | well, I'm feeling of the opinion that my lofty goal of 1 week freezes is pretty trashed by now | 13:45 |
| f13 | and I think we need to extend the amount of days we spend in freeze mode anyway | 13:45 |
| f13 | because the 1 week stuff is killing me. | 13:45 |
| f13 | so either way I'd want to extend freezes by a week~ | 13:46 |
| warren | f13, prior to you arriving at RH, very long freezes seemed to kill people's motivation too. | 13:46 |
| warren | f13, but back then there were fewer people to tag things in, so maybe that is less of an issue now. | 13:46 |
| jwb | i think he knows that | 13:46 |
| f13 | warren: that was when all freezes were blocking, and there was less communication | 13:46 |
| jeremy | long freezes just lead to more problems after the freeze | 13:46 |
| wwoods | I know long freezes suck. Also, on second thought: most of the instability we're dealing with is in things like kernel, X, etc | 13:47 |
| f13 | alright, so it seems like all my ideas are shit today, jeremy, what are we going to do? | 13:47 |
| wwoods | stuff we'd be accepting during a freeze anyway | 13:47 |
| wwoods | so maybe a longer freeze - while sounding good in theory - would just piss off all the devs that are trying to work on otherwise-stable stuff | 13:47 |
| warren | longer freeze pisses off devs who work on stuff that is irrelevant to stability | 13:48 |
| * jeremy really thinks we just need to bite the bullet and push back the freeze. the stuff that's causing us the pain is the stuff that's working on being stabilized | 13:48 | |
| wwoods | warren: right | 13:48 |
| f13 | jeremy: I'm ok with that, I'm talking about something else now. The fact that freeze on tuesday && have tree ready for mirrors by next Monday is just unrealistic | 13:48 |
| f13 | especially if we want any sort of testing beyond "hey look, it installed" on the bits we hand to the mirrors | 13:49 |
| jeremy | f13: I think that's one of the points of being a community-focused distro. we can't do all the testing before we call it a beta | 13:49 |
| jeremy | which, yes, leads to the beta having more problems than in the Old Days (tm) | 13:50 |
| f13 | jeremy: thus far, we can't really do /any/ testing either. | 13:50 |
| f13 | but now we're just arguing because people are pissed at the situation. | 13:50 |
| jeremy | f13: that's a little unfair -- there are a number of things that are getting tested. even on the install side of things (cf I fixed several bugs yesterday that people were finding and reporting) | 13:50 |
| f13 | so. | 13:51 |
| f13 | Proposal: Delay freeze by a week, keep it the same length as previously scheduled. | 13:51 |
| warren | proposed amendment: announce it today, but warn people to not add things that could cause further instability | 13:51 |
| jwb | or what? | 13:52 |
| warren | jwb, pointy sticks | 13:52 |
| wwoods | yelldown warhellride | 13:52 |
| jeremy | jwb: those who add instability will be called out in public | 13:52 |
| jeremy | jwb: that's the only real "or what" that we have -- peer pressure | 13:52 |
| warren | jwb, Point to the crater that used to be the orphanarium. | 13:52 |
| wwoods | we will seriously doubt your commitment to sparklemotion | 13:52 |
| jwb | i'm just going to stay silent because i have nothing nice to say about any of that | 13:53 |
| wwoods | fair 'nuff | 13:54 |
| wwoods | you may still reserve the right to say "toldja so" if something really terrible actually happens | 13:54 |
| jwb | no, that's not what i'm thinking | 13:54 |
| f13 | can we at least agree to the proposal? | 13:56 |
| jwb | basically, public shaming didn't work for the first beta freeze date. what makes us think it'll work now? | 13:56 |
| jwb | f13, +1 | 13:56 |
| jeremy | f13: +1 | 13:56 |
| wwoods | +1 to the proposal | 13:56 |
| notting | f13: +1, i suppose. not really any other option | 13:57 |
| spot | +1 | 13:57 |
| lmacken | f13: +1 | 13:57 |
| f13 | alright, that's probably good for Beta. | 13:57 |
| -!- f13 changed the topic of #fedora-meeting to: Fedora Release Engineering Meeting - Open Discussion | 13:57 | |
| notting | adjusting final as well? | 13:57 |
| f13 | the floor is open | 13:57 |
| f13 | notting: I wish we wouldn't have to | 13:58 |
| f13 | or I'd rather not that is | 13:58 |
| jwb | i wanted to talk about spins, but that's going to take too long | 13:58 |
| f13 | notting: if we don't slip final, that still gives us 3 weeks from beta release to final freee | 13:59 |
| warren | let's discuss the final schedule next week? | 13:59 |
| f13 | that doesn't seem terribly unreasonable. | 13:59 |
| jeremy | yeah, I'd like to get to looking at the doom that is sysfs | 14:01 |
| jwb | huh? | 14:01 |
| notting | jeremy: that's a little large of a project for a few weeks | 14:01 |
| notting | doom_level(sysfs) == MAX_INT | 14:02 |
| jeremy | true... and "want" is probably strong | 14:02 |
| jeremy | but I'd like to get the @!#$ bug fixed | 14:02 |
| f13 | alright, close of meeting | 14:03 |
Generated by irclog2html.py 2.3 by Marius Gedminas - find it at mg.pov.lt!