ReleaseEngineering/Meetings/2008-sep-08

Fedora Release Engineering Meeting :: Monday 2008-09-08

Fedora 8 & 9 Updates

• updates are pushed to a new location for 8 and 9 (testing too)
• doing testing with the updated fedora-release for transition and newer PackageKit builds to handle the key import
• working on issues where PackageKit is failing
• f13 sending out an announcement today about current status to keep people informed
• see IRC transcript for additional details

F10 Beta

• rawhide has been very unstable
• push beta freeze from 2008-09-09 to 2008-09-11 while holding to other scheduled dates?
  • take discussion and decision to fedora-devel-list

IRC Transcript

f13	ping: notting jeremy wwoods lmacken poelcat warren jwb rdieter spot	10:03
* wwoods here		10:03
* poelcat here		10:03
* notting is here		10:03
rdieter	here	10:04
* warren here		10:04
* jwb not here		10:05
* spot is here		10:06
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - F8/9 Updates		10:06
f13	we've got updates pushed to a new location for 8 and 9 (testing too), and we're doing testing with the updated fedora-release for transition and newer PackageKit builds to handle the key iimport	10:07
warren	http://kojipkgs.fedoraproject.org/packages/fedora-release/8/6.transition/noarch/fedora-release-8-6.transition.noarch.rpm	10:07
notting	how are things working *outside* of pk?	10:07
warren	http://kojipkgs.fedoraproject.org/packages/fedora-release/9/4.transition/noarch/fedora-release-9-4.transition.noarch.rpm	10:07
f13	we're running into some potential issues with PackageKit and discussing those with hughsie	10:07
f13	notting: pirut is untested IIRC, and yum just works	10:07
warren	yum confirmed on both F8 and F9. pirut untested.	10:07
warren	ppc untested	10:08
warren	f13: what was the "PK falling over" issue you ran into?	10:08
f13	warren: I tried to uncheck 200+ updates	10:08
f13	individually	10:08
warren	f13: hmm, that might be a separate bug	10:09
wwoods	I'm testing on ppc64 right now, it's perfectly happy	10:09
jwb	yay	10:09
f13	warren: it is, and not a blocker.	10:09
f13	warren: I didn't know about the right click -> uncheck all option	10:09
jwb	f13, please ping me when you get to open forum	10:09
f13	so I'm re-installing F9 and trying from the start again.	10:09
f13	the only thing I'm concerned about at this point is that the update logic in F9 GAs PK works so that you get the new backend next time you try to do updates.	10:10
warren	it worked for me from 2.3.x	10:10
warren	I'm also concerned that the current PackageKit-0.2.5 no longer shows the PK icon in the tray after you relog	10:11
wwoods	doesn't appear after a reboot either	10:11
warren	users might be confused if they no lnoger see notification	10:11
wwoods	AFAICT it doesn't work at all	10:11
notting	do we have our 'what is going on, and how to fix it if things go horribly wrong' FQ?	10:11
notting	(that's FAQ, sorry)	10:11
f13	notting: not as of yet IIRC.	10:11
warren	notting: well, assuming hughsie fixes these two issues, then there shouldn't be any common case of things going wrong.	10:12
f13	well	10:12
warren	notting: simply "Update once, then update again"	10:12
notting	doesn't mean we shouldn't document how to manually import the keys, etc	10:12
f13	there is always the "in case of PK fire, use yum from the command line"	10:12
f13	notting: +1	10:12
f13	I agree that we should have "if all else fails, here is how to do it manually"	10:12
notting	'smart and apt are left as exercises for the reader'	10:13
warren	notting: I dunno about manual import, it should sufficient to say "Use yum" which will do it with great certainty.	10:13
notting	*cough*	10:13
warren	notting: really don't want to confuse the announcement with lots of complicated things that wont matter to almost anyone	10:13
f13	warren: thats why they don't go in the announcement, they go in the FAQ	10:14
f13	the announcement just leads to the FAQ	10:14
warren	ok	10:14
warren	I strongly believe that we cannot release this with the current PackageKit-0.2.5	10:15
warren	unacceptable to have the tray icon gone	10:15
warren	sure it could be fixed later, but how will those users know it is available?	10:16
f13	well, I'm wanting to hear back from hughsie on this first, but I don't want to continue dragging things out when our non PK users could be up and going /now/.	10:17
wwoods	I think it would be irresponsible to push something that's known to be broken in an visible (and confusing) way without at least getting a first-glance opinion from hughsie	10:18
f13	wwoods: sure, which is why I want to hear back from hughsie first.	10:18
wwoods	right	10:20
f13	he's on the phone now, he'll get to us when he's done.	10:20
* nirik has pinged hughsie on gimpnet... he's on the phone, but will try and hop over in a bit.		10:20
warren	OK, aside from PK, does everyone understand the next steps?	10:21
f13	anything further on updates before we move on? Updates will be a constant item we're working on for the next few days	10:21
f13	warren: outline them for us?	10:21
warren	1) Build final fedora-release packages for F8 and F9. This appears to	10:21
warren	be identical to the packages I built previously for testing, since no	10:21
warren	errors were discovered.	10:21
warren	2) Sign this new fedora-release, PackageKit and gnome-packagekit with	10:21
warren	oldkey.	10:21
warren	3) Erase everything in oldrepo updates for F8. Put new fedora-release	10:21
warren	there and createrepo.	10:21
warren	4) Erase everything in oldrepo updates for F9. Put new fedora-release,	10:21
warren	PackageKit and gnome-packagekit there and createrepo.	10:21
warren	5) Erase oldrepo contents of F8 and F9 updates-testing and create empty	10:21
warren	metadata there.	10:21
warren	6) Send out announcement.	10:21
warren	7) Push to mirrors.	10:22
warren	#1 is done	10:22
warren	releases and Everything is left alone for now.	10:22
nirik	0) make the faq/announcement and get it reviewed and looking ok?	10:22
warren	well, that's #6	10:22
warren	but yes	10:22
warren	oh, also don't forget unique	10:23
poelcat	how about sending out announce today referencing this list and our progress so people know how things are going and where we are at?	10:23
nirik	New_Signing_Key and Enabling_new_signing_key exist now, FYI.	10:24
warren	poelcat: we really have to know the status of the things we push before sending out the announcement	10:24
warren	the announcement must be succinct and not confusing	10:24
warren	If hughsie can fix PK, then the announcement is pretty much just "Update, then update again", fingerprints of the new keys and links to the FAQ.	10:25
f13	warren: he's asking for a status announcement, not /the/ announcement	10:26
warren	If hughsie can't fix PK in time, then the announcement has to mention "oh btw, this is broken, and no we don't have a fix for it yet, and it wont tell you when a fix is available."	10:26
f13	warren: akin to what I did on Fridayish	10:26
warren	oh	10:26
f13	poelcat: I can do that.	10:26
poelcat	cool :)	10:27
* poelcat thinks/hopes this will help people remain patient		10:27
warren	f13: should we point people at the fedora-release packages and say "please test this, although packagekit is known broken"	10:28
wwoods	it's not *that* broken	10:28
warren	f13: perhaps fedora-devel-announce can be pointed it, while fedora-announce-list not.	10:28
warren	wwoods: the key import part is broken	10:28
wwoods	the icon doesn't appear. you can still list updates, apply them, etc.	10:28
wwoods	oh original f9	10:28
warren	f13: still nobody has tested pirut	10:29
f13	warren: I"d rather not on a announcement that goes to fedora-announce-list	10:30
f13	warren: you can re-post to fedora-devel-list or fedora-test-list	10:30
warren	f13: that's what I said	10:30
warren	f13: fedora-devel-announce only	10:30
f13	meh, make another mail.	10:30
mbonnet	warren: nitpick: do we really need the .transition in the fedora-release Release? It looks...non-standard.	10:31
warren	mbonnet: bikeshedding	10:31
mbonnet	warren: user experience	10:31
warren	mbonnet: doesn't hurt, and it actually is less confusing to know exactly what state the user is in	10:31
warren	mbonnet: this actually is only a transitional package. soon later we will push the final into newrepo.	10:32
f13	yeah, it'll get replaced with a standard n-v-r in the new updates location	10:32
mbonnet	warren: *shrug* Just saying, as an otherwise clueless user, seeing a package named like that might raise questions.	10:32
mbonnet	but I don't really care that much	10:33
warren	f13: well, I was thinking to name the next one 9-5.newkey to again be explicitly clear. because otherwise 9-2 and 9-5, at a glance you don't know which key you have, and that is pretty important.	10:33
warren	are others really against this?	10:33
warren	again this is bikeshedding.	10:33
f13	anything else on updates?	10:34
warren	oh yes	10:35
warren	So do we have agreement on how we will handle the key revoking?	10:35
warren	We didn't exactly ask Panu to add that hack to rpm yet.	10:35
f13	I haven't focused on that, due to wanting to get phase1 actually working	10:35
warren	ok	10:35
warren	let's revisit this	10:35
warren	next	10:35
f13	next meeting perhaps	10:36
f13	I'd rather discuss it once phase1 is done	10:36
f13	ok, moving on.	10:37
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - F10 Beta		10:37
f13	after the previous schedule adjustments, we were supposed to freeze tomorrow (well late late tonight)	10:38
f13	but we were also supposed to have a working rawhide for the past week+	10:38
f13	that didn't happen, so I've put the question out there as to wether the freeze should happen either	10:38
warren	what were the things preventing working rawhide?	10:38
f13	broken libxml2, broken package tickling a broken createrepo, anaconda being broken via NetworkManager	10:39
wwoods	build problems, then mishaps using NM in stage1	10:39
f13	so for a few of the days we had no rawhide, a few of them we had rawhide, no images, but repodata for packages, then we had rawhide, repodata, images but not working, and today we actually have somewhat working images.	10:40
f13	there has definitely been progress	10:40
wwoods	yes we have progressed quite a bit, and now we are at.. square one	10:40
warren	we need a few days of square oe	10:40
warren	one	10:41
f13	that's my thought as well, but we have Thanksgiving to consider	10:41
warren	hughsie!	10:41
hughsie	sorry for the delay guys, been sorting out family stuff	10:42
f13	hughsie: that's OK	10:42
f13	lets put the beta talk aside for a moment	10:42
hughsie	okay, what's the problem?	10:43
f13	hughsie: so, it's been noticed that the panel icon seems to go away and not come back with 0.2.5-1	10:43
hughsie	is the process running?	10:43
f13	it doesn't seem to come back to indicate that there are updates availabl.e	10:43
wwoods	hughsie: yes	10:43
hughsie	gpk-update-icon that is	10:43
warren	hughsie: upgrade to 0.2.5-1 and relog or reboot, tray icon is forever gone	10:43
hughsie	gpk-update-icon --verbose says?	10:43
warren	hughsie: yes running	10:43
hughsie	can someone pastebin th eoutput pls, i'm running my laptop on my lap in a living room	10:43
warren	hughsie: output with the running pkg-update-icon or after I kill it?	10:44
wwoods	unable to locate theme engine nodoka.. cannot register existing type 'LibGBus'.. assertion 'initialization_value' != 0 failed	10:44
f13	ooops	10:44
wwoods	g_object_new: assertion `G_TYPE_IS_OBJECT (object type)' failed	10:44
hughsie	warren: killall gpk-update-icon && gpk-update-icon --verbose	10:44
warren	http://fpaste.org/paste/5838 (without killing it)	10:45
warren	http://fpaste.org/paste/5839 (after killing it)	10:45
hughsie	urgh, i see the problem, but i dont know why it's worked before	10:46
hughsie	libpackagekit uses LibGBus, as does gpk-update-icon	10:46
hughsie	and as we strip any of the non pk- prefixes from the library i figured we shoul dbe okay	10:47
hughsie	but LIBGBus must be part of the internal private ABI	10:47
hughsie	as it's registered in the type system	10:47
hughsie	warren: just on 0.2.5 ot 0.2.4 as well?	10:48
warren	hughsie: I think I only tested 0.2.5	10:50
warren	hughsie: actually, I saw icons in 0.2.4	10:50
hughsie	right, so something changed	10:50
hughsie	let me check	10:50
hughsie	bingo	10:51
hughsie	commit aae0eb74e9bd527fa6e0e73579bf77b1ba83b291	10:51
hughsie	Author: Richard Hughes <richard@hughsie.com>	10:51
hughsie	Date: Wed Aug 6 12:40:45 2008 +0100	10:51
hughsie	bugfix: link in a local copy of gbus now we are compiling libpackagekit with -export-dynamic	10:51
hughsie	want a new rpm to try?	10:51
f13	please	10:51
f13	*sigh*	10:52
warren	anyone installing a fresh F8? we really need to test pirut as well.	10:52
f13	my test is halted at downloading file lists, probably because I'm connected to a slow ass mirror.	10:52
f13	is there any way to force PK to move to the next mirror?	10:52
nirik	warren: f8 gold? or any f8?	10:52
warren	nirik: we should really test both on all archs, but I highy doubt it is broken	10:53
* nirik powers on a x86_64 f8 guest he had laying around.		10:53
wwoods	PK still doesn't tell you when you should restart.. or are we just lacking metadata for that?	10:54
hughsie	f13: no, yum should timeout, right?	10:54
hughsie	wwoods: on rawhide yes	10:54
wwoods	no, on F9	10:54
f13	hughsie: depending on how you're using yum, it could take a long time to timeout, but if it's a slow mirror, it's getting data, just extremely slowly	10:54
f13	like a bit a minute	10:54
hughsie	hmm, there's not a lot pk can do in that situation	10:55
hughsie	we just rely on yum to DTRT	10:55
nirik	warren: whats the link to that transitional fedora-release?	10:56
f13	hughsie: yum allows you to ^c while downloading and that triggers a mirror switch	10:56
hughsie	f13: about to build new rpm	10:56
notting	f13: send the signal by hand to the backend?	10:56
f13	since you can see the rate you're getting and make a decision.	10:56
f13	notting: what would be the signal?	10:57
LetoTo1	is someone planning a test kernel-xen rpm with the "x86_64 dom0 is working" patch set from a few days ago?	10:57
* nirik got it.		10:57
wwoods	LetoTo1: not really sure what that has to do with the rel-eng meeting	10:58
f13	LetoTo1: wrong chan.	10:58
LetoTo1	oops	10:58
notting	f13: sigint	10:58
LetoTo1	sorry	10:58
f13	notting: ah, I'm just not sure how to send that via pkcon	10:58
notting	f13: kill -INT <pid>	11:00
warren	http://kojipkgs.fedoraproject.org/packages/fedora-release/8/6.transition/noarch/	11:00
warren	http://kojipkgs.fedoraproject.org/packages/fedora-release/9/4.transition/noarch/	11:00
* nirik runs pruit		11:00
warren	f13: hughsie: do we have to worry about the old backend running after upgrade, or is that part well tested?	11:00
hughsie	warren: 0.2.x to 0.2.y is well tested	11:01
f13	warren: I'd like hughsie to give me a test to verify, but it seems to be working	11:01
hughsie	0.1.x to y is less well tested (worked for me just now)	11:01
warren	how long ago was 0.1.x?	11:01
hughsie	the F9 gold release	11:02
hughsie	very quickly became 0.2.x	11:02
* wwoods installs F8 x86_64.. for funtimes		11:02
warren	ok	11:03
warren	if we're confident about the backend part	11:03
f13	notting: yeah, that didn't help.	11:06
hughsie	warren, f13: http://koji.fedoraproject.org/koji/taskinfo?taskID=813930	11:06
hughsie	that should be a-ok	11:06
hughsie	apologies for the balls-up, things are a bit frantic here at the moment	11:07
f13	understandable.	11:07
f13	we'll keep testing with that, should be fine.	11:07
hughsie	thanks dude	11:07
f13	I still can't do a complete good test because I'm handed a barely functional mirror and there is no way I can find to make PK move to a different mirror	11:07
hughsie	if you need to grab me, ping me with email and i'll try and jump on	11:07
hughsie	f13: you can do killall packagekitd if you like	11:08
* nirik waits for the download to finish here... pile of packages.		11:08
f13	hughsie: I wanted to test without killing the daemon	11:08
hughsie	i usually end up disabling the mirror and using fedora directly as the mirror bits seem, well, interesting	11:08
f13	oh here it goes	11:09
nirik	pirut asked for the new key, imported and appears to be running the transaction now. All looks ok from what I can see.	11:09
hughsie	i have to go visit some family, but i'll be back in a hour or so	11:12
f13	success.	11:12
f13	hughsie: ok, cheers.	11:12
f13	alright, lets move back to beta talk.	11:13
* nirik sees the update boom on the NetworkManager thing.		11:13
warren	nirik: F8 update?	11:13
nirik	yeah, using pirut.	11:13
nirik	on x86_64.	11:14
warren	is hte NM thing described somewhere?	11:14
nirik	yeah, let me find the bug. It's a multiarch thing.	11:14
dgilmore	how bad was todays rawhide?	11:15
wwoods	it's fine.	11:15
wwoods	it installs and everything.	11:15
wwoods	oh wait. maybe it doesn't?	11:17
wwoods	it starts anaconda, anyway	11:17
notting	is there any sane way to slip less than a wee?	11:18
notting	week?	11:18
f13	notting: yes, I was just getting to that	11:19
f13	notting: we could freeze on Thursday instead of Tuesday, and keep everything else the same	11:20
f13	essentially it gives developers a couple more days of free builds before we lock things down.	11:20
f13	which, given the timing of things, is probably the best choice right now	11:21
warren	+1 to Thursday	11:21
notting	wfm	11:21
warren	btw, about the translators issues	11:22
nirik	humm... can't find the bug now. ;( Perhaps I should file a new one.	11:22
warren	do we like the suggested idea (who was it?) to accept any builds after the freeze for a few days with only translation fixes?	11:22
warren	nirik: so NM multilib in F8 repo is broken?	11:23
f13	warren: are they really necessary for beta?	11:23
f13	warren: don't we have a rather large period between beta and final freeze when translation updates can be built?	11:23
nirik	give me this: http://www.scrye.com/pastebin/109	11:23
warren	f13: I'm not exactly sure how it works normally	11:23
f13	warren: I don't think they are either.	11:23
f13	warren: I think they thought if it didn't go into beta, it wasn't going in at all	11:24
f13	which is just plain wrong.	11:24
warren	f13: yeah	11:24
warren	let's just go ahead with Thursday	11:24
warren	nirik's issue we need to look at though	11:24
warren	that sounds bad	11:24
f13	it happens... frequently	11:25
nirik	looks like it doesn't put the i386 version of NetworkManager in the transaction...	11:25
warren	is i386 in the x86_64 repo?	11:25
warren	(repo or tools problem?)	11:25
nirik	it's not in the x86_64 repo...	11:28
nirik	the newkey one.	11:28
nirik	humm. or is it. Let me look at another mirror.	11:28
f13	I wonder if it's just not being seen as multilib anymore by our algorithm	11:28
f13	notting: this is 8?	11:28
warren	nirik: oldkey F8 repo has both?	11:29
nirik	old one didn't have it either.	11:30
f13	nirik: the master has NetworkManager-glib-devel-0.7.0-0.6.9.svn3675.fc8.i386.rpm NetworkManager-glib-0.7.0-0.6.9.svn3675.fc8.i386.rpm and NetworkManager-devel-0.7.0-0.6.9.svn3675.fc8.i386.rpm	11:30
nirik	I have seen this problem from some people in #fedora. Nuking the .i386 version makes the upgrade work... I thought it was filed, but I can't find the bug now.	11:30
nirik	right. No NetworkManager.i386	11:31
f13	this is yet another of those "no longer multilib" updates	11:31
f13	we have to fix those forcefully in anaconda, I don't recall what the rpm magic is (if any) to do it automatically	11:31
wwoods	there isn't	11:31
wwoods	hence the anaconda magic	11:31
f13	nod	11:31
nirik	right. :(	11:31
f13	moral of the story, don't do that in an update	11:31
* f13 wonders if this is something we should bug panu about allowing		11:32
nirik	so, this is already the way it was, nothing to see here on the new updates, move along. ;)	11:32
wwoods	f13: it depends on arch-specific req/prov, which is in the new rpm	11:32
warren	ok, not a regression	11:32
warren	is the new PK build done?	11:32
wwoods	so.. I can't install fedora-release-8-5.transitiontest1	11:32
wwoods	on F8 GA	11:33
warren	huh?	11:33
warren	what error?	11:33
wwoods	Missing Dependency: /etc/redhat-release is needed by package initscripts	11:33
* wwoods confused		11:33
warren	fedora-release-8-5 and 5.transitiontest1 doesn't even change that as a dep	11:34
wwoods	I dunno, but F8 stock yum doesn't want to localinstall it	11:36
wwoods	the F8 transitional repo might need a newer yum	11:37
f13	the fuck?	11:37
f13	did F8 gold's yum not understand filedeps?	11:38
wwoods	I have no idea	11:38
f13	wwoods: you sure that you just didn't get the filelists repodata downloaded?	11:38
warren	wwoods: F8 stock yum wont install it, then it wont install fedora-release along with the new yum	11:38
f13	anywho, I'm going to post the suggestion of moving the freeze to Thursday to the list	11:39
f13	moving on	11:39
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - open floor		11:39
f13	jwb: ping	11:39
jwb	hey	11:39
jwb	so, small not surprising announcement	11:39
nirik	wwoods: you need release notes too?	11:39
jwb	i need to bow out of rel-eng for a while	11:39
jwb	EOF	11:40
wwoods	confirmed.. works fine after upgrading to yum 3.2.8-2	11:40
wwoods	bluh!	11:40
wwoods	so it looks like the f8 transitional repo will need to have that yum package	11:41
f13	wwoods: erm	11:41
f13	wwoods: please to be looking closer at your output and make sure the yum update to yum didn't pull down filelists.xml	11:41
f13	or flilelists.sqlite	11:41
wwoods	do you mean yum upgrading fedora-release?	11:41
f13	wwoods: I want to make sure it wasn't just a buggy with yum localinstall that fails to drag in the filelists	11:41
wwoods	I definitely made yum pull down filelists	11:42
f13	wwoods: when?	11:42
wwoods	maybe yum 3.2.7 just fails to do that on localinstalls	11:42
warren	wwoods: did you try to make a local repo with createrepo?	11:42
f13	wwoods: that's what I was wondering	11:42
wwoods	so maybe this will work normally	11:42
warren	wwoods: try a local repo with createrepo just to be sure	11:42
wwoods	warren: no, I didn't	11:42
f13	wwoods: the scenario I saw was "yum localinstall foo" -> fail -> yum update yum (pulls filelists.) -> yum localinstall foo -> works	11:42
wwoods	f13: doesn't have anything to do with filelists AFAIK	11:43
f13	hrm, that file is in /etc/ so it might be in primary.xml ?	11:43
wwoods	I did: yum localinstall (fail), yum upgrade (pulls filelists).. cancelled, yum localinstall (fail)	11:43
wwoods	yum upgrade yum, yum localinstall.. success	11:43
f13	k	11:43
wwoods	so it's probably a yum bug. lemme see if I can roll back to the original yum and test again	11:44
f13	wwoods: that would be good to know	11:46
f13	jwb: ok, I thought you had a bit more of something to talk about.	11:47
warren	hmm, PK build not done yet?	11:47
f13	jwb: do you need us to take care of anything for you, remove you from lists, or whatnot?	11:47
jwb	f13, the only thing i was on tap to do was moving the F9 spins from Beta to final	11:48
f13	k	11:48
jwb	f13, and no, i can stay on all the lists and whatnot. just no time for actually doing anything (which is about how it's been for the past few months anyway...)	11:48
f13	ok.	11:49
wwoods	confirmed.. worked fine when upgrading from a test repo	11:50
wwoods	so yeah, it's just a localinstall bug	11:50
warren	grr	11:50
f13	ok, *whew*	11:50
warren	ohh!!	11:50
warren	it is gnome-packagekit	11:50
f13	wwoods:	11:51
f13	<skvidal> f13: just for fun - though I doubt it	11:51
f13	<skvidal> run	11:51
f13	<skvidal> yum update /path/to/rpm	11:51
f13	<skvidal> and tell me me if it magically works	11:51
wwoods	I don't think that will work	11:51
f13	ok, well we're at nearly 2 hours of meeting by now, I think it's time to wrap it up	11:53
f13	testing continues to look good	11:53
warren	what testing do we want to finish before we go live?	11:54
f13	warren: it'd be interesting to do a minimal + PackageKit install and see what all other things the newest pk and gnome-pk will need to pull in	11:54
f13	if that hasn't been done already	11:54
warren	f13: according to folks last week on f-d-l PackageKit* gnome-packagekit and unique	11:55
f13	ok	11:55

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!