ReleaseEngineering/Meetings/2009-feb-09

Fedora Release Engineering Meeting :: Monday 2009-02-09

Package Signing--sha256

• the createrepo stuff is already taken care of in createrepo, and in pungi upstream
• http://fedoraproject.org/wiki/Features/StrongerHashes
• need a plan around the SHA1SUM file
• set release note flag on associated bug

Automated QA

• f13 created a script that can watch a set of repos (specifically the updates repos), detect when they change, and allow for execution of <something>
  • this would allow us to react to a new updates push and run whatever tests the QA team comes up with
• wwoods working on RHTSlib stuff
• continue discussion on #fedora-qa

Signing Server

• trying to get test instance setup in PHX
• jwb to help out with package signing in the future

Mass Rebuild

• need to start on sooner rather than later
• waiting for several features to solidify

IRC Transcript

f13	ping: jwb wwoods warren spot lmacken notting rdieter poelcat	10:01
* lmacken is here		10:01
* warren here		10:01
rdieter	here	10:01
* jwb here		10:01
* wwoods here		10:02
f13	probably enough to get started	10:03
-!- f13 changed the topic of #fedora-meeting to: Fedora Releng - Alpha release wrapup		10:03
f13	So Alpha went out last week	10:04
f13	It was a very quiet and smooth release day.	10:04
f13	and from what I've seen, has been received fairly well	10:04
jwb	rock on	10:04
f13	as promised, I wrote a lot of what I was doing down locally, and I need to slap it into a wiki page. It's going to be rough and will need some love, but it'll be something	10:05
f13	Any other thoughts on the Alpha?	10:06
f13	deep thinkers (:	10:06
jwb	busy worrying about Beta	10:06
f13	heh	10:07
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Sha 256 sums		10:07
f13	http://fedoraproject.org/wiki/Features/StrongerHashes	10:07
f13	this hits releng in a couple places	10:07
f13	the hashes we produce with createrepo, and the SHA1SUM files we currently use for isos	10:07
* poelcat here		10:08
f13	the createrepo stuff is already taken care of in createrepo, and in pungi upstream	10:08
f13	the SHA1SUM file is a bit stickier. Obviously the file name is going to have to change	10:08
f13	and it wasn't too long ago that this file was MD5SUMS	10:08
jwb	HASHFILE	10:08
f13	the thought this time is to name it more generically so that we can avoid future issues	10:08
f13	When i posted about it onlist, there was expressed some desire to have the filename also include information about what's inside of it	10:09
f13	I'm not so sure that I can do this cleanly/easily	10:09
jwb	it seemed like a reasonable, if whiny, request	10:09
f13	yeah, the question becomes, how do I refer to the Fedora Fedora spin i386 CDs and DVD and netinst ?	10:10
f13	oh and insert an 11 in there somewhere.	10:10
f13	maybe just Fedora-11-Beta-i386-isos-HASHSUMS	10:11
f13	or	10:11
f13	Fedora-11-Beta-i386-isos-CHECKSUMS	10:11
jwb	SUMSBITCHES!	10:12
f13	(:	10:12
jwb	i like the CHECKSUMS one	10:12
f13	I think the latter is pretty easily done	10:12
f13	The live versions would be more like: Fedora-11-Beta-i686-Live{-$SPIN}-CHECKSUMS	10:13
f13	oh, lets drop the S off the end and just have it CHECKSUM	10:13
jwb	yeah	10:13
nim-nim	f13: as other said it would be much nicer if the checksum file format itself was something like	10:13
nim-nim	algo:sum file instead of	10:13
nim-nim	sum file	10:13
f13	nim-nim: I don't think it coudl then be used directly by sha256	10:13
f13	nim-nim: currently you can just do sha1sum -c <file>	10:14
f13	and ideally you'd be able to do sha256sum -c <file> as well	10:14
nim-nim	f13: well, there needs to be veryfymychecksumdamnit command that just takes any checksum file in and does the right thing	10:14
f13	nim-nim: I think "needs" is a bit strong there.	10:14
nim-nim	f13: otherwise it'll break again next format change	10:14
nim-nim	f13: highly desirable then :p	10:15
f13	then I'm sure patches will show up quickly	10:15
nim-nim	f13: sure :)	10:15
f13	writing our own software to validate isn't exactly the greatest ideal.	10:16
f13	idea.	10:16
f13	our files should be validatable on completely different distributions	10:16
f13	by standard tools	10:16
f13	but that's just my opinion	10:16
f13	and in the interest of getting this feature closer to 100%, I think I'll go the route of the CHECKSUM format and call it good	10:17
wwoods	I think the file could have a generic name so long as we specify what algorithm is used in the file itself	10:17
nim-nim	f13: you know just as well as me than having a fixed name but variable commands to check will jsut push the breakage into verify scripts	10:17
f13	wwoods: there could be a comment at the top of the file as to what algo is used	10:18
f13	that's not too hard to add to the code, a bit more difficult when creating the files manually like for the Live images	10:18
wwoods	yeah that'd work. plus a stupid little wrapper script to read that and then exec $whichever_sum_program	10:18
f13	(but that could just mean that the live tools should be generating them as well)	10:18
f13	wwoods: I'll leave that script up to somebody else.	10:19
f13	Any further thoughts on stronger hashes?	10:20
poelcat	add to the release notes?	10:21
f13	yeah, I mentioned above I'd poke the docs folks about this change	10:21
f13	and flag the bug itself that is open for this	10:21
f13	alright, moving along	10:22
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Automated QA		10:22
f13	I've made the first big step toward my promises at FUDCon	10:23
f13	I wrote up a script on Friday that can watch a set of repos (specifically the updates repos), detect when they change, and allow for execution of <something>	10:23
f13	this would allow us to react to a new updates push and run whatever tests the QA team comes up with	10:23
wwoods	yay!	10:24
f13	it's a pretty simple script and should work both inside and outside of PHX	10:24
f13	I've also decided that the best use of my time is going to be coming up with the harnesses for detecting changes vs writing the actual tests themselves	10:24
wwoods	definitely agree	10:24
wwoods	all you need to do is write - and document - the hook	10:25
f13	I'm also going to poke Infra today and see about a F10~ instance to (ab)use for these things	10:25
wwoods	so this will get run whenever *any* repo changes, or specifically updates-testing, or what?	10:25
f13	wwoods: the script I have now is specifically designed to monitor the updates repos	10:26
f13	f9-updates f9-updates-testing f10-updates f10-updates-testing	10:27
wwoods	the basic idea is that the autoqa git repo will have a directory for each 'hook'	10:27
f13	it's just going to run in daemon mode, with a cache of repomd.xml for each repo, and it'll just diff them at a regular interval	10:27
wwoods	and each dir will have a description of the expected environment / arguments for the scripts	10:27
wwoods	and other folks can write scripts for that	10:27
f13	I haven't written anything yet in the script for once it detects a change	10:28
wwoods	so in this case we've got a post-repo-update hook	10:28
wwoods	so your harness could just assume that there's a dir full of scripts and run each of them with the same arguments and dump their output somewhere	10:28
wwoods	probably I need to check on the RHTSlib stuff to see what standards we have for output and return values and such	10:29
f13	yeah	10:29
f13	I wasn't sure if we wanted the harness to handle the output, or each individual test, or if there was a top level script we coudl call from the harness that would handle all that	10:29
wwoods	but yeah, please document the args and environment - like, say, maybe the scripts will get the path to an nfs-mounted copy of the repo?	10:29
f13	say if there was a top level 'run-tests' script for the post-repo directory	10:29
f13	the harness would call it, and wait for it to finish, that script would inturn call all the sub-scripts and bundle the results.	10:30
f13	wwoods: more like a http url	10:30
f13	or rather	10:30
wwoods	hm. more likely we'll have a top-level 'run-tests' script that will be called as: run-tests $hookname $arg $arg ...	10:30
f13	they'll get a URI that is yum compatible?	10:30
f13	ok.	10:30
wwoods	and it'll just run each script in the $hookname dir with the rest of the given args	10:30
f13	this is something we could probably discuss in #fedora-qa	10:30
wwoods	whatever you like - just write up the definition so people know what to write scripts against	10:31
f13	cool	10:31
wwoods	I'll work on RHTSLib info and the run-tests harness	10:31
f13	cool.	10:31
f13	I'm pretty excited about this, I think we should be able to have tests running in a short number of days	10:32
wwoods	for the moment, just assume there's going to be a script named something like 'run-tests' that will act as I described it	10:32
f13	roger	10:32
f13	So I'm going to fine tune this script, and then explore other "buckets" that I can create a harness for easily	10:33
f13	Automated_QA_Testing_Project has a list of what we came up with at FUDCon	10:34
f13	I'll try to add links to sourcecode there as it gets checked into public repos	10:34
wwoods	so, wait, post-repo-update is only going to get the URL of the newly-changed repo?	10:34
wwoods	can you think of a way to also get the previous version of the repo, so we can run some diffs?	10:35
f13	that might be harder	10:35
f13	I think eventually it can be done, but I don't want to target that for version 0.1	10:35
wwoods	okay, I added a 'post-repo-update' dir to autoqa	10:36
f13	Eventually we could rely on a message bus and for bodhi to drop on the bus "Hey I just composed this tag, to this path" and our monitoring script would already have in its history the previous path for that tag	10:36
wwoods	please feel free to modify its README as you see fit. or just tell me how to change it.	10:36
f13	rgr	10:36
f13	Anything further on Auto QA?	10:37
f13	and releng's role in it?	10:37
f13	cool.	10:38
wwoods	actually	10:38
f13	oh?	10:38
wwoods	so one of the big goals that we've set for this year for fedora QA is making rawhide more usable	10:39
wwoods	I feel silly saying that because that's pretty much our goal *every* year	10:39
wwoods	but seriously. for real.	10:39
f13	we meanses it this time	10:39
wwoods	anyway we're taking a look at the rawhide process and trying to figure out good places to insert tests and ways to streamline it	10:39
wwoods	so people get fixes faster and there's less "oh rawhide is broken" time	10:40
wwoods	so this is a general heads-up.	10:40
f13	rawhide is my next 'bucket' to look at	10:40
wwoods	more specific requests will follow, I'm sure	10:40
wwoods	right now we're just trying to figure out stuff. like why mashing takes so damn long.	10:41
f13	but it's likely to be the same style as updates, watching a static location for repodata change and calling off a set of tests.	10:41
f13	wwoods: mash is far far far faster than it was months ago	10:41
f13	we're down to about 2~ hours to make rawhide	10:41
wwoods	right, but adamw points out that pretty much every other distro has their devel branch update as builds are made	10:42
wwoods	rather than nightly	10:42
f13	maybe 2:25 from cron start to bits on the master mirror.	10:42
f13	wwoods: how are they possibly keeping mirrors in sync?	10:42
f13	and do these other distros do multilib?	10:42
wwoods	it's 2:15 to make rawhide, and 1:45 of that is mash.	10:43
jwb	continuous update sounds wrong-ish	10:43
wwoods	the other :30 is, basically, pungi	10:43
jwb	our buildroots do that, but not the repo	10:43
f13	If we didn't have multilib, and we didn't purge older builds from the repos, we could have our rawhide repos update with new content in about 5 minutes after every build	10:44
wwoods	f13: I'm fuzzy on whether mandriva did multilib the way we think of it	10:44
f13	of course getting mirrors to be able to process our tree that fast would be a joke	10:44
wwoods	and as for mirrors, I think they had less trouble keeping mirrors in sync because they didn't have to sync it as far (or to as many) to get it to their testers	10:44
wwoods	i.e. I think it was feasible for their testers to all use the Master Mirror (or a mirror a single hop away)	10:45
wwoods	like I said, we're still analyzing stuff	10:45
f13	they also likely had far far less on the mirror	10:45
wwoods	also true	10:45
f13	our mirror tree is a bit... bloated	10:45
f13	as is our package set	10:45
wwoods	but then we don't really need to sync everything that was in Extras more than daily	10:45
f13	wwoods: so in theory, we do have repos that are continuously updated throughout the day. But it would mean forcing everybody to use PHX, and not having multilib	10:46
f13	wwoods: Extras is empty, but yes	10:46
wwoods	it's just old Core that would be most useful to have updating continuously	10:46
f13	but the more complicated you make our sync suggestions, the less likely it'll be that mirrors do it right.	10:46
f13	but it's good that you're looking into it	10:47
wwoods	everything worth doing is hard.	10:47
f13	I'll try to help with as much information as I can	10:47
wwoods	but then, plenty of things *not* worth doing are also hard	10:47
wwoods	so we gotta be sure we're doing the right thing.	10:47
wwoods	sometimes I feel like "not using multilib" isn't a huge problem	10:47
wwoods	but anyway	10:48
wwoods	research phase, actionable stuff to come, hold onto yr butts and brace for a lot of stupid questions	10:48
f13	sometimes I think "stop supporting multilib" would solve a /lot/ of our problems.	10:48
f13	well, more than sometimes.	10:49
wwoods	faster RPM dependency solving, smaller repos, cut rawhide building and updates pushes down to a 30-minute process	10:49
f13	kill a ton of conflicts	10:49
* f13 sighs		10:50
f13	wwoods: thanks for the info, we'll brace ourselves for the questions and be open minded	10:50
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Signing Server		10:50
f13	so yeah, that magical signing server thing	10:50
jwb	uh huh	10:51
f13	I've asked jwb to help out here and get a test instance or 2 setup in PHX so that we can further evaluate it in its natural enviroment.	10:51
jwb	yep	10:51
jwb	i need to file a ticket	10:51
f13	since I've been focused on alpha and QA, I'm going to try this delegation thing and see how that works out (:	10:51
jwb	was derailed by trying to learn about signing as it is today, and the fullfilelist thing	10:51
f13	nod	10:51
f13	that's been quite helpful too	10:51
jwb	will be when i get it done anyway :)	10:52
f13	heh	10:52
jwb	need to figure out the sudo ftpuser thing and that should be it	10:52
f13	I talked to mjcox about the usb gpg device he was using, but he told me it doesn't support keys as big as we need and it would be pretty slow	10:53
f13	so I think we'll just use software	10:53
f13	As jwb mentioned, I've also started training him on doing signings as they stand today	10:53
f13	It is just silly to rely upon a single person to do all that	10:53
f13	and I'm really confident that we can trust jwb with the F9/10 keys	10:54
f13	so with luck we'll have another signer in the near future for pushing updates	10:54
f13	anybody have any objections / thoughts on this?	10:55
f13	(would be nice to have notting here right now)	10:55
f13	I'll take silence as consent.	10:56
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Mass Rebuilds		10:56
f13	last topic of today	10:56
* wwoods consents!		10:56
f13	We've got potential for mass rebuilds on the horizon	10:56
rdieter	kudos to jwb, may ye not get drunk with power	10:56
f13	GCC hit rawhide, needs a bit of time to stew.	10:56
jwb	rdieter, thanks. promise i wont :)	10:56
f13	rpm for 256 checksums, I'm not sure the status of that	10:57
f13	rpm for font provides, close, but not there yet	10:57
f13	Feature freeze is coming up soon, we should get these rebuilds done prior to feature freeze	10:57
f13	which means we need to start them very soon	10:57
jwb	is it automated, or maintainer driven?	10:57
f13	since we're going to do /every/ package, we should probably try to automate it	10:58
f13	time is short and waiting sucks	10:59
jwb	aye	10:59
f13	so whomever is in FESCo, please start pushing hard to get some dates on these items	11:00
jwb	i'll propose we start monday, assuming all the pieces are in place?	11:01
f13	Sure, I'm not doing much next week (:	11:02
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Open Floor		11:04
f13	any open topics?	11:04
f13	(I noticed that poelcat filed a ticket looking for a new meeting minder)	11:04
poelcat	https://fedorahosted.org/rel-eng/ticket/1275	11:05
poelcat	:)	11:05
f13	alright, time to wrap up, lots to do!	11:11
f13	thanks all, and thanks poelcat for the time and effort you've continued to put into this team.	11:11

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!