Latest revision as of 09:02, 18 September 2016

Attendees

People present (lines said):

jlaska (108)
wwoods (66)
kparal (23)
quaid (18)
maxamillion (12)
jskladan (2)
jeff_hann (2)
ianweller (1)
Viking-Ice (1)

Regrets:

Agenda

Previous meeting follow-up

No previous action items

Fedora 13 RC#3 test status

Upcoming test milestones:

2010-05-18 - Fedora 13 Final Go/No-Go Meeting (20:00 EST)

Current test activities:

Test_Results:Current_Installation_Test
- Outstanding test results (i386): 4
- Outstanding test results (x86_64): 1
Test_Results:Current_Desktop_Test
- Outstanding test results: 32 (8 tests * 4 desktops)
CommonBugs? - 14 bugs

Fedora 13 QA Retrospective

You've heard me talk about it many times, but time is running out to voice yours concerns and suggestions about Fedora 13 Testing. Be heard at Fedora_13_QA_Retrospective.

Summer of Code

Last week, kparal asked the group whether there was interest in QA Summer of code projects. Feedback on QA:Summer_of_Code_Ideas was requested by May 13, 2010.

Open discussion - <Your topic here>

Ideas to make QA sexy

Adam Miller expressed an interest in working with the Design team to come up with some swag we could award exceptional QA members with. The discussion led to the selection process for awarding swag. Some thoughts included:

a method for packagers/maintainers to recognize an exceptional QA effort (in triaging, testing or reporting bugs)
recognizing notable QA contributions based on several metrics (bugs reported, karma provided, bug triage, mailing list feedback, QA namespace wiki contributions).

James would update the Fedora_13_QA_Retrospective document with this new information, and encouraged others to do so.

AutoQA post-bodhi-update update

Will Woods information the team of his progress on writing a post-bodhi-update watcher script. The basic purpose of post-bodhi-update is to fire autoqa tests whenever a maintainer uses bodhi to request an update be moved into the testing or stable repos. Code is available in autoqa.git wwoods branch. Will reviewed several limitations with the watcher, and asked for feedback on resolving. Will expects to have watcher completed this week, and remaining tasks (autoqa hook code, test templates and hello-bodhi proof-of-concept test) completed next week.

Upcoming QA events

2010-01-21 - Pre-Alpha Rawhide Acceptance Test Plan #1
2010-01-28 - Pre-Alpha Rawhide Acceptance Test Plan #2
2010-02-04 - Pre-Alpha Rawhide Acceptance Test Plan #3
2010-02-04 - NFSv4 Test Day
2010-02-05 - Alpha Blocker Meeting (F13Alpha) #1 (recap)
2010-02-11 - Test Alpha 'Test Compose' (boot media testing)
2010-02-12 - Alpha Blocker Meeting (F13Alpha) #2 (recap)
2010-02-12 - Alpha Test Candidate verification (announcement)
2010-02-18 - Alpha Release Candidate verification
2010-02-18 - Color Management test day
2010-02-19 - Alpha Blocker Meeting (F13Alpha) #3 (recap)
2010-02-24 - Alpha Go/No-Go Meeting (20:00 EST)
2010-02-25 - Yum Langpack plugin test day
2010-03-10 - Pre-Beta Acceptance Test Plan #1
2010-03-12 - Beta Blocker Meeting (F13Beta) #1 (recap)
2010-03-18 - Gnome Disk Utility Storage Test Day
2010-03-18 - Test Beta 'Test Compose' DELAYED
2010-03-19 - Beta Blocker Meeting (F13Beta) #2 (recap)
2010-03-25 - Test Beta Candidate
2010-03-25 - Printing Test Day
2010-03-26 - Beta Blocker Meeting (F13Beta) #3
2010-03-30 - SSSD by default Test Day
2010-03-31 - Beta Go/No-Go Meeting (20:00 EST)
2010-04-01 - ABRT Test Day
2010-04-08 - Virtualization Test Day
2010-04-13 - Test_Day:2010-04-13_Nouveau
2010-04-14 - Test_Day:2010-04-14_Radeon
2010-04-15 - Test_Day:2010-04-15_Intel
2010-04-16 - Final Blocker Meeting (f13blocker) #1 (recap)
2010-04-16 - Pre-RC Branched Acceptance Test Plan - DELAYED (see RHBZ #581794)
2010-04-22 - Test_Day:2010-04-22_StorageFiltering
2010-04-23 - Final Blocker Meeting (f13blocker) #2 (recap)
2010-04-29 - Test_Day:2010-04-29_Preupgrade
2010-04-29 - Test 'Final' Test Compose
2010-04-30 - Final Blocker Meeting (f13blocker) #3 (recap)
2010-05-05 - Final Blocker Meeting (f13blocker) #4 MISSED
2010-05-06 - Test 'Final' RC
WE ARE HERE

Action items

jlaska and adamw to empty the CommonBugs? queue

IRC Transcript

jlaska	#startmeeting Fedora QA Meeting	15:00
zodbot	Meeting started Mon May 17 15:00:12 2010 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot.	15:00
zodbot	Useful Commands: #action #agreed #halp #info #idea #link #topic.	15:00
jlaska	#meetingname fedora-qa	15:00
zodbot	The meeting name has been set to 'fedora-qa'	15:00
jlaska	#topic Gathering crit-mass	15:00
* kparal jumps		15:01
jlaska	kparal: howdy	15:01
jlaska	anyone else lurking for the QA meeting?	15:01
* maxamillion is here-ish		15:01
* wwoods appears in a puff of smoke		15:02
maxamillion	we re-changed $dayjob meeting and now I have internets	15:02
jlaska	maxamillion: wwoods: heyo!	15:02
jlaska	adamw Viking-Ice robatino?	15:02
jlaska	I'll wait another minute or two before starting	15:02
* Viking-Ice rides in on ash cloud..		15:03
jlaska	Viking-Ice: heh	15:03
jlaska	okay let's get started	15:04
jlaska	#topic Previous meeting follow-up	15:04
jlaska	This is easy, I've got nothing listed from last week for follow-up	15:04
jlaska	#info no action items from last week	15:04
jlaska	anything I missed?	15:04
jlaska	well, if so ... raise it during open discussion	15:05
jlaska	#topic Fedora 13 RC#3 test status	15:05
jlaska	No doubt you've all read/heard about the 1 week slip	15:05
jlaska	#info Fedora 13 RC#3 is the current release candidate	15:06
jlaska	Test results available at http://fedoraproject.org/wiki/Category:Fedora_13_Final_RC_Test_Results	15:06
jlaska	assuming I'm counting correctly ... we have 5 tests remaining on the installation matrix (4 - i386, 1 - x86_64)	15:07
jlaska	The desktop matrix is empty, which means 32 outstanding tests	15:07
jlaska	I'm unclear whether test tests from RC#2 can apply against RC#3	15:07
jlaska	nonetheless, I'll try to pick off a few GNOME tests later today	15:08
jlaska	#info Because of all this testing, we have a queue of 14 CommonBugs? that need to be addressed	15:09
jlaska	#link https://bugzilla.redhat.com/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=CommonBugs%3F&sharer_id=141215	15:09
jlaska	#action jlaska and adamw to empty the CommonBugs? queue	15:09
* jeff_hann says hello		15:09
jlaska	jeff_hann: welcome :)	15:09
jlaska	that's all I have on my radar for F-13-RC3 testing	15:10
jeff_hann	:)	15:10
jlaska	bug#580226 was moved from CLOSED/ERRATA -> ASSIGNED earlier today. But the reporters problem was different from the failure originally reported.	15:10
jlaska	Any other issues that need to be on the radar?	15:11
jlaska	The go/no_go meeting is scheduled for tomorrow. However, there is no reason we can't give the "GO" sooner if test results ared completed and all good	15:11
jlaska	if no other F-13-RC3 issues, I'll move on ...	15:12
jlaska	okay, next topic	15:12
jlaska	#topic Fedora 13 QA Retrospective	15:12
jlaska	You've heard me talk about it, perhaps 5 million times already ...	15:13
jlaska	#info Please contribute your thoughts/concerns on Fedora 13 testing to Fedora_13_QA_Retrospective	15:13
jlaska	Once F-13 is behind us, I plan to organize the feedback and work with folks to come up with some tangible improvements for Fedora 14 Testing	15:14
* jskladan attends the meeting from bed :)		15:14
jlaska	jskladan: ouch :( welcome!	15:14
jlaska	okay everyone quiet now, no more talking about jskladan :)	15:15
jskladan	:-D i knew so :))	15:15
jlaska	hehe	15:15
jlaska	alright, next topic was a carry over from last week	15:15
jlaska	#topic QA Summer of Code	15:15
kparal	we already missed the deadline	15:16
jlaska	Last week, kparal put out a request for feedback for anyone with experience or interest in a QA summer of code	15:16
jlaska	kparal: the deadline was May 13, right?	15:16
kparal	right	15:16
jlaska	kparal: I take it no feedback received?	15:16
kparal	no	15:17
jlaska	booo :(	15:17
quaid	well, the students have until this week to propose	15:17
jlaska	well, if nothing else, we have a landing place for QA project ideas	15:17
jlaska	#link QA:Summer_of_Code_Ideas	15:18
quaid	so if there was a student to do the proposal, it can still be on any idea(s)	15:18
quaid	we are looking strong for a bigger, better run this Sep to Feb	15:18
quaid	(southern hemisphere summer)	15:18
kparal	well, we are short of mentors I'm afraid	15:18
jlaska	quaid: oh, so there's another round planned Sep - Feb	15:19
quaid	ok, you can watch how this summer goes and see if it helps you attract mentors	15:19
quaid	jlaska: yes	15:19
kparal	but we can take it as a test run this time, and have something better prepared for the next time	15:19
quaid	and I am working with RHT partner marketing people to put our proposal in front of many partners who may be interested in being sponsors, so more bigger betterness :)	15:19
jlaska	oooh	15:20
quaid	kparal: +1	15:20
kparal	quaid: maybe Dec - Feb?	15:20
quaid	that's about what the _whole_ project islike	15:20
quaid	kparal: for coding, yes	15:20
kparal	I see	15:20
quaid	I think the preparations -- open for ideas, etc. has to start in Sep so we aren't so rushed this time :)	15:20
quaid	we haven't worked on the schedule yet, though.	15:20
quaid	jlaska: so, if we have friends who work Fedora QA and work for companies that benefit from QA advancements ...	15:21
quaid	let's see if we can work that company as a sponsor from multiple angles.	15:21
* quaid cough s/390 cough		15:21
jlaska	quaid: hah! I think you coughed up a mainframe	15:22
jlaska	quaid: I can start asking around in that space and connect you with the right people?	15:22
jlaska	kparal: anything we need to track here, or anything that you'd like to see completed in advance of this next round?	15:23
quaid	sure	15:23
quaid	and I'll be talking through all the partner level at some point ~August	15:23
jlaska	perhaps I can complete the idea stubs I posted, but haven't fleshed out yet	15:23
quaid	good call	15:23
kparal	jlaska: I think that's too far ahead, to create an action point several months forward :)	15:24
jlaska	kparal: hah, no kidding :)	15:24
jlaska	okay then ... we'll drop this topic for now ... and hopefully pick this back up again in the end of the summer	15:24
* jlaska makes note		15:25
jlaska	kparal: quaid: thanks	15:25
jlaska	alright ... it's that time again folks	15:26
jlaska	#topic Open discussion - <Your topic here>	15:26
wwoods	the problem is making the QA tasks seem interesting, when QA is boring by design	15:26
wwoods	heh	15:26
jlaska	wwoods: exactly, I don't think we're looking for people to push buttons and run through pre-planned test execution	15:27
jlaska	this should be fun project ideas that will either grow, or facilitate our QA community in some way	15:27
wwoods	yeah obviously. there's some interesting stuff but none of it is sexy really	15:27
wwoods	that's the Curse of QA: the people are sexy, the code isn't	15:27
wwoods	it is our burden to bear.	15:27
jlaska	well, let's make it sexy!	15:27
wwoods	heh	15:28
jlaska	there's a blog title in the making here	15:28
wwoods	anyway, so the bodhi watcher	15:28
kparal	lets give away hot chicks instead of money as a reward	15:28
maxamillion	I've thought of ways to bring in more testers and I've talked a little with jlaska about it but now is a good time to bring it up	15:28
jlaska	kparal: I assume you're talking about this http://letthemeatlentils.files.wordpress.com/2009/12/20_kent_chicks.jpg	15:28
jlaska	#topic Open discussion - make QA sexy	15:28
kparal	jlaska: almost D:	15:28
maxamillion	I've thought about working with the design team to come up with "special" swag that we make and give away to testers who devote their time	15:29
wwoods	http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=hooks/post-bodhi-update/watch-bodhi-requests.py;hb=wwoods	15:29
wwoods	err oops	15:29
wwoods	disregard for the moment!	15:29
maxamillion	one thing I was thinking of is a modification to the Fedora splat t-shirt that says something like "construction crew" on the back	15:29
wwoods	maxamillion: hah nice	15:29
* jlaska notes wwoods is next in queue for the microphone		15:29
jlaska	maxamillion: do we have a way to gather data on bodhi karma feedback?	15:30
kparal	we should have some small rewards, yes	15:30
maxamillion	or maybe a t-shirt that has something to do with a fedora logo squishing a bug	15:30
maxamillion	jlaska: I'm not entirely sure	15:30
jlaska	#idea small rewards process for QA volunteers	15:30
wwoods	there's other measures of QA awesomeness	15:30
kparal	maybe show the most active people in some "top statistics page" of QA contributors?	15:31
jlaska	#idea work with design team for Fedora branded QA swag	15:31
* ianweller starts lurking at the sound of statistics		15:31
jlaska	wwoods: definitely, that was just one I've been missing recently	15:31
wwoods	I'd love it if we had some way for devs to hit a button for "the person who reported this bug did an awesome job"	15:31
maxamillion	ianweller: you have an unique fascination with statistics	15:31
maxamillion	wwoods: that'd be nice	15:31
jlaska	wwoods: ah, right ... so a way for maintainers to say "This person did a nice job"	15:31
kparal	collecting stars?	15:32
wwoods	yeah. I mean it's a little nervous-making because it introduces a gameable element	15:32
jlaska	#idea maintainer driver kudos -- a way for developers/maintainers to recognize a quality job	15:32
jlaska	wwoods: that's fine ... all stats are like that	15:32
wwoods	oh, true, good point	15:32
jlaska	which I think is why we'd need multiple vectors on this stuff	15:32
wwoods	anyway: In my view, doing a good job of actually tracing a bug to its cause	15:32
jlaska	bugs, QA wiki namespace contributions, mailing list assistance ...	15:32
jlaska	wwoods: right on	15:32
wwoods	is possibly the most valuable thing a QA contributor can do	15:33
jlaska	I think we've got some of this captured on the retrospective already ... so this is good	15:33
maxamillion	wwoods: I think that depends on the situation and what the bug is	15:33
wwoods	(note that "QA contributor" here is a slightly broader audience than, say, "QA developer", which includes those of us working on test frameworks and other craziness)	15:34
wwoods	maxamillion: well, yeah. I mean there's plenty of other very, very valuable contributions and not every bug can be traced like that	15:34
wwoods	but that is definitely a skill we want to encourage and reward	15:34
wwoods	if there's one thing I hear from developers constantly it's: "god I wish the people reporting these bugs could actually explain what was going on"	15:35
jlaska	I'll update the retrospective after this meeting with some of these ideas	15:35
maxamillion	wwoods: agreed	15:35
jlaska	but feel free to note things yourself (all)	15:35
* jlaska looking at a new wwoods entry around improving bug reporting... yay!		15:35
jlaska	#topic Open discussion - <your topic here>	15:36
jlaska	who is next?	15:36
wwoods	me!	15:36
jlaska	wwoods: autoqa?	15:36
jlaska	#topic Open discusssion - autoqa update from wwoods	15:36
wwoods	yeah, specifically the post-bodhi-update hook	15:36
wwoods	so here's that link again	15:36
jlaska	wwoods: take it away!	15:37
wwoods	http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=hooks/post-bodhi-update/watch-bodhi-requests.py;hb=wwoods	15:37
wwoods	that's the current post-bodhi-update watcher. It's not quite finished but it's 95% there	15:37
wwoods	The basic purpose of this thing is to fire autoqa tests whenever a maintainer uses bodhi to request an update be moved into the testing or stable repos	15:38
wwoods	unfortunately current bodhi has some limitations that we need to work around so there's a couple things to keep in mind	15:38
wwoods	this watcher can miss an update, if rel-eng pushes it into testing/stable before the watcher notices it	15:38
wwoods	but if rel-eng force-pushes the update without waiting for test results, they probably have a good reason for it	15:39
wwoods	so they'll be responsible for its safety in those cases anyway	15:39
* jlaska goes #info crazy		15:39
maxamillion	lol	15:39
jlaska	#info The basic purpose of [post-bodhi-update] is to fire autoqa tests whenever a maintainer uses bodhi to request an update be moved into the testing or stable repos	15:40
jlaska	#info Some limitations exist for the watcher (see minutes and link above for details)	15:40
wwoods	second problem: unfortunately critpath updates will appear without their target repo listed	15:40
wwoods	originally (as the comments say) I was going to just fire the tests anyway, and we'd just have to test against both sets of repos	15:41
kparal	wwoods: what about the future changes to bodhi that mean that all updates should be accepted "in sets"? will that require heavy changes in your script?	15:41
wwoods	kparal: not sure what you mean by "in sets", but it shouldn't require changes to the watcher	15:42
wwoods	the watcher's job is to notice when the contents of that set change	15:42
wwoods	the tests themselves can choose to either a) test all the updates in the set of proposed updates,	15:42
wwoods	or b) test the individual new update(s)	15:42
kparal	that sounds cool, I will have to check it out	15:43
wwoods	e.g. depcheck needs to test the entire set of proposed updates	15:43
kparal	same for package sanity	15:43
wwoods	right. but a simpler test that only needs to inspect the package	15:43
wwoods	e.g. a test that checked to make sure the package is signed with the correct key	15:43
wwoods	could just test the individual package	15:44
* jlaska would like that test		15:44
* jlaska <hearts> package signature test		15:44
kparal	so, when we can expect final release of the watcher? :)	15:44
wwoods	anyway, to finish my thought: critpath is a problem, and I'm not sure how to solve it	15:44
kparal	do we have to wait for some bodhi changes?	15:45
wwoods	there's three choices:	15:45
wwoods	1) fire test with target repo unset, let the tests decide what to do	15:45
wwoods	2) fire two tests, one for each target repo, let the users/testers decide which results they care about	15:46
wwoods	3) ignore the update until it has a target repo (i.e. when it gets the required karma), and then fire the test as normal	15:46
kparal	my questions is why it doesn't have target repo set from the beginning?	15:47
jlaska	is this a bodhi-1.0 design decision?	15:47
wwoods	long story short: because current bodhi is kind of dumb, and they're rewriting it, and that's gonna take a while	15:47
wwoods	so this watcher is similarly kind of dumb, and will be rewritten when bodhi2 appears	15:47
kparal	I don't like neither 1) nor 2), and I'm not sure how 3) works	15:47
wwoods	this is basically just supposed to be a "good enough" solution to get us something to work with until the stuff we need goes into bodhi 2	15:48
jlaska	As a proventester, I'd want to see these test results before applying my karma. So I'm not sure about option #3	15:48
kparal	especially about critpath packages, right	15:48
wwoods	kparal: we just keep a list of updates that appeared with 'request == None', and keep checking until they have a valid 'request' value	15:48
wwoods	yeah, that's the problem	15:48
wwoods	so basically, this watcher is supposed to be temporary so we just need to recognize and accept its limitations	15:49
jlaska	wwoods: is there an option that leaves the tests unchanged when we move to the new bodhi watcher?	15:49
wwoods	and try to choose whatever is the least bad option	15:49
wwoods	and work toward making it work correctly later	15:49
wwoods	yes, choice #2	15:49
wwoods	or #3, actually	15:49
wwoods	but I don't like delaying testing until after karma is applied	15:49
wwoods	which is why #2 was my proposed solution, even though I don't like it	15:50
jlaska	+1 on #2 -- noting your same regrets	15:50
wwoods	I think #2 is the least bad option	15:50
wwoods	if anyone has a better idea I'd love to hear it, but this is what we have to work with right now	15:51
wwoods	anyway given that decision, I've got a couple more things to finish up and we should have the first working version in a day or two	15:51
kparal	ok, great	15:51
wwoods	so give it the rest of the week to shake some bugs out, and I'd say we'll have a functioning hook next week	15:52
jlaska	wahooo!	15:52
wwoods	still need to write the autoqa hook code and the test templates	15:52
wwoods	and a hello-bodhi proof-of-concept test	15:52
wwoods	but yeah, end of next week is my guess	15:52
jlaska	Our watchers need to have icons associated with them. Post-bodhi-update would be a hydra	15:52
jlaska	wwoods: good stuff, thanks for the update	15:53
jlaska	anything else on autoqa post-bodhi-update?	15:53
jlaska	#topic Open discussion - <your topic here>	15:54
jlaska	anything not already covered?	15:54
jlaska	If not, I'll close out the meeting in 2 minutes	15:54
jlaska	30 seconds ...	15:56
* jlaska queues the Jeopardy theme music		15:56
jlaska	Okay gang, thanks for a great meeting!	15:56
jlaska	As always, I'll follow-up to the list with minutes	15:57
jlaska	feel free to raise additional topics not discuss there as well	15:57
jlaska	#endmeeting	15:57

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!

Search

QA/Meetings/20100517: Difference between revisions