From Fedora Project Wiki

< QA‎ | Meetings
Revision as of 16:52, 17 May 2010 by Jlaska (talk | contribs) (Add open discussion items)

Attendees

People present (lines said):

  1. jlaska (108)
  2. wwoods (66)
  3. kparal (23)
  4. quaid (18)
  5. maxamillion (12)
  6. jskladan (2)
  7. jeff_hann (2)
  8. ianweller (1)
  9. Viking-Ice (1)

Regrets:

Agenda

Previous meeting follow-up

  • No previous action items

Fedora 13 RC#3 test status

Upcoming test milestones:

  • 2010-05-18 - Fedora 13 Final Go/No-Go Meeting (20:00 EST)

Current test activities:

Fedora 13 QA Retrospective

You've heard me talk about it many times, but time is running out to voice yours concerns and suggestions about Fedora 13 Testing. Be heard at Fedora_13_QA_Retrospective.

Summer of Code

Last week, kparal asked the group whether there was interest in QA Summer of code projects. Feedback on QA:Summer_of_Code_Ideas was requested by May 13, 2010.

Open discussion - <Your topic here>

Ideas to make QA sexy

Adam Miller expressed an interest in working with the Design team to come up with some swag we could award exceptional QA members with. The discussion led to the selection process for awarding swag. Some thoughts included:

  • a method for packagers/maintainers to recognize an exceptional QA effort (in triaging, testing or reporting bugs)
  • recognizing notable QA contributions based on several metrics (bugs reported, karma provided, bug triage, mailing list feedback, QA namespace wiki contributions).

James would update the Fedora_13_QA_Retrospective document with this new information, and encouraged others to do so.

AutoQA post-bodhi-update update

Will Woods information the team of his progress on writing a post-bodhi-update watcher script. The basic purpose of post-bodhi-update is to fire autoqa tests whenever a maintainer uses bodhi to request an update be moved into the testing or stable repos. Code is available in autoqa.git wwoods branch. Will reviewed several limitations with the watcher, and asked for feedback on resolving. Will expects to have watcher completed this week, and remaining tasks (autoqa hook code, test templates and hello-bodhi proof-of-concept test) completed next week.

Upcoming QA events

Action items

  1. jlaska and adamw to empty the CommonBugs? queue

IRC Transcript

jlaska #startmeeting Fedora QA Meeting 15:00
zodbot Meeting started Mon May 17 15:00:12 2010 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00
zodbot Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00
jlaska #meetingname fedora-qa 15:00
zodbot The meeting name has been set to 'fedora-qa' 15:00
jlaska #topic Gathering crit-mass 15:00
* kparal jumps 15:01
jlaska kparal: howdy 15:01
jlaska anyone else lurking for the QA meeting? 15:01
* maxamillion is here-ish 15:01
* wwoods appears in a puff of smoke 15:02
maxamillion we re-changed $dayjob meeting and now I have internets 15:02
jlaska maxamillion: wwoods: heyo! 15:02
jlaska adamw Viking-Ice robatino? 15:02
jlaska I'll wait another minute or two before starting 15:02
* Viking-Ice rides in on ash cloud.. 15:03
jlaska Viking-Ice: heh 15:03
jlaska okay let's get started 15:04
jlaska #topic Previous meeting follow-up 15:04
jlaska This is easy, I've got nothing listed from last week for follow-up 15:04
jlaska #info no action items from last week 15:04
jlaska anything I missed? 15:04
jlaska well, if so ... raise it during open discussion 15:05
jlaska #topic Fedora 13 RC#3 test status 15:05
jlaska No doubt you've all read/heard about the 1 week slip 15:05
jlaska #info Fedora 13 RC#3 is the current release candidate 15:06
jlaska Test results available at http://fedoraproject.org/wiki/Category:Fedora_13_Final_RC_Test_Results 15:06
jlaska assuming I'm counting correctly ... we have 5 tests remaining on the installation matrix (4 - i386, 1 - x86_64) 15:07
jlaska The desktop matrix is empty, which means 32 outstanding tests 15:07
jlaska I'm unclear whether test tests from RC#2 can apply against RC#3 15:07
jlaska nonetheless, I'll try to pick off a few GNOME tests later today 15:08
jlaska #info Because of all this testing, we have a queue of 14 CommonBugs? that need to be addressed 15:09
jlaska #link https://bugzilla.redhat.com/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=CommonBugs%3F&sharer_id=141215 15:09
jlaska #action jlaska and adamw to empty the CommonBugs? queue 15:09
* jeff_hann says hello 15:09
jlaska jeff_hann: welcome :) 15:09
jlaska that's all I have on my radar for F-13-RC3 testing 15:10
jeff_hann :) 15:10
jlaska bug#580226 was moved from CLOSED/ERRATA -> ASSIGNED earlier today. But the reporters problem was different from the failure originally reported. 15:10
jlaska Any other issues that need to be on the radar? 15:11
jlaska The go/no_go meeting is scheduled for tomorrow. However, there is no reason we can't give the "GO" sooner if test results ared completed and all good 15:11
jlaska if no other F-13-RC3 issues, I'll move on ... 15:12
jlaska okay, next topic 15:12
jlaska #topic Fedora 13 QA Retrospective 15:12
jlaska You've heard me talk about it, perhaps 5 million times already ... 15:13
jlaska #info Please contribute your thoughts/concerns on Fedora 13 testing to https://fedoraproject.org/wiki/Fedora_13_QA_Retrospective 15:13
jlaska Once F-13 is behind us, I plan to organize the feedback and work with folks to come up with some tangible improvements for Fedora 14 Testing 15:14
* jskladan attends the meeting from bed :) 15:14
jlaska jskladan: ouch :( welcome! 15:14
jlaska okay everyone quiet now, no more talking about jskladan :) 15:15
jskladan :-D i knew so :)) 15:15
jlaska hehe 15:15
jlaska alright, next topic was a carry over from last week 15:15
jlaska #topic QA Summer of Code 15:15
kparal we already missed the deadline 15:16
jlaska Last week, kparal put out a request for feedback for anyone with experience or interest in a QA summer of code 15:16
jlaska kparal: the deadline was May 13, right? 15:16
kparal right 15:16
jlaska kparal: I take it no feedback received? 15:16
kparal no 15:17
jlaska booo :( 15:17
quaid well, the students have until this week to propose 15:17
jlaska well, if nothing else, we have a landing place for QA project ideas 15:17
jlaska #link https://fedoraproject.org/wiki/QA:Summer_of_Code_Ideas 15:18
quaid so if there was a student to do the proposal, it can still be on any idea(s) 15:18
quaid we are looking strong for a bigger, better run this Sep to Feb 15:18
quaid (southern hemisphere summer) 15:18
kparal well, we are short of mentors I'm afraid 15:18
jlaska quaid: oh, so there's another round planned Sep - Feb 15:19
quaid ok, you can watch how this summer goes and see if it helps you attract mentors 15:19
quaid jlaska: yes 15:19
kparal but we can take it as a test run this time, and have something better prepared for the next time 15:19
quaid and I am working with RHT partner marketing people to put our proposal in front of many partners who may be interested in being sponsors, so more bigger betterness :) 15:19
jlaska oooh 15:20
quaid kparal: +1 15:20
kparal quaid: maybe Dec - Feb? 15:20
quaid that's about what the _whole_ project islike 15:20
quaid kparal: for coding, yes 15:20
kparal I see 15:20
quaid I think the preparations -- open for ideas, etc. has to start in Sep so we aren't so rushed this time :) 15:20
quaid we haven't worked on the schedule yet, though. 15:20
quaid jlaska: so, if we have friends who work Fedora QA and work for companies that benefit from QA advancements ... 15:21
quaid let's see if we can work that company as a sponsor from multiple angles. 15:21
* quaid *cough* s/390 *cough* 15:21
jlaska quaid: hah! I think you coughed up a mainframe 15:22
jlaska quaid: I can start asking around in that space and connect you with the right people? 15:22
jlaska kparal: anything we need to track here, or anything that you'd like to see completed in advance of this next round? 15:23
quaid sure 15:23
quaid and I'll be talking through all the partner level at some point ~August 15:23
jlaska perhaps I can complete the idea stubs I posted, but haven't fleshed out yet 15:23
quaid good call 15:23
kparal jlaska: I think that's too far ahead, to create an action point several months forward :) 15:24
jlaska kparal: hah, no kidding :) 15:24
jlaska okay then ... we'll drop this topic for now ... and hopefully pick this back up again in the end of the summer 15:24
* jlaska makes note 15:25
jlaska kparal: quaid: thanks 15:25
jlaska alright ... it's that time again folks 15:26
jlaska #topic Open discussion - <Your topic here> 15:26
wwoods the problem is making the QA tasks seem interesting, when QA is boring by design 15:26
wwoods heh 15:26
jlaska wwoods: exactly, I don't think we're looking for people to push buttons and run through pre-planned test execution 15:27
jlaska this should be fun project ideas that will either grow, or facilitate our QA community in some way 15:27
wwoods yeah obviously. there's some interesting stuff but none of it is *sexy* really 15:27
wwoods that's the Curse of QA: the people are sexy, the code isn't 15:27
wwoods it is our burden to bear. 15:27
jlaska well, let's make it sexy! 15:27
wwoods heh 15:28
jlaska there's a blog title in the making here 15:28
wwoods anyway, so the bodhi watcher 15:28
kparal lets give away hot chicks instead of money as a reward 15:28
maxamillion I've thought of ways to bring in more testers and I've talked a little with jlaska about it but now is a good time to bring it up 15:28
jlaska kparal: I assume you're talking about this http://letthemeatlentils.files.wordpress.com/2009/12/20_kent_chicks.jpg 15:28
jlaska #topic Open discussion - make QA sexy 15:28
kparal jlaska: almost D: 15:28
maxamillion I've thought about working with the design team to come up with "special" swag that we make and give away to testers who devote their time 15:29
wwoods http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=hooks/post-bodhi-update/watch-bodhi-requests.py;hb=wwoods 15:29
wwoods err oops 15:29
wwoods disregard for the moment! 15:29
maxamillion one thing I was thinking of is a modification to the Fedora splat t-shirt that says something like "construction crew" on the back 15:29
wwoods maxamillion: hah nice 15:29
* jlaska notes wwoods is next in queue for the microphone 15:29
jlaska maxamillion: do we have a way to gather data on bodhi karma feedback? 15:30
kparal we should have some small rewards, yes 15:30
maxamillion or maybe a t-shirt that has something to do with a fedora logo squishing a bug 15:30
maxamillion jlaska: I'm not entirely sure 15:30
jlaska #idea small rewards process for QA volunteers 15:30
wwoods there's other measures of QA awesomeness 15:30
kparal maybe show the most active people in some "top statistics page" of QA contributors? 15:31
jlaska #idea work with design team for Fedora branded QA swag 15:31
* ianweller starts lurking at the sound of statistics 15:31
jlaska wwoods: definitely, that was just one I've been missing recently 15:31
wwoods I'd love it if we had some way for devs to hit a button for "the person who reported this bug did an awesome job" 15:31
maxamillion ianweller: you have an unique fascination with statistics 15:31
maxamillion wwoods: that'd be nice 15:31
jlaska wwoods: ah, right ... so a way for maintainers to say "This person did a nice job" 15:31
kparal collecting stars? 15:32
wwoods yeah. I mean it's a little nervous-making because it introduces a gameable element 15:32
jlaska #idea maintainer driver kudos -- a way for developers/maintainers to recognize a quality job 15:32
jlaska wwoods: that's fine ... all stats are like that 15:32
wwoods oh, true, good point 15:32
jlaska which I think is why we'd need multiple vectors on this stuff 15:32
wwoods anyway: In my view, doing a good job of actually tracing a bug to its cause 15:32
jlaska bugs, QA wiki namespace contributions, mailing list assistance ... 15:32
jlaska wwoods: right on 15:32
wwoods is possibly the most valuable thing a QA contributor can do 15:33
jlaska I think we've got some of this captured on the retrospective already ... so this is good 15:33
maxamillion wwoods: I think that depends on the situation and what the bug is 15:33
wwoods (note that "QA contributor" here is a slightly broader audience than, say, "QA developer", which includes those of us working on test frameworks and other craziness) 15:34
wwoods maxamillion: well, yeah. I mean there's plenty of other very, very valuable contributions and not every bug can be traced like that 15:34
wwoods but that is definitely a skill we want to encourage and reward 15:34
wwoods if there's one thing I hear from developers *constantly* it's: "god I wish the people reporting these bugs could actually explain what was going on" 15:35
jlaska I'll update the retrospective after this meeting with some of these ideas 15:35
maxamillion wwoods: agreed 15:35
jlaska but feel free to note things yourself (all) 15:35
* jlaska looking at a new wwoods entry around improving bug reporting... yay! 15:35
jlaska #topic Open discussion - <your topic here> 15:36
jlaska who is next? 15:36
wwoods me! 15:36
jlaska wwoods: autoqa? 15:36
jlaska #topic Open discusssion - autoqa update from wwoods 15:36
wwoods yeah, specifically the post-bodhi-update hook 15:36
wwoods so here's that link again 15:36
jlaska wwoods: take it away! 15:37
wwoods http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=hooks/post-bodhi-update/watch-bodhi-requests.py;hb=wwoods 15:37
wwoods that's the current post-bodhi-update watcher. It's not quite finished but it's 95% there 15:37
wwoods The basic purpose of this thing is to fire autoqa tests whenever a maintainer uses bodhi to request an update be moved into the testing or stable repos 15:38
wwoods unfortunately current bodhi has some limitations that we need to work around so there's a couple things to keep in mind 15:38
wwoods this watcher *can* miss an update, if rel-eng pushes it into testing/stable before the watcher notices it 15:38
wwoods but if rel-eng force-pushes the update without waiting for test results, they probably have a good reason for it 15:39
wwoods so they'll be responsible for its safety in those cases anyway 15:39
* jlaska goes #info crazy 15:39
maxamillion lol 15:39
jlaska #info The basic purpose of [post-bodhi-update] is to fire autoqa tests whenever a maintainer uses bodhi to request an update be moved into the testing or stable repos 15:40
jlaska #info Some limitations exist for the watcher (see minutes and link above for details) 15:40
wwoods second problem: unfortunately critpath updates will appear *without* their target repo listed 15:40
wwoods originally (as the comments say) I was going to just fire the tests anyway, and we'd just have to test against both sets of repos 15:41
kparal wwoods: what about the future changes to bodhi that mean that all updates should be accepted "in sets"? will that require heavy changes in your script? 15:41
wwoods kparal: not sure what you mean by "in sets", but it shouldn't require changes to the watcher 15:42
wwoods the watcher's job is to notice when the contents of that set change 15:42
wwoods the tests themselves can choose to either a) test all the updates in the set of proposed updates, 15:42
wwoods or b) test the individual new update(s) 15:42
kparal that sounds cool, I will have to check it out 15:43
wwoods e.g. depcheck needs to test the entire set of proposed updates 15:43
kparal same for package sanity 15:43
wwoods right. but a simpler test that only needs to inspect the package 15:43
wwoods e.g. a test that checked to make sure the package is signed with the correct key 15:43
wwoods could just test the individual package 15:44
* jlaska would like that test 15:44
* jlaska <hearts> package signature test 15:44
kparal so, when we can expect final release of the watcher? :) 15:44
wwoods anyway, to finish my thought: critpath is a problem, and I'm not sure how to solve it 15:44
kparal do we have to wait for some bodhi changes? 15:45
wwoods there's three choices: 15:45
wwoods 1) fire test with target repo unset, let the tests decide what to do 15:45
wwoods 2) fire two tests, one for each target repo, let the users/testers decide which results they care about 15:46
wwoods 3) ignore the update until it has a target repo (i.e. when it gets the required karma), and then fire the test as normal 15:46
kparal my questions is why it doesn't have target repo set from the beginning? 15:47
jlaska is this a bodhi-1.0 design decision? 15:47
wwoods long story short: because current bodhi is kind of dumb, and they're rewriting it, and that's gonna take a while 15:47
wwoods so this watcher is similarly kind of dumb, and will be rewritten when bodhi2 appears 15:47
kparal I don't like neither 1) nor 2), and I'm not sure how 3) works 15:47
wwoods this is basically just supposed to be a "good enough" solution to get us something to work with until the stuff we need goes into bodhi 2 15:48
jlaska As a proventester, I'd want to see these test results before applying my karma. So I'm not sure about option #3 15:48
kparal especially about critpath packages, right 15:48
wwoods kparal: we just keep a list of updates that appeared with 'request == None', and keep checking until they have a valid 'request' value 15:48
wwoods yeah, that's the problem 15:48
wwoods so basically, this watcher is supposed to be temporary so we just need to recognize and accept its limitations 15:49
jlaska wwoods: is there an option that leaves the tests unchanged when we move to the new bodhi watcher? 15:49
wwoods and try to choose whatever is the least bad option 15:49
wwoods and work toward making it work correctly later 15:49
wwoods yes, choice #2 15:49
wwoods or #3, actually 15:49
wwoods but I don't like delaying testing until after karma is applied 15:49
wwoods which is why #2 was my proposed solution, even though I don't like it 15:50
jlaska +1 on #2 -- noting your same regrets 15:50
wwoods I think #2 is the least bad option 15:50
wwoods if anyone has a better idea I'd love to hear it, but this is what we have to work with right now 15:51
wwoods anyway given that decision, I've got a couple more things to finish up and we should have the first working version in a day or two 15:51
kparal ok, great 15:51
wwoods so give it the rest of the week to shake some bugs out, and I'd say we'll have a functioning hook next week 15:52
jlaska wahooo! 15:52
wwoods still need to write the autoqa hook code and the test templates 15:52
wwoods and a hello-bodhi proof-of-concept test 15:52
wwoods but yeah, end of next week is my guess 15:52
jlaska Our watchers need to have icons associated with them. Post-bodhi-update would be a hydra 15:52
jlaska wwoods: good stuff, thanks for the update 15:53
jlaska anything else on autoqa post-bodhi-update? 15:53
jlaska #topic Open discussion - <your topic here> 15:54
jlaska anything not already covered? 15:54
jlaska If not, I'll close out the meeting in 2 minutes 15:54
jlaska 30 seconds ... 15:56
* jlaska queues the Jeopardy theme music 15:56
jlaska Okay gang, thanks for a great meeting! 15:56
jlaska As always, I'll follow-up to the list with minutes 15:57
jlaska feel free to raise additional topics not discuss there as well 15:57
jlaska #endmeeting 15:57

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!

Retrieved from "https://fedoraproject.org/w/index.php?title=QA/Meetings/20100517&oldid=174415"