Latest revision as of 20:11, 12 July 2012

Attendees

People present (lines said)

jlaska (123)
adamw (45)
maxamillion (32)
wwoods (19)
kparal (15)
jskladan (3)
zodbot (3)
Viking-Ice (3)
Southern_Gentlem (2)

Regrets:

Agenda

Previous meeting follow-up

adamw to check-in with wwoods on tooling needs for priv esc. test
- Waiting for input from wwoods, will check-in again next week
maxamillion seeking input from the Mentors group for guidance on mentor responsibilities
- got 1 response so far, see below for continued discussion

Fedora 13 test status

Upcoming test milestones:

2010-03-18 - Test Beta 'Test Compose'
- Jkeating and dlehman working together to determine what packages to include in 'test compose' ... expected end of day today
2010-03-25 - Test Beta 'Release Candidate'

Upcoming test days:

2010-03-25 - Printing Test Day -- twaugh
2010-03-30 - SSSD by default Test Day -- jlaska
2010-04-01 - ABRT Test Day -- kparal

Wwoods noted it would be great if there was an SOP available so that direct involvement with QA wasn't required. Jlaska pointed to https://fedoraproject.org/wiki/QA/SOP_Test_Day_management

Updates Testing

There are some disparate efforts underway designed to improve the Updates testing workflow. Currently, the only defined process discusses using yum to enable updates-testing (see QA:Updates Testing). Jlaska asked the group for ideas on what tasks are needed to define a process by which testers can provide test feedback for updated packages.

Ideas included:

we need a policy/sop for the 'proventesters' group
we need a guide to providing updates-testing feedback for branched + released explaining what should be tested and how to give feedback
we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check
Improve documenting around using fedora-easy-karma

Open discussion - <Your topic here>

Adjust meeting time?

Kparal and jskladan asked whether the QA meeting time could adjust to keep the localtime the same, but change UTC. This would moving the meeting from 16:00 UTC to 15:00 UTC during spring+summer. No objections were made during the meeting, jlaska would raise the topic on the list as well.

Upcoming QA events

2010-01-21 - Pre-Alpha Rawhide Acceptance Test Plan #1
2010-01-28 - Pre-Alpha Rawhide Acceptance Test Plan #2
2010-02-04 - Pre-Alpha Rawhide Acceptance Test Plan #3
2010-02-04 - NFSv4 Test Day
2010-02-05 - Alpha Blocker Meeting (F13Alpha) #1 (recap)
2010-02-11 - Test Alpha 'Test Compose' (boot media testing)
2010-02-12 - Alpha Blocker Meeting (F13Alpha) #2 (recap)
2010-02-12 - Alpha Test Candidate verification (announcement)
2010-02-18 - Alpha Release Candidate verification
2010-02-18 - Color Management test day
2010-02-19 - Alpha Blocker Meeting (F13Alpha) #3 (recap)
2010-02-24 - Alpha Go/No-Go Meeting (20:00 EST)
2010-02-25 - Yum Langpack plugin test day
2010-03-10 - Pre-Beta Acceptance Test Plan #1
2010-03-12 - Beta Blocker Meeting (F13Beta) #1 (recap)
2010-03-18 - Gnome Disk Utility Storage Test Day
2010-03-18 - Test Beta 'Test Compose' DELAYED
2010-03-19 - Beta Blocker Meeting (F13Beta) #2 (recap)
WE ARE HERE
2010-03-25 - Test Beta Candidate
2010-03-25 - Printing Test Day
2010-03-26 - Beta Blocker Meeting (F13Beta) #3
2010-03-30 - SSSD by default Test Day
2010-03-31 - Beta Go/No-Go Meeting (20:00 EST)
2010-04-01 - ABRT Test Day

Action items

wwoods and adamw to discuss tooling needs for priv esc. test
maxamillion will work on second draft of provenpackagers proposal
adamw will provide a draft for the update test plan
jlaska to post meeting time change request to test@l.fp.org

IRC transcript

jlaska	#startmeeting Fedora QA Meeting	16:00
zodbot	Meeting started Mon Mar 22 16:00:17 2010 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot.	16:00
zodbot	Useful Commands: #action #agreed #halp #info #idea #link #topic.	16:00
jlaska	#meetingname fedora-qa	16:00
zodbot	The meeting name has been set to 'fedora-qa'	16:00
jlaska	#topic Gathering	16:00
* kparal appears		16:00
jlaska	alright folks, who do we have around for the QA meeting?	16:00
jlaska	kparal: jskladan howdy gang	16:01
adamw	morning	16:01
jlaska	adamw: heyo	16:01
* jskladan likes meeting time		16:01
* Southern_Gentlem		16:02
jlaska	Southern_Gentlem: hey there!	16:02
* jlaska waits a few more minutes		16:02
jlaska	Oxf13: said he'd be out for the meeting	16:05
jlaska	is wwoods around?	16:05
jskladan	haven't seen him (or at least he did not respond to my ping)	16:06
jlaska	okay, well ... let's get moving, I don't want to keep you folks longer than we need to	16:06
jlaska	I'll be walking through the proposed agenda at - http://lists.fedoraproject.org/pipermail/test/2010-March/089489.html	16:06
jlaska	#topic Previous meeting follow-up	16:06
jlaska	First up ...	16:07
jlaska	#info adamw to check-in with wwoods on tooling needs for priv esc. test	16:07
adamw	well, I sent an email	16:07
adamw	he hasn't replied yet :)	16:07
jlaska	I saw adam reached out on this, wwoods	adamw have you guys had a chance to talk further?	16:07
jlaska	okay ... I spoke to wwoods on Friday and he was putting a lot of brain cycles into depcheck ... I'll keep this on for follow-up next week	16:08
adamw	okay	16:08
jlaska	#action wwoods and adamw to discuss tooling needs for priv esc. test	16:08
jlaska	This next topic is related to our agenda today, so we don't need to dive too deep on it	16:09
jlaska	#info maxamillion seeking input from the Mentors group for guidance on mentor responsibilities	16:09
* jlaska seeing if maxamillion is around on #fedora-qa		16:09
maxamillion	sorry I'm late	16:10
jlaska	maxamillion: hey there, no worries	16:10
adamw	hey maxa	16:10
jlaska	maxamillion: just following up on your check-in with the Mentors group for guidance on mentor responsibilities	16:10
jlaska	I think this will touch on a larger topic I was hoping we could discuss later on, so we can move discussion to that point if you like	16:11
maxamillion	jlaska: lemme find the archive link .... I got 1 response that I'd like to put up for discussion	16:11
maxamillion	yeah, lets do that	16:11
jlaska	maxamillion: okay ...	16:12
jlaska	#info got 1 response and will discuss later during QA meeting	16:12
maxamillion	its probably going to be a long discussion so I think the "previous meeting follow-up" might not be the best spot to bring it up :)	16:12
jlaska	maxamillion: heh, okay :)	16:12
* Viking-Ice half in half out..		16:12
jlaska	Viking-Ice: howdy	16:13
jlaska	#topic F13 Test Status	16:13
jlaska	just a quick update on where things are with F13 Beta	16:13
jlaska	We had the pre-beta acceptance drop last week - https://fedoraproject.org/wiki/Test_Results:Fedora_13_Pre-Beta_Acceptance_Test_1	16:14
jlaska	nothing horribly broken detected from that test run	16:14
jlaska	#info Oxf13 is working with dlehman to identify an anaconda build to deliver for the Beta 'test compose'	16:14
jlaska	Once that's available ... we'll update rhe's announcement to the list	16:15
jlaska	Folks are encouraged to post test results to https://fedoraproject.org/wiki/Category:Fedora_13_Beta_TC_Test_Results	16:15
jlaska	in the upcoming test news front ...	16:15
jlaska	#info assuming positive test results - Beta 'release candidate' planned for this Thursday (2010-03-25)	16:16
jlaska	alongside the planned F13 testing, just a reminder on the test day front ...	16:16
jlaska	thanks to all for participating in the gnome-disk-utility test day last week (https://fedoraproject.org/wiki/Test_Day:2010-03-18_Palimpsest)	16:16
* wwoods here		16:16
* maxamillion downloads the latest nightly build to fire up some tests		16:17
jlaska	wwoods: hey there	16:17
jlaska	this week we have ...	16:17
adamw	yep, disk test day went well	16:17
adamw	i'll send a recap to the list soon	16:17
adamw	12 bug reports, 2 already fixed	16:17
jlaska	adamw: sweet, thanks ... I was really stoked at the improved UI	16:17
kparal	yes, it looks great	16:17
jlaska	kudos to davidz	16:17
jlaska	#info Thursday (2010-03-25) we have a Printing test day - https://fedoraproject.org/wiki/Test_Day:2010-03-25_Printing	16:18
jlaska	I don't think anyone from the QA team is involved in organizing this event? Mostly twaugh driving here?	16:18
wwoods	it'd be pretty cool if we had the Test Day SOPs so well-defined that QA involvement wasn't required to run a Test Day	16:19
* maxamillion would be willing to help but doesn't actually own a printer so thinks it would be a situational mismatch		16:20
jlaska	wwoods: sarcasm? :)	16:20
adamw	jlaska: yeah, I haven't done anything on it yet, and the page looks good	16:21
jlaska	the wiki looks fairly well defined for the printing event, I might just check-in with twaugh to see if they need anything from us	16:21
adamw	kudos to twaugh	16:21
jlaska	right on!	16:21
jlaska	alright, next week we have 2 test days ... hosted by QA	16:22
jlaska	#info 2010-03-30 - SSSD by Default - https://fedoraproject.org/wiki/Test_Day:2010-03-30_SSSDByDefault	16:22
jlaska	#info 2010-04-01 - ABRT - https://fedoraproject.org/wiki/Test_Day:2010-04-01_ABRT	16:22
wwoods	jlaska: no, not at all - is that already the case? because awesome	16:22
jlaska	Both have have a QA and devel pair established, so I believe they are moving forward nicely	16:23
jlaska	wwoods: right on ... pass it along ... https://fedoraproject.org/wiki/QA/SOP_Test_Day_management	16:23
jlaska	that's all I have for F-13 testing ...	16:23
jlaska	it's just full of test runs and test days	16:23
wwoods	oh that's awesome!	16:23
jlaska	did I miss anything?	16:23
wwoods	I'll send notes / make edits if there's anything missing. anyway, please continue	16:24
jlaska	wwoods: kudos to adamw for that	16:24
jlaska	okay ... moving on to next topic ...	16:24
jlaska	#topic Updates testing brainstorm	16:24
jlaska	I was hoping to spend time today discussing what we want (or think we need) out of an updates testing workflow	16:25
jlaska	so much of our time is spent testing the branched release	16:25
jlaska	but with the increased exposure of using bodhi for branched (and already released updates), I would like to start collecting ideas so we can document a roadmap/plan	16:26
jlaska	there are a lot of efforts underway here already ...	16:26
jlaska	kparal has an acceptance test plan in draft -- https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan	16:26
jlaska	maxamillion has been putting out feelers for defining the 'proventesters' group -- http://lists.fedoraproject.org/pipermail/test/2010-March/088980.html	16:27
jlaska	but right now ... what we document to our testers is https://fedoraproject.org/wiki/QA/Updates_Testing	16:27
adamw	so these are the things I think we need: a policy/sop for the 'proventesters' group, and a guide to providing updates-testing feedback for a) branched and b) stable releases, explaining what actually should be tested and how feedback should be given	16:27
maxamillion	and I got 1 reponse --> http://lists.fedoraproject.org/pipermail/mentors/2010-March/000177.html	16:27
maxamillion	response even ...	16:27
maxamillion	adamw: +1	16:28
jlaska	#info we need a policy/sop for the 'proventesters' group	16:28
jlaska	#info we need a guide to providing updates-testing feedback for branched + released explaining what should be tested and how to give feedback	16:28
kparal	we can leverage the fedora-easy-karma script	16:28
adamw	kparal: have a 'usage' message in it?	16:29
kparal	adamw: sorry?	16:29
jlaska	related to your 'how to give feedback' ... lmacken and wwoods have talked about providing a wiki structure for testers (and contributors) to expand on that provides test instructions on a per-package basis	16:29
adamw	kparal: just wondering how you meant to 'leverage' it	16:29
kparal	We can document how best to use it	16:29
kparal	and mention it in the guide in the first place :)	16:29
adamw	jlaska: that sounds rather like the 'how_to_debug' pages?	16:30
jlaska	adamw: similar, in that it's on the wiki and has a structure around the pages	16:30
jlaska	what does it mean to test an update?	16:30
adamw	okay	16:30
jlaska	does that mean you've run through a documented series of tests	16:31
jlaska	or just validated the listed bugs are resolved?	16:31
jlaska	or something else?	16:31
kparal	good point	16:31
adamw	yes, that's what I was concerned with	16:31
adamw	i think we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check	16:32
jlaska	adamw: great point ... so continue to lock down the details drafted in https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan ?	16:32
adamw	do you think we should work that up ourselves, or go back to fesco for guidance?	16:32
maxamillion	I think that's hard to write down because updates generally do one of two things: either it fixes a previous bug or adds an enhancement	16:33
jlaska	#info we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check	16:33
adamw	jlaska: sort of, but I think that's mostly focused on automated tests right now?	16:33
maxamillion	and in the event of a bug fix, you need to test that the bug is fixed but also test that there aren't any regressions	16:33
adamw	maxamillion: there can be multiple criteria for different types of updates, it doesn't need to be one-size-fits-all	16:33
jlaska	adamw: yeah, just the tests required to accept the update for additional testing	16:33
adamw	maxamillion: just needs to be clear enough that you can always tell what you should be testing for any given update	16:33
wwoods	adamw: that's the acceptance plan	16:34
maxamillion	adamw: true, maybe I just don't entirely understand how to capture that into a procedure	16:34
adamw	jlaska: right, i'm thinking more about what people should be looking for at the bodhi stage. we can certainly put the criteria into that document, though.	16:34
jlaska	adamw: I'm with ya	16:34
wwoods	acceptance plans are (kind of by definition) simple and fast	16:34
wwoods	which lends them to automation	16:34
adamw	wwoods: well, the point is 'acceptance into what'	16:34
wwoods	actual testing.	16:34
adamw	right	16:34
jlaska	what I'm hoping to drill out here is what happens after we've accepted the update	16:34
adamw	so either we expand the scope of that document or we have two	16:35
jlaska	wwoods: adamw: exactly, just what you guys are talking about	16:35
wwoods	the point of an Acceptance Test Plan is to quickly decide whether or not the thing is even testable	16:35
adamw	i don't think that question is super-important, though, the important thing is to nail down all the actual information	16:35
adamw	then we can look at how exactly to arrange it	16:35
jlaska	sure, let's get more ideas ...	16:35
wwoods	and once something passes the acceptance testing, it's accepted for actual testing, and then moves on to another (separate) test plan	16:35
adamw	wwoods: sure, so if you want it to stay as an acceptance plan, we need a separate document for what we're discussing now (defining what people should look at when deciding whether to mark an update as good or bad)	16:36
adamw	for now i was just thinking about hashing out the actual criteria on a mailing list thread	16:36
wwoods	in short: acceptance tests are designed to be the first-line test in a series of test plans. they're simple, very automatable, and not intended to be exhaustive.	16:36
adamw	we can write it down all fancy-like later =)	16:36
wwoods	I think you're missing the point I'm trying to make here	16:36
adamw	i may be. what is it?	16:36
jlaska	does it help for us to continue engaging in discussion to detail the different types of updates (e.g. https://fedoraproject.org/wiki/Stable_Release_Updates_Proposal) ?	16:37
* jlaska holds question until wwoods is done		16:37
wwoods	we need a separate document for what we're discussing now, because an Acceptance Test Plan is, by definition, not supposed to be the sole test plean	16:37
wwoods	err test plan	16:37
adamw	wwoods: sure. I got that. that's why I said that if we want it to be an acceptance plan, it can't cover this stuff and we need a new one.	16:37
jlaska	so does it help to think of this discussion as just a "Package Update test plan"	16:38
wwoods	an Acceptance Test Plan is always, by definition, the first in a series of test plans, so anything beyond the scope of "can this thing actually be tested properly" should be moved into subsequent plan(s)	16:38
adamw	if we're thinking in formal QA terms, sure, that sounds about right	16:38
kparal	I'm lost now. are we talking about acceptance test plan in general or about the work the proventesters will be doing (mainly critical path stuff)?	16:39
jlaska	I think we all agree on that point, but I don't want to get bogged down in what final document this takes form in	16:39
wwoods	the basic existence of an Acceptance Plan implies the intent to create more separate plans.	16:39
adamw	kparal: that's a good point, i think we're talking about both, and you're right that under the fesco-approved policy they'll be distinct	16:39
jlaska	kparal: I'd like ideas around what testing needs to occur for updates in general	16:39
maxamillion	ok .... so: AcceptancePlan -> UpdatePlan -> Testing -> Karma -> CheckedForStable -> Bodhi	16:39
maxamillion	?	16:39
wwoods	I just wanted to make sure we're all clear on this point, 'cuz it keeps coming up	16:40
wwoods	but yeah. point belabored.	16:40
adamw	maxamillion: er, the UpdatePlan is how you do the Testing, and how can it go from Karma to Bodhi? :)	16:40
maxamillion	adamw: yeah, basically that's the piece of the puzzle we need to sort out .... which is more or less the root of this conversation and what we need outlined for the ProvenTester group	16:40
adamw	erm, so can we reboot? do we want to talk about exactly what candidate updates should be tested for in this meeting, or do we just want to agree that's what we need to define and do it on the list?	16:40
jlaska	adamw: thank you, the later was my intent	16:41
maxamillion	adamw: at least that's how I understand it	16:41
jlaska	#info focus for this meeting - agree that's what we need to define and do it on the list	16:41
jlaska	so right now ... I've got that we need to ...	16:41
adamw	i'm happy to write up a draft to get the ball rolling on discussion for that	16:41
adamw	if anyone would like that	16:41
jlaska	create proventesters group	16:41
jlaska	maxamillion: you've been looking into supporting details around mentoring people into this group?	16:42
adamw	maxamillion: are we expecting another draft of the proventesters policy/sop proposal?	16:42
* jskladan has to go - see you around tomorrow, gang!		16:43
jlaska	jskladan: see you tomorrow :)	16:43
maxamillion	jlaska: yes, but I haven't been able to find much ... I only got one response to my query with the mentors and I didn't feel it was the information I was hoping for	16:43
maxamillion	adamw: yeah, I'm going to need to touch things up and write up a proposal for how the mentor program will be handled, but I worry that will spawn into a rather large topic of its own	16:44
adamw	oh fun! i like topics.	16:44
jlaska	as adamw pointed out, I think we can work the details of that out of the meeting ... unless there are specific roadblocks you needed to walk through here?	16:45
maxamillion	nope, none in particular	16:45
jlaska	alright ... so we've got defining 'proventesters' group, and drafting a policy/sop around detailing the group	16:46
adamw	so should we take an action item for maxamillion to provide the next draft of the provenpackagers end of things, and me to provide a draft for the update test plan end of things?	16:46
jlaska	what's after that?	16:46
* jlaska recommends excessive use of #action and #info		16:46
maxamillion	who has control of meetbot?	16:47
jlaska	I think anyone can add #action and #info tags, no?	16:47
Southern_Gentlem	yes	16:47
maxamillion	#action maxamillion will work on second draft of provenpackagers proposal	16:47
maxamillion	#action adamw will provide a draft for the update test plan	16:48
jlaska	okay, so I'm still trying to wrap my brain around the big picture ...	16:48
* maxamillion doesn't think meetbot picked those up ....		16:48
maxamillion	doesn't meetbot normally echo the command or something?	16:48
jlaska	not for those	16:48
maxamillion	oh ok	16:48
maxamillion	:)	16:48
jlaska	alright, so we have proventesters and an update test plan ... are those the only 2 things standing in the way of having a defining updates testing workflow?	16:49
jlaska	are there other things we need on the radar, perhaps further down the line?	16:49
jlaska	s/defining/defined/	16:49
maxamillion	not that I can think of ... but I'm sure there's something I missed from my inbox or I'm just simply out of the loop on	16:49
maxamillion	or maybe I'm not and that'd kick hind parts :)	16:49
adamw	i think that's enough to work on for now anyway	16:50
maxamillion	agreed	16:50
adamw	once we get those done there's some joining-up to do in the wiki but i can't think of anything else major	16:50
maxamillion	+1	16:50
jlaska	okay	16:50
maxamillion	I have to run, have to grab a bite to eat before a meeting	16:51
jlaska	adamw: I'll need to bug you after meeting to get a sense for hte scope of your test plan ... you might be knocking out a lot of stuff with just that	16:51
jlaska	maxamillion: okay, thanks for joining	16:51
maxamillion	jlaska: always happy to be here :)	16:51
jlaska	wwoods: kparal: any other points not raised that you'd like to consider as eventual tasks?	16:52
kparal	not currently	16:52
jlaska	wwoods: anything else on your radar for this topic?	16:54
jlaska	alright, in the interest of time ... let's move on to open-discussion	16:55
jlaska	#topic open discussion - change meeting time?	16:55
jlaska	once Europe changes their clocks for daylight savings, this meeting will be a bit late for kparal and jskladan I believe	16:56
kparal	well, 6 pm	16:56
jlaska	should I send out a http://whenisgood.net/ for a new meeting time	16:57
kparal	there is this proposal to change the meeting time one hour back during the summer time (15.00 UTC) and revert it back once the winter time comes (16.00 UTC)	16:57
kparal	that way the meeting time would be always the same for all of us	16:57
kparal	but it would require announcing changes every half a year	16:57
jlaska	UTC would change, but the localtime would remain the same	16:57
adamw	i don't really mind	16:58
kparal	jlaska: thanks for clarification	16:58
jlaska	adamw: so that puts it @ 8am pacific?	16:58
jlaska	alright, I'll follow-up to the list, but I don't have objections to that ... I don't think we're going to confuse lots of people with that. The UTC / DST change already confuses people	16:59
kparal	I think adamw is farthest in the west of all of us?	16:59
jlaska	#action jlaska to post meeting time change request to test@l.fp.org	17:00
jlaska	#topic open discussion - <your topic here>	17:00
Viking-Ice	Except for me since i'm on 0 GMT	17:00
adamw	yeah, 8am. 8am's fine	17:00
jlaska	Viking-Ice: oh true	17:00
jlaska	anything else not discuss that people would like to bring up?	17:00
Viking-Ice	+ check with 0xf13 think he has always had problem with the meeting time	17:00
jlaska	Viking-Ice: will do, thanks	17:01
jlaska	I'll close out the meeting in another minute, unless there are other topics raised	17:01
jlaska	alrighty ... that'll do it for today then	17:02
jlaska	thanks everyone, I'll follow-up with minutes to the list	17:02
jlaska	#endmeeting	17:03

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!

Search

QA/Meetings/20100322: Difference between revisions