From Fedora Project Wiki

< QA‎ | Meetings

(Updated notes)
(update updates-testing link)
 
Line 44: Line 44:
== Updates Testing ==
== Updates Testing ==


There are some disparate efforts underway designed to improve the Updates testing workflow.  Currently, the only defined process discusses using {{command|yum}} to enable ''updates-testing'' (see [[QA/Updates Testing]]).  Jlaska asked the group for ideas on what tasks are needed to define a process by which testers can provide test feedback for updated packages.
There are some disparate efforts underway designed to improve the Updates testing workflow.  Currently, the only defined process discusses using {{command|yum}} to enable ''updates-testing'' (see [[QA:Updates Testing]]).  Jlaska asked the group for ideas on what tasks are needed to define a process by which testers can provide test feedback for updated packages.


Ideas included:
Ideas included:

Latest revision as of 20:11, 12 July 2012

Attendees

People present (lines said)

  • jlaska (123)
  • adamw (45)
  • maxamillion (32)
  • wwoods (19)
  • kparal (15)
  • jskladan (3)
  • zodbot (3)
  • Viking-Ice (3)
  • Southern_Gentlem (2)

Regrets:

Agenda

Previous meeting follow-up

  1. adamw to check-in with wwoods on tooling needs for priv esc. test
    • Waiting for input from wwoods, will check-in again next week
  2. maxamillion seeking input from the Mentors group for guidance on mentor responsibilities
    • got 1 response so far, see below for continued discussion

Fedora 13 test status

Upcoming test milestones:

Upcoming test days:

Wwoods noted it would be great if there was an SOP available so that direct involvement with QA wasn't required. Jlaska pointed to https://fedoraproject.org/wiki/QA/SOP_Test_Day_management

Updates Testing

There are some disparate efforts underway designed to improve the Updates testing workflow. Currently, the only defined process discusses using yum to enable updates-testing (see QA:Updates Testing). Jlaska asked the group for ideas on what tasks are needed to define a process by which testers can provide test feedback for updated packages.

Ideas included:

  • we need a policy/sop for the 'proventesters' group
  • we need a guide to providing updates-testing feedback for branched + released explaining what should be tested and how to give feedback
  • we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check
  • Improve documenting around using fedora-easy-karma

Open discussion - <Your topic here>

Adjust meeting time?

Kparal and jskladan asked whether the QA meeting time could adjust to keep the localtime the same, but change UTC. This would moving the meeting from 16:00 UTC to 15:00 UTC during spring+summer. No objections were made during the meeting, jlaska would raise the topic on the list as well.

Upcoming QA events

Action items

  1. wwoods and adamw to discuss tooling needs for priv esc. test
  2. maxamillion will work on second draft of provenpackagers proposal
  3. adamw will provide a draft for the update test plan
  4. jlaska to post meeting time change request to test@l.fp.org

IRC transcript

jlaska #startmeeting Fedora QA Meeting 16:00
zodbot Meeting started Mon Mar 22 16:00:17 2010 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00
zodbot Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00
jlaska #meetingname fedora-qa 16:00
zodbot The meeting name has been set to 'fedora-qa' 16:00
jlaska #topic Gathering 16:00
* kparal appears 16:00
jlaska alright folks, who do we have around for the QA meeting? 16:00
jlaska kparal: jskladan howdy gang 16:01
adamw morning 16:01
jlaska adamw: heyo 16:01
* jskladan likes meeting time 16:01
* Southern_Gentlem 16:02
jlaska Southern_Gentlem: hey there! 16:02
* jlaska waits a few more minutes 16:02
jlaska Oxf13: said he'd be out for the meeting 16:05
jlaska is wwoods around? 16:05
jskladan haven't seen him (or at least he did not respond to my ping) 16:06
jlaska okay, well ... let's get moving, I don't want to keep you folks longer than we need to 16:06
jlaska I'll be walking through the proposed agenda at - http://lists.fedoraproject.org/pipermail/test/2010-March/089489.html 16:06
jlaska #topic Previous meeting follow-up 16:06
jlaska First up ... 16:07
jlaska #info adamw to check-in with wwoods on tooling needs for priv esc. test 16:07
adamw well, I sent an email 16:07
adamw he hasn't replied yet :) 16:07
jlaska I saw adam reached out on this, wwoods adamw have you guys had a chance to talk further? 16:07
jlaska okay ... I spoke to wwoods on Friday and he was putting a lot of brain cycles into depcheck ... I'll keep this on for follow-up next week 16:08
adamw okay 16:08
jlaska #action wwoods and adamw to discuss tooling needs for priv esc. test 16:08
jlaska This next topic is related to our agenda today, so we don't need to dive too deep on it 16:09
jlaska #info maxamillion seeking input from the Mentors group for guidance on mentor responsibilities 16:09
* jlaska seeing if maxamillion is around on #fedora-qa 16:09
maxamillion sorry I'm late 16:10
jlaska maxamillion: hey there, no worries 16:10
adamw hey maxa 16:10
jlaska maxamillion: just following up on your check-in with the Mentors group for guidance on mentor responsibilities 16:10
jlaska I think this will touch on a larger topic I was hoping we could discuss later on, so we can move discussion to that point if you like 16:11
maxamillion jlaska: lemme find the archive link .... I got 1 response that I'd like to put up for discussion 16:11
maxamillion yeah, lets do that 16:11
jlaska maxamillion: okay ... 16:12
jlaska #info got 1 response and will discuss later during QA meeting 16:12
maxamillion its probably going to be a long discussion so I think the "previous meeting follow-up" might not be the best spot to bring it up :) 16:12
jlaska maxamillion: heh, okay :) 16:12
* Viking-Ice half in half out.. 16:12
jlaska Viking-Ice: howdy 16:13
jlaska #topic F13 Test Status 16:13
jlaska just a quick update on where things are with F13 Beta 16:13
jlaska We had the pre-beta acceptance drop last week - https://fedoraproject.org/wiki/Test_Results:Fedora_13_Pre-Beta_Acceptance_Test_1 16:14
jlaska nothing horribly broken detected from that test run 16:14
jlaska #info Oxf13 is working with dlehman to identify an anaconda build to deliver for the Beta 'test compose' 16:14
jlaska Once that's available ... we'll update rhe's announcement to the list 16:15
jlaska Folks are encouraged to post test results to https://fedoraproject.org/wiki/Category:Fedora_13_Beta_TC_Test_Results 16:15
jlaska in the upcoming test news front ... 16:15
jlaska #info assuming positive test results - Beta 'release candidate' planned for this Thursday (2010-03-25) 16:16
jlaska alongside the planned F13 testing, just a reminder on the test day front ... 16:16
jlaska thanks to all for participating in the gnome-disk-utility test day last week (https://fedoraproject.org/wiki/Test_Day:2010-03-18_Palimpsest) 16:16
* wwoods here 16:16
* maxamillion downloads the latest nightly build to fire up some tests 16:17
jlaska wwoods: hey there 16:17
jlaska this week we have ... 16:17
adamw yep, disk test day went well 16:17
adamw i'll send a recap to the list soon 16:17
adamw 12 bug reports, 2 already fixed 16:17
jlaska adamw: sweet, thanks ... I was really stoked at the improved UI 16:17
kparal yes, it looks great 16:17
jlaska kudos to davidz 16:17
jlaska #info Thursday (2010-03-25) we have a Printing test day - https://fedoraproject.org/wiki/Test_Day:2010-03-25_Printing 16:18
jlaska I don't think anyone from the QA team is involved in organizing this event? Mostly twaugh driving here? 16:18
wwoods it'd be pretty cool if we had the Test Day SOPs so well-defined that QA involvement wasn't required to run a Test Day 16:19
* maxamillion would be willing to help but doesn't actually own a printer so thinks it would be a situational mismatch 16:20
jlaska wwoods: sarcasm? :) 16:20
adamw jlaska: yeah, I haven't done anything on it yet, and the page looks good 16:21
jlaska the wiki looks fairly well defined for the printing event, I might just check-in with twaugh to see if they need anything from us 16:21
adamw kudos to twaugh 16:21
jlaska right on! 16:21
jlaska alright, next week we have 2 test days ... hosted by QA 16:22
jlaska #info 2010-03-30 - SSSD by Default - https://fedoraproject.org/wiki/Test_Day:2010-03-30_SSSDByDefault 16:22
jlaska #info 2010-04-01 - ABRT - https://fedoraproject.org/wiki/Test_Day:2010-04-01_ABRT 16:22
wwoods jlaska: no, not at all - is that already the case? because awesome 16:22
jlaska Both have have a QA and devel pair established, so I believe they are moving forward nicely 16:23
jlaska wwoods: right on ... pass it along ... https://fedoraproject.org/wiki/QA/SOP_Test_Day_management 16:23
jlaska that's all I have for F-13 testing ... 16:23
jlaska it's just full of test runs and test days 16:23
wwoods oh that's awesome! 16:23
jlaska did I miss anything? 16:23
wwoods I'll send notes / make edits if there's anything missing. anyway, please continue 16:24
jlaska wwoods: kudos to adamw for that 16:24
jlaska okay ... moving on to next topic ... 16:24
jlaska #topic Updates testing brainstorm 16:24
jlaska I was hoping to spend time today discussing what we want (or think we need) out of an updates testing workflow 16:25
jlaska so much of our time is spent testing the branched release 16:25
jlaska but with the increased exposure of using bodhi for branched (and already released updates), I would like to start collecting ideas so we can document a roadmap/plan 16:26
jlaska there are a lot of efforts underway here already ... 16:26
jlaska kparal has an acceptance test plan in draft -- https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan 16:26
jlaska maxamillion has been putting out feelers for defining the 'proventesters' group -- http://lists.fedoraproject.org/pipermail/test/2010-March/088980.html 16:27
jlaska but right now ... what we document to our testers is https://fedoraproject.org/wiki/QA/Updates_Testing 16:27
adamw so these are the things I think we need: a policy/sop for the 'proventesters' group, and a guide to providing updates-testing feedback for a) branched and b) stable releases, explaining what actually should be tested and how feedback should be given 16:27
maxamillion and I got 1 reponse --> http://lists.fedoraproject.org/pipermail/mentors/2010-March/000177.html 16:27
maxamillion response even ... 16:27
maxamillion adamw: +1 16:28
jlaska #info we need a policy/sop for the 'proventesters' group 16:28
jlaska #info we need a guide to providing updates-testing feedback for branched + released explaining what should be tested and how to give feedback 16:28
kparal we can leverage the fedora-easy-karma script 16:28
adamw kparal: have a 'usage' message in it? 16:29
kparal adamw: sorry? 16:29
jlaska related to your 'how to give feedback' ... lmacken and wwoods have talked about providing a wiki structure for testers (and contributors) to expand on that provides test instructions on a per-package basis 16:29
adamw kparal: just wondering how you meant to 'leverage' it 16:29
kparal We can document how best to use it 16:29
kparal and mention it in the guide in the first place :) 16:29
adamw jlaska: that sounds rather like the 'how_to_debug' pages? 16:30
jlaska adamw: similar, in that it's on the wiki and has a structure around the pages 16:30
jlaska what does it mean to test an update? 16:30
adamw okay 16:30
jlaska does that mean you've run through a documented series of tests 16:31
jlaska or just validated the listed bugs are resolved? 16:31
jlaska or something else? 16:31
kparal good point 16:31
adamw yes, that's what I was concerned with 16:31
adamw i think we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check 16:32
jlaska adamw: great point ... so continue to lock down the details drafted in https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan ? 16:32
adamw do you think we should work that up ourselves, or go back to fesco for guidance? 16:32
maxamillion I think that's hard to write down because updates generally do one of two things: either it fixes a previous bug or adds an enhancement 16:33
jlaska #info we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check 16:33
adamw jlaska: sort of, but I think that's mostly focused on automated tests right now? 16:33
maxamillion and in the event of a bug fix, you need to test that the bug is fixed but also test that there aren't any regressions 16:33
adamw maxamillion: there can be multiple criteria for different types of updates, it doesn't need to be one-size-fits-all 16:33
jlaska adamw: yeah, just the tests required to accept the update for additional testing 16:33
adamw maxamillion: just needs to be clear enough that you can always tell what you should be testing for any given update 16:33
wwoods adamw: that's the *acceptance* plan 16:34
maxamillion adamw: true, maybe I just don't entirely understand how to capture that into a procedure 16:34
adamw jlaska: right, i'm thinking more about what people should be looking for at the bodhi stage. we can certainly put the criteria into that document, though. 16:34
jlaska adamw: I'm with ya 16:34
wwoods acceptance plans are (kind of by definition) simple and fast 16:34
wwoods which lends them to automation 16:34
adamw wwoods: well, the point is 'acceptance into what' 16:34
wwoods actual testing. 16:34
adamw right 16:34
jlaska what I'm hoping to drill out here is what happens after we've accepted the update 16:34
adamw so either we expand the scope of that document or we have two 16:35
jlaska wwoods: adamw: exactly, just what you guys are talking about 16:35
wwoods the point of an Acceptance Test Plan is to quickly decide whether or not the thing is even *testable* 16:35
adamw i don't think that question is super-important, though, the important thing is to nail down all the actual information 16:35
adamw then we can look at how exactly to arrange it 16:35
jlaska sure, let's get more ideas ... 16:35
wwoods and once something passes the acceptance testing, it's accepted for actual testing, and then moves on to another (separate) test plan 16:35
adamw wwoods: sure, so if you want it to stay as an acceptance plan, we need a separate document for what we're discussing now (defining what people should look at when deciding whether to mark an update as good or bad) 16:36
adamw for now i was just thinking about hashing out the actual criteria on a mailing list thread 16:36
wwoods in short: acceptance tests are designed to be the first-line test in a series of test plans. they're simple, very automatable, and not intended to be exhaustive. 16:36
adamw we can write it down all fancy-like later =) 16:36
wwoods I think you're missing the point I'm trying to make here 16:36
adamw i may be. what is it? 16:36
jlaska does it help for us to continue engaging in discussion to detail the different types of updates (e.g. https://fedoraproject.org/wiki/Stable_Release_Updates_Proposal) ? 16:37
* jlaska holds question until wwoods is done 16:37
wwoods we need a separate document for what we're discussing now, because an Acceptance Test Plan is, *by definition*, not supposed to be the sole test plean 16:37
wwoods err test plan 16:37
adamw wwoods: sure. I got that. that's why I said that if we want it to be an acceptance plan, it can't cover this stuff and we need a new one. 16:37
jlaska so does it help to think of this discussion as just a "Package Update test plan" 16:38
wwoods an Acceptance Test Plan is always, by definition, the first in a series of test plans, so anything beyond the scope of "can this thing actually be tested properly" should be moved into subsequent plan(s) 16:38
adamw if we're thinking in formal QA terms, sure, that sounds about right 16:38
kparal I'm lost now. are we talking about acceptance test plan in general or about the work the proventesters will be doing (mainly critical path stuff)? 16:39
jlaska I think we all agree on that point, but I don't want to get bogged down in what final document this takes form in 16:39
wwoods the basic existence of an Acceptance Plan implies the intent to create more separate plans. 16:39
adamw kparal: that's a good point, i think we're talking about both, and you're right that under the fesco-approved policy they'll be distinct 16:39
jlaska kparal: I'd like ideas around what testing needs to occur for updates in general 16:39
maxamillion ok .... so: AcceptancePlan -> UpdatePlan -> Testing -> Karma -> CheckedForStable -> Bodhi 16:39
maxamillion ? 16:39
wwoods I just wanted to make sure we're all clear on this point, 'cuz it keeps coming up 16:40
wwoods but yeah. point belabored. 16:40
adamw maxamillion: er, the UpdatePlan is how you do the Testing, and how can it go from Karma to Bodhi? :) 16:40
maxamillion adamw: yeah, basically that's the piece of the puzzle we need to sort out .... which is more or less the root of this conversation and what we need outlined for the ProvenTester group 16:40
adamw erm, so can we reboot? do we want to talk about exactly what candidate updates should be tested for in this meeting, or do we just want to agree that's what we need to define and do it on the list? 16:40
jlaska adamw: thank you, the later was my intent 16:41
maxamillion adamw: at least that's how I understand it 16:41
jlaska #info focus for this meeting - agree that's what we need to define and do it on the list 16:41
jlaska so right now ... I've got that we need to ... 16:41
adamw i'm happy to write up a draft to get the ball rolling on discussion for that 16:41
adamw if anyone would like that 16:41
jlaska create proventesters group 16:41
jlaska maxamillion: you've been looking into supporting details around mentoring people into this group? 16:42
adamw maxamillion: are we expecting another draft of the proventesters policy/sop proposal? 16:42
* jskladan has to go - see you around tomorrow, gang! 16:43
jlaska jskladan: see you tomorrow :) 16:43
maxamillion jlaska: yes, but I haven't been able to find much ... I only got one response to my query with the mentors and I didn't feel it was the information I was hoping for 16:43
maxamillion adamw: yeah, I'm going to need to touch things up and write up a proposal for how the mentor program will be handled, but I worry that will spawn into a rather large topic of its own 16:44
adamw oh fun! i like topics. 16:44
jlaska as adamw pointed out, I think we can work the details of that out of the meeting ... unless there are specific roadblocks you needed to walk through here? 16:45
maxamillion nope, none in particular 16:45
jlaska alright ... so we've got defining 'proventesters' group, and drafting a policy/sop around detailing the group 16:46
adamw so should we take an action item for maxamillion to provide the next draft of the provenpackagers end of things, and me to provide a draft for the update test plan end of things? 16:46
jlaska what's after that? 16:46
* jlaska recommends excessive use of #action and #info 16:46
maxamillion who has control of meetbot? 16:47
jlaska I think anyone can add #action and #info tags, no? 16:47
Southern_Gentlem yes 16:47
maxamillion #action maxamillion will work on second draft of provenpackagers proposal 16:47
maxamillion #action adamw will provide a draft for the update test plan 16:48
jlaska okay, so I'm still trying to wrap my brain around the big picture ... 16:48
* maxamillion doesn't think meetbot picked those up .... 16:48
maxamillion doesn't meetbot normally echo the command or something? 16:48
jlaska not for those 16:48
maxamillion oh ok 16:48
maxamillion :) 16:48
jlaska alright, so we have proventesters and an update test plan ... are those the only 2 things standing in the way of having a defining updates testing workflow? 16:49
jlaska are there other things we need on the radar, perhaps further down the line? 16:49
jlaska s/defining/defined/ 16:49
maxamillion not that I can think of ... but I'm sure there's something I missed from my inbox or I'm just simply out of the loop on 16:49
maxamillion or maybe I'm not and that'd kick hind parts :) 16:49
adamw i think that's enough to work on for now anyway 16:50
maxamillion agreed 16:50
adamw once we get those done there's some joining-up to do in the wiki but i can't think of anything else major 16:50
maxamillion +1 16:50
jlaska okay 16:50
maxamillion I have to run, have to grab a bite to eat before a meeting 16:51
jlaska adamw: I'll need to bug you after meeting to get a sense for hte scope of your test plan ... you might be knocking out a lot of stuff with just that 16:51
jlaska maxamillion: okay, thanks for joining 16:51
maxamillion jlaska: always happy to be here :) 16:51
jlaska wwoods: kparal: any other points not raised that you'd like to consider as eventual tasks? 16:52
kparal not currently 16:52
jlaska wwoods: anything else on your radar for this topic? 16:54
jlaska alright, in the interest of time ... let's move on to open-discussion 16:55
jlaska #topic open discussion - change meeting time? 16:55
jlaska once Europe changes their clocks for daylight savings, this meeting will be a bit late for kparal and jskladan I believe 16:56
kparal well, 6 pm 16:56
jlaska should I send out a http://whenisgood.net/ for a new meeting time 16:57
kparal there is this proposal to change the meeting time one hour back during the summer time (15.00 UTC) and revert it back once the winter time comes (16.00 UTC) 16:57
kparal that way the meeting time would be always the same for all of us 16:57
kparal but it would require announcing changes every half a year 16:57
jlaska UTC would change, but the localtime would remain the same 16:57
adamw i don't really mind 16:58
kparal jlaska: thanks for clarification 16:58
jlaska adamw: so that puts it @ 8am pacific? 16:58
jlaska alright, I'll follow-up to the list, but I don't have objections to that ... I don't think we're going to confuse lots of people with that. The UTC / DST change already confuses people 16:59
kparal I think adamw is farthest in the west of all of us? 16:59
jlaska #action jlaska to post meeting time change request to test@l.fp.org 17:00
jlaska #topic open discussion - <your topic here> 17:00
Viking-Ice Except for me since i'm on 0 GMT 17:00
adamw yeah, 8am. 8am's fine 17:00
jlaska Viking-Ice: oh true 17:00
jlaska anything else not discuss that people would like to bring up? 17:00
Viking-Ice + check with 0xf13 think he has always had problem with the meeting time 17:00
jlaska Viking-Ice: will do, thanks 17:01
jlaska I'll close out the meeting in another minute, unless there are other topics raised 17:01
jlaska alrighty ... that'll do it for today then 17:02
jlaska thanks everyone, I'll follow-up with minutes to the list 17:02
jlaska #endmeeting 17:03

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!

Retrieved from "https://fedoraproject.org/w/index.php?title=QA/Meetings/20100322&oldid=295295"