QA/Meetings/20090603

Attendees

• Adam Williamson (adamw)
• Will Woods (wwoods)
• Jóhann Guðmundsson (viking_ice)
• John Poelstra (poelcat)
• James Laska (jlaska)
• Jesse Keating (f13)
• Paul Frields (stickster)

Agenda

Previous meeting follow-up

1. [adamw] - BugStatusWorkFlow - ask bugzilla guys to add a link from the bugzilla fields description page to the wiki page, for fedora
   • Completed (see RHBZ #495985), waiting for bugzilla code update during next outage window for changes to go live
2. [jlaska] - Send informal test day feedback survey to fedora-test-list and test day participants
   • Survey sent to participants and to fedora-test-list
3. [wwoods] - create preupgrade test case and update test matrix template
   • Test cases created (see QA:Testcase_Preupgrade and QA:Testcase_Preupgrade_from_older_release) and added to QA:Fedora_11_Install_Results_Template for any future test matrices
4. [jlaska] - create new RC1 test results page
   • Several pages created to track test results, including: QA:Fedora_11_RC0_Install_Test_Results, QA:Fedora_11_RC1_Install_Test_Results, QA:Fedora_11_RC2_Install_Test_Results, QA:Fedora_11_RC3_Install_Test_Results
5. [adamw] - create common bug entry for RHBZ #502077
   • Done and could probably be removed now (see Common_F11_bugs#865-hangs) since newer kernel resolves the problem

F-11-GA Preparation

• Blocker bugs (by component) - http://tinyurl.com/pqeq6n (currently 0 OPEN issues)
• Schedule - http://fedoraproject.org/wiki/Releases/11/Schedule
• Installation test results - QA:Fedora_11_RC4_Install_Test_Results

Adamw noted that the new test results wiki page wasn't created yet. Jlaska indicated it would be created soon after the meeting, once he received notification that RC4 composition had completed.

f13 indicated that he will continue to upload more RC4 bits as they are created (live images remain).

jlaska provided an update on RC test results. The RC3 results are looking fairly good (when combined with RC0, RC1, RC2). Once initial scrubbing of RC4 media kits has completed, jlaska asked if attention could be made to verify the 9 MODIFIED bugs on the F11 blocker list.

As indicated in the Releases/11/Schedule, staging will begin Thu Jun 4.

jlaska asked if RHBZ #503824 (anaconda-maint-list, NEW, x86_64 upgrade in KVM hangs (OOM) with 512MB RAM + encrypted root_) should be added to the common bugs page. Wwoods indicated that he would confirm on bare metal hardware first, and that this might be a candidate for common bugs. Further discussion began around whether Fedora has a minimal system requirements list. For additional comments, see open discussion below.

F-11 QA Post-mortem discussion

Jlaska indicated that a release-wide post-mortem review has been discussed for F11, and that he would like to begin the brainstorming around QA-specific topics. For example, what went well, and what needed improvement from a QA perspective.

jlaska had to step out, adamw lead the brainstorming.

Highlights (see IRC transcript for full details) ...

What worked well

1. Fedora Test Days
   • 20 test events during F11
   • Formalized creation of live images for test days QA/Test_Days/Live_Image
   • Improve presentation and increased library of test cases
2. Common_F11_bugs
   • More contributors and collaboration around existing wiki page
   • Consistent use of the common bug page as a resolution pathway for unresolved bugs and throughout mailing list communication
3. Release Candidate testing
   • Increased QA community involvement in testing release candidates (see Category:Fedora_11_Test_Results)
   • Community driven Delta ISO generation

What needs improvement

1. Blocker Bugs
   • Schedule & host earlier blocker bug list reviews
   • Improve messaging and guidelines around how to escalate a blocker bug for review
   • No workflow around verifying MODIFIED bugs
   • Large number ( of installation blocker bugs not reviewed prior to RC's
2. Fedora Test Days
   • No test day for sound during F11
   • Complex or tightly coupled features need to be scheduled later in the cycle when things have stabalized
   • No capacity concept ... QA would be interested in defining # of events capable of hosting
   • Create a self-hosting test day procedure for non-QA hosted test events
3. Release decisions
   • Clarify hand-off procedures between release engineering and quality assurance
   • Increase transparency around release slip meetings (or publicize minutes)
4. Automation
   • No distro-wide automated testing ... daily manual installation and repo testing
5. Metrics
   • No clear data for the health of the QA community, is it growing or shrinking?
   • Not clear how many testers contribute to each milestone (Alpha, Beta, Preview)

Open discussion

What are Fedora minimal requirements?

wwoods is seeing an OOM kill issue on a x86_64 KVM guest when upgrading from F10 -> F11 with only 512Mb of memory.

Everyone noted there isn't a clear definition of what minimal system requirements are for Fedora. Adamw directed folks to the bug he and Rahul filed RHBZ #499585.

Upcoming QA meetings/events

• 2009-06-09 - BugZappers/Triage_days
• 2009-06-10 16:00 UTC - Next QA Meeting QA/Meetings/20090610

Action items

• [stickster] - who will be handling release_notes bugs to help with RHBZ #499585
• [adamw] - propose draft wording of minimal requirements for the release_notes team to digest

IRC Transcript

--- | jlaska has changed the topic to: Fedora Quality Assurance Meeting | gathering		Jun 03 12:00
wwoods	| huzzah	Jun 03 12:00
jlaska	| wwoods: greetings	Jun 03 12:00
adamw	| hooray	Jun 03 12:01
jlaska	| QA meeting time, anyone around please clap your ends (against the keyboard)	Jun 03 12:01
jlaska	| adamw: heyo	Jun 03 12:01
Viking-Ice	| Here	Jun 03 12:01
jlaska	| Viking-Ice: hey there	Jun 03 12:01
jlaska	| I've got a hard stop in an hour today, so if we go over I'll need help from someone to drive	Jun 03 12:02
jlaska	| okay, getting started with last weeks items	Jun 03 12:02
adamw	| we'll figure something out	Jun 03 12:02
jlaska	| heh	Jun 03 12:02
--- | jlaska has changed the topic to: Fedora QA Meeting | previous week follow-up		Jun 03 12:03
jlaska	| okay, I'm walking the list ... QA/Meetings/20090603#Previous_meeting_follow-up	Jun 03 12:03
jlaska	| our lucky first up is ... <drumroll>	Jun 03 12:03
jlaska	| # [adamw] - BugStatusWorkFlow - ask bugzilla guys to add a link from the bugzilla fields description page to the wiki page, for fedora	Jun 03 12:03
adamw	| well, i did it	Jun 03 12:03
adamw	| and they say it's done	Jun 03 12:03
adamw	| i'm not actually seeing it on the page though	Jun 03 12:04
jlaska	| oh funky	Jun 03 12:04
adamw	| oh, dave says "Will show up in the next code update."	Jun 03 12:04
adamw	| so, presumably there hasn't been one o' those yet	Jun 03 12:04
jlaska	| ah right, during the next outage window	Jun 03 12:04
jlaska	| once or twice a month, I forget	Jun 03 12:04
jlaska	| there might be a staging/devel server if you wanted to test the changes out before the push?	Jun 03 12:05
adamw	| he didn't mention anything. it's a pretty straightforward change.	Jun 03 12:05
jlaska	| okay	Jun 03 12:06
jlaska	| any other updates or next-steps needed on that front?	Jun 03 12:06
adamw	| not really, it was a pretty simple thing :)	Jun 03 12:07
jlaska	| roger thanks	Jun 03 12:07
jlaska	| okay next up ...	Jun 03 12:07
jlaska	| # [jlaska] - Send informal test day feedback survey to fedora-test-list and test day participants	Jun 03 12:07
jlaska	| After 2 weeks of delaying I finally sent this out to the list (and bcc'd participants)	Jun 03 12:07
jlaska	| posted just this morning, have some feedback (list and direct mail) coming already ... but expect to see much more	Jun 03 12:08
adamw	| great	Jun 03 12:08
f13	| I'm here	Jun 03 12:08
jlaska	| adamw: gave a "shout out" to your planet post as well :)	Jun 03 12:08
jlaska	| f13: welcome	Jun 03 12:08
* | jlaska moving right along		Jun 03 12:09
jlaska	| next up ...	Jun 03 12:09
jlaska	| # [wwoods] - create preupgrade test case and update test matrix template	Jun 03 12:09
jlaska	| I think you had that done before the meeting finished last week?	Jun 03 12:09
jlaska	| wwoods: did we lose ya?	Jun 03 12:11
jlaska	| okay, I think that one is completed, but will revisit later if needed	Jun 03 12:12
jlaska	| # [jlaska] - create new RC1 test results page	Jun 03 12:12
wwoods	| sorry	Jun 03 12:12
jlaska	| np ... you've got the floor	Jun 03 12:12
wwoods	| yeah, we've got two preupgrade cases	Jun 03 12:12
wwoods	| and the matrix template and current results page are updated	Jun 03 12:12
wwoods	| QA:Fedora_11_RC3_Install_Test_Results#Upgrade_system	Jun 03 12:12
jlaska	| eggsellent, and I also saw you posting results+bugs for those too ... nice	Jun 03 12:13
jlaska	| any other things to track on that front?	Jun 03 12:13
wwoods	| not that I'm aware of	Jun 03 12:14
jlaska	| okay thanks	Jun 03 12:14
jlaska	| as you saw in wwoods post earlier, we have not just 1, but 3 RC results pages so far	Jun 03 12:14
jlaska	| with another on the way ... I'll create that once we get word that RC4 is live	Jun 03 12:15
jlaska	| okay last up ...	Jun 03 12:15
jlaska	| # [adamw] - create common bug entry for RHBZ #502077	Jun 03 12:15
jlaska	| adamw ... aka Mr. Common Bugs	Jun 03 12:15
adamw	| done, and has been outpaced by events	Jun 03 12:15
adamw	| Common_F11_bugs#865-hangs	Jun 03 12:15
adamw	| but i should take it out	Jun 03 12:15
adamw	| as the updated kernel fixed it and we're putting that in GA	Jun 03 12:15
jlaska	| oh we are?	Jun 03 12:16
f13	| jlaska: it's live, forgot to tell you	Jun 03 12:16
adamw	| so...it's gone	Jun 03 12:16
jlaska	| f13: I see partial images	Jun 03 12:16
jlaska	| adamw: ah okay, does this move to the fixed section ... or just dropped entirely?	Jun 03 12:16
adamw	| for now i'm just dropping them entirely	Jun 03 12:17
f13	| oh that's right, I made putting it on kojipkgs a higher priority for wwoods	Jun 03 12:17
wwoods	| sorry?	Jun 03 12:17
adamw	| for mdv i had separate 'modes' for pre-release and post-release for the common issues page	Jun 03 12:17
jlaska	| adamw: okay	Jun 03 12:17
adamw	| before release, i'd have a 'resolved issues' section which noted problems that had been fixed during the pre-release cycle, then that would be cleared out once we hit final release and the 'resolved issues' section would start over again	Jun 03 12:17
adamw	| i think that's a good model, but for f11 i'm just working in the 'final release' mode, we started too close to final release to bother with the pre-release mode i think	Jun 03 12:18
adamw	| so anything for which the fix goes into GA just gets taken out	Jun 03 12:18
jlaska	| adamw: are there steps we should consider now for the F12 common bugs process?	Jun 03 12:18
jlaska	| adamw: that seems fair	Jun 03 12:18
adamw	| not really, just create the page once we have something to write in it, copy all the boilerplate over from f11	Jun 03 12:19
adamw	| i'd probably do it around the time we release a first snapshot, but it can be done earlier if anyone wants to	Jun 03 12:19
* | jlaska not sure who owns it ... and when it's expected to land		Jun 03 12:20
jlaska	| but that can be a topic for another time	Jun 03 12:20
adamw	| yeah	Jun 03 12:20
jlaska	| okay that's it for last week ... in record time too! :)	Jun 03 12:20
--- | jlaska has changed the topic to: Fedora QA Meeting | F-11-GA Prep		Jun 03 12:20
jlaska	| didn't want to spend a lot of time here ... just reviewing the links and "who has the ball"	Jun 03 12:21
jlaska	| * Blocker bugs (by component) - http://tinyurl.com/pqeq6n (currently 0 OPEN issues)	Jun 03 12:21
jlaska	| * Schedule - http://fedoraproject.org/wiki/Releases/11/Schedule	Jun 03 12:21
jlaska	| * Installation test results - QA:Fedora_11_RC4_Install_Test_Results	Jun 03 12:21
jlaska	|	Jun 03 12:21
jlaska	| we're expecting RC4 any second now (looks like content just about uploaded to the double secret password location)	Jun 03 12:22
adamw	| the wiki page doesn't seem to exist yet	Jun 03 12:22
jlaska	| with that I'll get an RC4 page posted after the meeting to highlight any changes against RC3	Jun 03 12:22
jlaska	| adamw: right on, I'll pull that together after the meeting	Jun 03 12:22
f13	| I'll continue to upload more bits of RC4 as they are created, mostly the live stuff	Jun 03 12:23
f13	| but first I have to make jigdo of the other stuff.	Jun 03 12:23
jlaska	| f13: thanks ... I've got most of the content downloaded, except for x86_64+ppc DVD's	Jun 03 12:23
f13	| it really sucks not having enough disc space to do it all on one system before uploading :/	Jun 03 12:23
<-- | mcepl [n=mcepl@49-117-207-85.strcechy.adsl-llu.static.bluetone.cz] has left #fedora-meeting ( )		Jun 03 12:23
jlaska	| f13: as far as next steps ... the staging begins tomorrow at close of business?	Jun 03 12:23
f13	| if not sooner	Jun 03 12:23
jlaska	| okay, the RC3 results (QA:Fedora_11_RC3_Install_Test_Results) are looking fairly good (when combined with RC0, RC1, RC2)	Jun 03 12:24
jlaska	| perhaps we can focus on knocking out the remaining 9 MODIFIED bugs?	Jun 03 12:24
jlaska	| http://tinyurl.com/pqeq6n	Jun 03 12:24
jlaska	| I at least plan to circle around the anaconda issues again	Jun 03 12:25
jlaska	| that's really all I had for RC4 ... unless any quick questions or concerns, we can move on to post-mortem prep	Jun 03 12:25
jlaska	| wwoods: is bug#503824 a candiate for CommonBugs love?	Jun 03 12:26
buggbot	| Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=503824 medium, low, ---, anaconda-maint-list, NEW, x86_64 upgrade in KVM hangs (OOM) with 512MB RAM + encrypted root	Jun 03 12:26
jlaska	| okay, changing topics ...	Jun 03 12:28
--- | jlaska has changed the topic to: Fedora QA Meeting | F-11 QA Post-mortem discussion		Jun 03 12:28
wwoods	| jlaska: maybe, yeah	Jun 03 12:28
wwoods	| I haven't reproduced on real hardware and I haven't confirmed the workaround	Jun 03 12:28
jlaska	| wwoods: is this sometihng where booting with mem=512 on real hardware would be enough to simulate?	Jun 03 12:29
wwoods	| haven't attempted	Jun 03 12:30
wwoods	| possibly	Jun 03 12:30
jlaska	| wwoods: okay, if it's something you feel > 2 people will hit ... let's get 'er on there	Jun 03 12:31
adamw	| i would want it to get tested on real hardware ideally	Jun 03 12:31
adamw	| unfortunately i have nothing left with memory chips that small =)	Jun 03 12:31
adamw	| i might ask on the forum	Jun 03 12:31
f13	| I think there are more than one way to get anaconda to OOM with only 512 megs of ram	Jun 03 12:32
f13	| doing a PXE boot install on x86_64 graphical http is likely one way	Jun 03 12:32
f13	| or ppc	Jun 03 12:32
adamw	| btw, with my curmudgeon hat on, it's a bit sad for an installer to need 512MB of RAM. ah well.	Jun 03 12:32
jlaska	| f13: I haven't stumbled on it under those conditions yet	Jun 03 12:32
jlaska	| f13: oh you mean with under a low memory environment?	Jun 03 12:33
wwoods	| adamw: well, that's why I'm still retesting - there are Conditions involved	Jun 03 12:33
adamw	| yeah	Jun 03 12:33
wwoods	| like, installs from CD/DVD media might be fine	Jun 03 12:33
wwoods	| it might work OK on 32-bit systems	Jun 03 12:33
f13	| jlaska: right, with a machine with only 512megs of ram	Jun 03 12:33
wwoods	| might work on real hardware	Jun 03 12:33
jlaska	| adamw: yeah ... there's a tangeant there :)	Jun 03 12:33
adamw	| yep, that's why I said it'd be nice to test on real hardware	Jun 03 12:33
wwoods	| and the installer itself doesn't actually require 512MB	Jun 03 12:33
jlaska	| roger	Jun 03 12:33
adamw	| x86-64 / 512MB is a combination that's only ever likely to happen in a VM	Jun 03 12:33
wwoods	| the installer is fine now (it used to die in depsolving)	Jun 03 12:34
adamw	| thinking about it	Jun 03 12:34
wwoods	| but the glibc-common %post kills it now	Jun 03 12:34
jlaska	| is there anything else to discuss aroudn this in the meeting, or should we just track this in the bz's and test results wiki?	Jun 03 12:34
f13	| the problem with our minimum hardware requirements is that "It depends"	Jun 03 12:34
adamw	| so...actually...i guess we should either document it as a virt issue or not at all	Jun 03 12:34
f13	| it depends on what type of install you're doing, and how you launched it	Jun 03 12:34
f13	| but virt, where you're likely just booting kernel and initrd, and otherwise loading stage2 into memory, and doing a graphical install, 512megs on x86_64 is just Not Enough	Jun 03 12:35
f13	| it's not a bug really, it's just a needed documentation	Jun 03 12:35
adamw	| yeah...i'm thinking this is really almost release notes / hardware requirement material rather than common bugs	Jun 03 12:36
jlaska	| agreed	Jun 03 12:36
* | jlaska looks at the RHEL5 xen minimum requirements		Jun 03 12:36
wwoods	| right, like - I can specifically trigger it with virt preupgrade, but virt DVD upgrade would likely be OK	Jun 03 12:36
adamw	| me and rahul already have one bug open for minimum hardware requirements...https://bugzilla.redhat.com/show_bug.cgi?id=499585	Jun 03 12:37
buggbot	| Bug 499585: medium, low, ---, stickster, ASSIGNED, clarify minimum hardware requirements	Jun 03 12:37
adamw	| i can take an action item to bug the docs team about that, and add this to it	Jun 03 12:37
jlaska	| oh nice	Jun 03 12:37
* | stickster needs to get that reassigned!		Jun 03 12:37
jlaska	| adamw: yes please :)	Jun 03 12:37
adamw	| stickster: who should it be assigned to?	Jun 03 12:38
stickster	| adamw: I need to find out who's taking release notes bugs	Jun 03 12:38
adamw	| oh, and what would we say are realistic minimums for the virt case? 512MB for x86-32, 1GB for x86-64?	Jun 03 12:38
* | stickster asks in #f-docs		Jun 03 12:39
Viking-Ice	| defaults as virt manager puts it	Jun 03 12:39
f13	| adamw: and again it depends on how you boot the virt guest	Jun 03 12:41
f13	| adamw: 512 on x86_64 is likely fine if you boot virt with a .iso image or DVD	Jun 03 12:41
jlaska	| so more information is needed on this topic, is there anythign else that needs to be ironed out in this meeting? Action items etc...	Jun 03 12:41
adamw	| right. ug. we're going to have to write an essay...	Jun 03 12:41
f13	| 256 or 312 megs might be OK on x86_32 if you boot with boot.iso or dvd.iso or physical media, and do a text mode install	Jun 03 12:41
adamw	| i'll take the action item and try to come up with some appropriate text in conjunction with you guys	Jun 03 12:42
f13	| adamw: or you could just use the "worst case" install type (pxe, graphical) and state that it may be possible to use less ram in a different scenario and leave tips on how to reduce memory requirements	Jun 03 12:42
adamw	| but i know jlaska wants to move on :)	Jun 03 12:42
jlaska	| it's a good topic, but do want to leave some time for post-mortem talk	Jun 03 12:43
jlaska	| adamw: thanks, we can follow-up during the week on that	Jun 03 12:45
jlaska	| so, I suspect (or hope) there will be a post-mortem discussion for the Fedora 11 release as a whole	Jun 03 12:45
f13	| yeah, we should have that scheduled.	Jun 03 12:45
jlaska	| but before that gets started, I wanted to spend some time discussion what went well, and what needed improvement from a QA perspective	Jun 03 12:46
jlaska	| not sure of a good way to get the ideas flowing, but perhaps just opening the floor to 'things that worked well'	Jun 03 12:46
adamw	| everything I did worked particularly stunningly, I thought!	Jun 03 12:47
jlaska	| haha	Jun 03 12:47
adamw	| you guys sucked, though. i was totally carrying you.	Jun 03 12:47
Viking-Ice	| Mine even better than adamw's	Jun 03 12:47
adamw	| no, uh - i think for a start it was a good thing that we had the testicular fortitude to delay the release twice	Jun 03 12:47
f13	| so I got a few	Jun 03 12:48
Viking-Ice	| it's proven it self if we delay any of the milestones we end up delaying the whole release anyway	Jun 03 12:48
adamw	| though it seems we maybe didn't catch some of the blocker issues early enough	Jun 03 12:48
f13	| We need to be better about promoting things to the BLOCKER lists	Jun 03 12:48
f13	| and allowing the review board of folks to take them off the list as necessary	Jun 03 12:48
f13	| we saw too many things slip through because somebody was afraid to make it a blocker	Jun 03 12:49
wwoods	| the test days were awesome. part of my brain says that a) we should only accept features that we can plan test days for, and b) large-scale changes to the video drivers count a feature	Jun 03 12:49
adamw	| agreed...we also need to be more effective at getting other people to add things to the blocker list	Jun 03 12:49
Viking-Ice	| like who?	Jun 03 12:49
adamw	| for e.g., while i was pushing the i8x5 opengl issue, krh was working on a text corruption bug that affected all chipsets...which i wasn't aware of because he hadn't marked it blocker (and neither had anyone else)	Jun 03 12:49
poelcat	| review the blocker lists farther in advance of freezes and releases	Jun 03 12:50
adamw	| but he was _treating_ it as a blocker (working on it right up to the ga deadline)	Jun 03 12:50
f13	| poelcat: good one	Jun 03 12:50
adamw	| so we obviously haven't yet spread the gospel of marking things as blockers	Jun 03 12:50
poelcat	| clarify/document the handoff process between release engineering and QA of content to be tested	Jun 03 12:51
poelcat	| clarify/document expectations around when said content needs to be tested by	Jun 03 12:51
* | poelcat notes these are just my "brainstorming ideas"... not "must do next time"		Jun 03 12:52
adamw	| btw, jlaska has had to run, so he asked me to drive - i'm just leaving it open for people to throw things out there	Jun 03 12:53
adamw	| we're just trying to build a list of ideas at this point	Jun 03 12:53
adamw	| please do throw in anything in your head, doesn't have to be fully-formed	Jun 03 12:53
adamw	| i think we could maybe think about the release candidate distribution issue as well; f13 has a lot of reasons why it's difficult but it seems like there's a lot of genuine productive energy to try and make it happen (we actually had someone go out and build and share deltaisos - something productive rather than the usual whining about how someone else should do it)	Jun 03 12:53
f13	| ideate you mean	Jun 03 12:53
adamw	| every time you use that word, a kitten dies	Jun 03 12:53
f13	| adamw: hrm, have you not been through our design thinking sessions?	Jun 03 12:54
adamw	| yes, that doesn't mean i have to like the terminology =)	Jun 03 12:54
f13	| ideate is part of Design Thinking, a very very productive way of approaching problem solving.	Jun 03 12:54
adamw	| sorry, we're sidetracking - i like the idea i just hate the word	Jun 03 12:54
adamw	| so, yes, ideate (shudder) - everybody ideate	Jun 03 12:55
Viking-Ice	| f13: is this so called FAD day of yours done ? if not would it be possible to record the session(s)	Jun 03 12:55
f13	| it's not done, it's next week	Jun 03 12:55
f13	| recording will be difficult.	Jun 03 12:55
f13	| We asked RHT and they're going to record some of the first day for a promotional video I think	Jun 03 12:55
Viking-Ice	| Dam we need cams..	Jun 03 12:55
f13	| but we'll have an open conference call line, we'll be blogging, reading email, maybe even an open IRC channel	Jun 03 12:56
poelcat	| more transparency around the decision to slip the release or not... public discussion or at least meeting notes	Jun 03 12:56
adamw	| right - that's a good one	Jun 03 12:56
adamw	| we had generally positive response to the decision to slip but lots of people who were just guessing why we did it	Jun 03 12:56
adamw	| there's no reason we couldn't have had a log out there which just had the reasoning written down right there	Jun 03 12:57
poelcat	| try to get a better idea of how many people test each test release	Jun 03 12:58
poelcat	| maybe using smolt	Jun 03 12:58
poelcat	| or even rawhide	Jun 03 12:58
poelcat	| yep, i'm waiting to play that card :)better idea of how many people test each test release and rawhide	Jun 03 12:59
poelcat	| hm, two conversations in one :)	Jun 03 12:59
adamw	| do we not get numbers on users of test releases?	Jun 03 13:00
wwoods	| not directly	Jun 03 13:00
adamw	| can't we apply that "ten bazillion fedora users!" calculation? count repo hits?	Jun 03 13:00
wwoods	| no	Jun 03 13:01
wwoods	| obviously it suffers from the same problems as getting statistics of normal users	Jun 03 13:01
Viking-Ice	| the whole fedora stats needs rewrite infra needs to get familiar with rrdtools	Jun 03 13:01
adamw	| that doesn't stop us pretending :)	Jun 03 13:01
wwoods	| but the normal releases have two things we can count: hits to the release-specific mirror URL	Jun 03 13:01
wwoods	| and new smolt IDs	Jun 03 13:01
wwoods	| I'm not sure about counting hits on the rawhide mirror URL	Jun 03 13:02
adamw	| well, we can still do something with mirror numbers, right? we should have a decent 'background' number for rawhide hits when there's no test release	Jun 03 13:02
wwoods	| but people installing test releases rarely send in smolt stats for that machine, since it's just a test installation	Jun 03 13:02
adamw	| so we could derive numbers for test releases...roughly, anyway?	Jun 03 13:02
wwoods	| so yes, you could probably put together a very rough guess of how many people are following rawhide.	Jun 03 13:02
wwoods	| but is that actually the question you asked?	Jun 03 13:03
adamw	| i dunno, it was poelcat's question :)	Jun 03 13:03
poelcat	| i think it would be interesting to have a rough idea what kind of "testing community" we have and if it is growing or shrinking	Jun 03 13:03
adamw	| poelcat...what's the ultimate goal, here? just try and make sure we're increasing the use of rawhide and test releases?	Jun 03 13:03
poelcat	| or affecting the quality of our releases	Jun 03 13:03
wwoods	| ideally I'd like to have a notion of our usable capacity to test	Jun 03 13:03
wwoods	| and limit the number of accepted features based on that	Jun 03 13:04
poelcat	| or if we need to go to big daddy and ask for more resources	Jun 03 13:04
adamw	| ok...	Jun 03 13:05
adamw	| anyone got any more ideating to do?	Jun 03 13:05
* | Viking-Ice lot's of ideas no time at the moment to explain them : )		Jun 03 13:06
f13	| better rallying around NEEDSRETESTING	Jun 03 13:07
adamw	| Viking-Ice: just throw a quick one-line summary, so we have a record	Jun 03 13:07
Viking-Ice	| As I replied to devel today maintainers need to user better the resource they have at their disposal fedora-test-list along with QA trac instance	Jun 03 13:08
Viking-Ice	| if they Need retesting ( reporter unavailable ) or quick bodhi passing	Jun 03 13:08
adamw	| so encouraging maintainers to use QA as a resource they can call upon, yep	Jun 03 13:09
Viking-Ice	| Coordinating with upstream testing lot of thing we need to look at ( referring to our ff testing )	Jun 03 13:10
Viking-Ice	| Thats on the todo list	Jun 03 13:10
wwoods	| I'm confused. what are we supposed to be brainstorming about?	Jun 03 13:11
adamw	| wwoods: supposed to be f11 cycle post-mortem	Jun 03 13:12
adamw	| i think a lot of these are things that the f11 cycle made people think "it would be cool if..."	Jun 03 13:12
wwoods	| things specific to the F11 cycle?	Jun 03 13:12
adamw	| anyone have anything specific about anything we actually _did_ during f11 cycle?	Jun 03 13:13
wwoods	| I mentioned test days already...	Jun 03 13:13
wwoods	| testopia was nice while we had it	Jun 03 13:13
f13	| god yes	Jun 03 13:13
wwoods	| (was that F11 or F10?)	Jun 03 13:13
adamw	| we had the semantic proposal to provide something like that	Jun 03 13:13
Viking-Ice	| One thing I've discussed with jlaska about is not holding a test day for something that depends on other thing that's broken like hey we schedueled a Gnome Test day but X is not working	Jun 03 13:13
adamw	| which is currently stalled, blocking on jlaska/infrastructure	Jun 03 13:14
f13	| whatever our next test case management system is, we should code name it "Flying Car"	Jun 03 13:14
adamw	| Viking-Ice: did we do that?	Jun 03 13:14
f13	| Viking-Ice: knowing the future would be a great boon yes	Jun 03 13:14
Viking-Ice	| alpha cycle test have had some test day going south due to rawhide status at the time basically bad scheduling	Jun 03 13:14
f13	| (IE it's hard to know if a given component will be working or broken from update to update)	Jun 03 13:15
Viking-Ice	| we roughly know the shape of rawhide at the time so we could adjust..	Jun 03 13:15
wwoods	| the lesson there, I think, is that we need to leave a couple of test day slots open for rescheduling	Jun 03 13:15
wwoods	| like leaving room in the school schedule for snow day make-up	Jun 03 13:15
adamw	| and try to schedule ones which rely on a more complex pile of components till a later (and hopefully stabler) point in the cycle	Jun 03 13:15
Viking-Ice	| one example is anaconda is borked or we cant create live cd's at the given time etc..	Jun 03 13:16
adamw	| and also, make rawhide 100% stable, i think that's a clear and achievable goal ;)	Jun 03 13:16
Viking-Ice	| rawhide is not supposed to be stable :)=	Jun 03 13:16
f13	| yes, no bugs.	Jun 03 13:16
wwoods	| absolutely. just redefine "stable"	Jun 03 13:16
f13	| Viking-Ice: it is if you want to use it for a test day	Jun 03 13:16
adamw	| so, ok, we can definitely make improvements with the scheduling to try and be more flexible for brokenness	Jun 03 13:17
adamw	| and we should continue to start trying to build live CDs several days before the test day, so we're not screwed if it happens to be broken just for one day	Jun 03 13:17
Viking-Ice	| one thing we should be working at is to make everyday a test day	Jun 03 13:17
adamw	| alright, getting too general again!	Jun 03 13:17
adamw	| think f11 follow up :)	Jun 03 13:17
wwoods	| we were supposed to get autoqa stuff going for F11	Jun 03 13:18
wwoods	| and instead we had to choose between doing autoqa or testing F11	Jun 03 13:18
Viking-Ice	| which reminds me have any taken a look at litmus?	Jun 03 13:18
adamw	| that sounds pretty much like the old classic everything takes longer and needs more people than you think it will, though	Jun 03 13:18
Viking-Ice	| https://wiki.mozilla.org/Litmus	Jun 03 13:19
adamw	| it shouldn't be a recurring issue, right? by the time we hit the rush phase for f12, lots of autoqa stuff should be up and humming over nicely	Jun 03 13:19
wwoods	| adamw: right	Jun 03 13:19
f13	| wwoods: we did get some progress on autoqa	Jun 03 13:19
wwoods	| yeah	Jun 03 13:19
wwoods	| and we should probably have actually stuff running automatically very soon	Jun 03 13:19
adamw	| right	Jun 03 13:19
wwoods	| err, actual	Jun 03 13:19
adamw	| so that was just a "it's unfortunate but crap happens" moment, i think...not one we can learn any specific lessons from...	Jun 03 13:20
wwoods	| "more autoqa" would be very, very good	Jun 03 13:20
adamw	| yeah, definitely	Jun 03 13:20
adamw	| just to take autoqa on its own - autoqa is awesome and we need as much of it as possible :)	Jun 03 13:20
wwoods	| people do use what scant reports we produce	Jun 03 13:20
wwoods	| so doing more can't possibly hurt	Jun 03 13:20
adamw	| Viking-Ice: i don't think anyone's looked specifically into litmus, btw, no - i don't remember it coming up in our previous discussions about test case management...we can put the topic on next week's meeting agenda	Jun 03 13:21
Viking-Ice	| Great	Jun 03 13:21
Viking-Ice	| What ever we decide to use we need to spread the word and get other high profile upstream project to use so test cases can be exported/imported	Jun 03 13:22
f13	| little steps	Jun 03 13:22
adamw	| right...for this meeting, we can say, we would definitely benefit from proper test case management, the f11 cycle did show us that.	Jun 03 13:22
adamw	| ok, are we all done ideating?	Jun 03 13:24
Viking-Ice	| One other thing which F11 cycle ( an previous ones ) wiki is not that good in many things like test results etc..	Jun 03 13:24
Viking-Ice	| we need to come up with some replacement hopefully in the F12 cycle	Jun 03 13:25
adamw	| well, we already noted for test cases...did you have anything else you specifically thought the wiki was a bad way to handle?	Jun 03 13:25
Viking-Ice	| stats :)	Jun 03 13:25
adamw	| which stats?	Jun 03 13:25
Viking-Ice	| Statistics <-- HELLO	Jun 03 13:26
adamw	| ok, i see	Jun 03 13:26
adamw	| does QA own that, though?	Jun 03 13:26
Viking-Ice	| charts people charts. .	Jun 03 13:26
adamw	| if not, it's not really our problem (to take a cynical organizational view of things :>)	Jun 03 13:27
Viking-Ice	| QA <--	Jun 03 13:27
adamw	| i think that's coming from stickster with his pr hat on - it may be something you could talk to him about	Jun 03 13:27
Viking-Ice	| infra web space simple html page with autogenerated stats on charts	Jun 03 13:28
adamw	| stickster: is that right? what would be the group to talk about improving how those statistics are presented?	Jun 03 13:28
adamw	| well...stickster's not around. but yeah, i think that goes under the 'not-QA' heading	Jun 03 13:29
* | stickster gets back, sorry		Jun 03 13:29
adamw	| stickster: quick answer to the above?	Jun 03 13:30
stickster	| I would say the Websites team	Jun 03 13:30
stickster	| But the answer will be, "Who's going to write that autogeneration material?"	Jun 03 13:30
* | stickster puts on Great Carnac prognostication hat		Jun 03 13:30
Viking-Ice	| the one that controls the logs	Jun 03 13:30
adamw	| alright, so, talk to the websites team and bring patches =)	Jun 03 13:31
adamw	| it does sound sensible to me though	Jun 03 13:31
adamw	| surely writing a quick auto-generation script is preferable to some mug running it manually and updating the wiki page every week	Jun 03 13:31
stickster	| I don't think anyone "controls" the logs per se, but I think anyone's entitled to look at them and help write stuff	Jun 03 13:31
* | stickster has some really simple bash+cron stuff he does to get those numbers		Jun 03 13:31
stickster	| and I'm happy to share those	Jun 03 13:31
stickster	| It takes me about 5 minutes every Tuesday to add them to the wiki page	Jun 03 13:31
stickster	| But I'm all for eliminating that step in favor of something shinier.	Jun 03 13:32
adamw	| alright	Jun 03 13:32
adamw	| we're definitely off QA topic	Jun 03 13:32
stickster	| I think we're leaving the QA space.	Jun 03 13:32
adamw	| and we're over time already	Jun 03 13:32
stickster	| thanks adamw, sorry.	Jun 03 13:32
adamw	| so, let's cut that one there :)	Jun 03 13:32
adamw	| alright...i guess we're done with the post-mortem f11 topic for now	Jun 03 13:32
adamw	| so, that's the last topic on the list - which leaves us at open floor	Jun 03 13:32
adamw	| anyone have anything else to bring up?	Jun 03 13:33
Viking-Ice	| I think we can just wrap it up right ( been kinda mixed post mortem open floor discussions )	Jun 03 13:33
adamw	| well, let's see (maybe everyone else left already :>)	Jun 03 13:33
adamw	| going...going...	Jun 03 13:34
adamw	| alright, end of meeting! thanks everyone.	Jun 03 13:34
* | Viking-Ice smurf...		Jun 03 13:35

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!