From Fedora Project Wiki

< QA‎ | Meetings

m (Update notes)
m (Updated notes)
Line 61: Line 61:
#* Improve messaging and guidelines around how to escalate a blocker bug for review
#* Improve messaging and guidelines around how to escalate a blocker bug for review
#* No workflow around verifying MODIFIED bugs
#* No workflow around verifying MODIFIED bugs
#* Large number ( of installation blocker bugs not reviewed prior to RC's
# Fedora Test Days
# Fedora Test Days
#* No test day for sound during F11
#* No test day for sound during F11
Line 72: Line 73:
#* No distro-wide automated testing ... daily manual installation and repo testing
#* No distro-wide automated testing ... daily manual installation and repo testing
# Metrics
# Metrics
#* No sense for the health of the QA community, is it growing or shrinking?
#* No clear data for the health of the QA community, is it growing or shrinking?
#* Not clear how many testers contribute to each milestone (Alpha, Beta, Preview)
#* Not clear how many testers contribute to each milestone (Alpha, Beta, Preview)


Revision as of 12:57, 16 June 2009

Attendees

  • Adam Williamson (adamw)
  • Will Woods (wwoods)
  • Jóhann Guðmundsson (viking_ice)
  • John Poelstra (poelcat)
  • James Laska (jlaska)
  • Jesse Keating (f13)
  • Paul Frields (stickster)

Agenda

Previous meeting follow-up

  1. [adamw] - BugStatusWorkFlow - ask bugzilla guys to add a link from the bugzilla fields description page to the wiki page, for fedora
    • Completed (see RHBZ #495985), waiting for bugzilla code update during next outage window for changes to go live
  2. [jlaska] - Send informal test day feedback survey to fedora-test-list and test day participants
    • Survey sent to participants and to fedora-test-list
  3. [wwoods] - create preupgrade test case and update test matrix template
  4. [jlaska] - create new RC1 test results page
  5. [adamw] - create common bug entry for RHBZ #502077

F-11-GA Preparation

Adamw noted that the new test results wiki page wasn't created yet. Jlaska indicated it would be created soon after the meeting, once he received notification that RC4 composition had completed.

f13 indicated that he will continue to upload more RC4 bits as they are created (live images remain).

jlaska provided an update on RC test results. The RC3 results are looking fairly good (when combined with RC0, RC1, RC2). Once initial scrubbing of RC4 media kits has completed, jlaska asked if attention could be made to verify the 9 MODIFIED bugs on the F11 blocker list.

As indicated in the Releases/11/Schedule, staging will begin Thu Jun 4.

jlaska asked if RHBZ #503824 (anaconda-maint-list, NEW, x86_64 upgrade in KVM hangs (OOM) with 512MB RAM + encrypted root_) should be added to the common bugs page. Wwoods indicated that he would confirm on bare metal hardware first, and that this might be a candidate for common bugs. Further discussion began around whether Fedora has a minimal system requirements list. For additional comments, see open discussion below.

F-11 QA Post-mortem discussion

Jlaska indicated that a release-wide post-mortem review has been discussed for F11, and that he would like to begin the brainstorming around QA-specific topics. For example, what went well, and what needed improvement from a QA perspective.

jlaska had to step out, adamw lead the brainstorming.

Highlights (see IRC transcript for full details) ...

What worked well

  1. Fedora Test Days
  2. Common_F11_bugs
    • More contributors and collaboration around existing wiki page
    • Consistent use of the common bug page as a resolution pathway for unresolved bugs and throughout mailing list communication

What needs improvement

  1. Blocker Bugs
    • Schedule & host earlier blocker bug list reviews
    • Improve messaging and guidelines around how to escalate a blocker bug for review
    • No workflow around verifying MODIFIED bugs
    • Large number ( of installation blocker bugs not reviewed prior to RC's
  2. Fedora Test Days
    • No test day for sound during F11
    • Complex or tightly coupled features need to be scheduled later in the cycle when things have stabalized
    • No capacity concept ... QA would be interested in defining # of events capable of hosting
    • Create a self-hosting test day procedure for non-QA hosted test events
  3. Release decisions
    • Clarify hand-off procedures between release engineering and quality assurance
    • Increase transparency around release slip meetings (or publicize minutes)
  4. Automation
    • No distro-wide automated testing ... daily manual installation and repo testing
  5. Metrics
    • No clear data for the health of the QA community, is it growing or shrinking?
    • Not clear how many testers contribute to each milestone (Alpha, Beta, Preview)

Open discussion

What are Fedora minimal requirements?

wwoods is seeing an OOM kill issue on a x86_64 KVM guest when upgrading from F10 -> F11 with only 512Mb of memory.

Everyone noted there isn't a clear definition of what minimal system requirements are for Fedora. Adamw directed folks to the bug he and Rahul filed RHBZ #499585.

Upcoming QA meetings/events

Action items

  • [stickster] - who will be handling release_notes bugs to help with RHBZ #499585
  • [adamw] - propose draft wording of minimal requirements for the release_notes team to digest

IRC Transcript

--- | jlaska has changed the topic to: Fedora Quality Assurance Meeting | gathering Jun 03 12:00
wwoods | huzzah Jun 03 12:00
jlaska | wwoods: greetings Jun 03 12:00
adamw | hooray Jun 03 12:01
jlaska | QA meeting time, anyone around please clap your ends (against the keyboard) Jun 03 12:01
jlaska | adamw: heyo Jun 03 12:01
Viking-Ice | Here Jun 03 12:01
jlaska | Viking-Ice: hey there Jun 03 12:01
jlaska | I've got a hard stop in an hour today, so if we go over I'll need help from someone to drive Jun 03 12:02
jlaska | okay, getting started with last weeks items Jun 03 12:02
adamw | we'll figure something out Jun 03 12:02
jlaska | heh Jun 03 12:02
--- | jlaska has changed the topic to: Fedora QA Meeting | previous week follow-up Jun 03 12:03
jlaska | okay, I'm walking the list ... https://fedoraproject.org/wiki/QA/Meetings/20090603#Previous_meeting_follow-up Jun 03 12:03
jlaska | our lucky first up is ... <drumroll> Jun 03 12:03
jlaska | # [adamw] - BugStatusWorkFlow - ask bugzilla guys to add a link from the bugzilla fields description page to the wiki page, for fedora Jun 03 12:03
adamw | well, i did it Jun 03 12:03
adamw | and they say it's done Jun 03 12:03
adamw | i'm not actually seeing it on the page though Jun 03 12:04
jlaska | oh funky Jun 03 12:04
adamw | oh, dave says "Will show up in the next code update." Jun 03 12:04
adamw | so, presumably there hasn't been one o' those yet Jun 03 12:04
jlaska | ah right, during the next outage window Jun 03 12:04
jlaska | once or twice a month, I forget Jun 03 12:04
jlaska | there might be a staging/devel server if you wanted to test the changes out before the push? Jun 03 12:05
adamw | he didn't mention anything. it's a pretty straightforward change. Jun 03 12:05
jlaska | okay Jun 03 12:06
jlaska | any other updates or next-steps needed on that front? Jun 03 12:06
adamw | not really, it was a pretty simple thing :) Jun 03 12:07
jlaska | roger thanks Jun 03 12:07
jlaska | okay next up ... Jun 03 12:07
jlaska | # [jlaska] - Send informal test day feedback survey to fedora-test-list and test day participants Jun 03 12:07
jlaska | After 2 weeks of delaying I finally sent this out to the list (and bcc'd participants) Jun 03 12:07
jlaska | posted just this morning, have some feedback (list and direct mail) coming already ... but expect to see much more Jun 03 12:08
adamw | great Jun 03 12:08
f13 | I'm here Jun 03 12:08
jlaska | adamw: gave a "shout out" to your planet post as well :) Jun 03 12:08
jlaska | f13: welcome Jun 03 12:08
* | jlaska moving right along Jun 03 12:09
jlaska | next up ... Jun 03 12:09
jlaska | # [wwoods] - create preupgrade test case and update test matrix template Jun 03 12:09
jlaska | I think you had that done before the meeting finished last week? Jun 03 12:09
jlaska | wwoods: did we lose ya? Jun 03 12:11
jlaska | okay, I think that one is completed, but will revisit later if needed Jun 03 12:12
jlaska | # [jlaska] - create new RC1 test results page Jun 03 12:12
wwoods | sorry Jun 03 12:12
jlaska | np ... you've got the floor Jun 03 12:12
wwoods | yeah, we've got two preupgrade cases Jun 03 12:12
wwoods | and the matrix template and current results page are updated Jun 03 12:12
wwoods | https://fedoraproject.org/wiki/QA:Fedora_11_RC3_Install_Test_Results#Upgrade_system Jun 03 12:12
jlaska | eggsellent, and I also saw you posting results+bugs for those too ... nice Jun 03 12:13
jlaska | any other things to track on that front? Jun 03 12:13
wwoods | not that I'm aware of Jun 03 12:14
jlaska | okay thanks Jun 03 12:14
jlaska | as you saw in wwoods post earlier, we have not just 1, but 3 RC results pages so far Jun 03 12:14
jlaska | with another on the way ... I'll create that once we get word that RC4 is live Jun 03 12:15
jlaska | okay last up ... Jun 03 12:15
jlaska | # [adamw] - create common bug entry for RHBZ #502077 Jun 03 12:15
jlaska | adamw ... aka Mr. Common Bugs Jun 03 12:15
adamw | done, and has been outpaced by events Jun 03 12:15
adamw | https://fedoraproject.org/wiki/Common_F11_bugs#865-hangs Jun 03 12:15
adamw | but i should take it out Jun 03 12:15
adamw | as the updated kernel fixed it and we're putting that in GA Jun 03 12:15
jlaska | oh we are? Jun 03 12:16
f13 | jlaska: it's live, forgot to tell you Jun 03 12:16
adamw | so...it's gone Jun 03 12:16
jlaska | f13: I see partial images Jun 03 12:16
jlaska | adamw: ah okay, does this move to the fixed section ... or just dropped entirely? Jun 03 12:16
adamw | for now i'm just dropping them entirely Jun 03 12:17
f13 | oh that's right, I made putting it on kojipkgs a higher priority for wwoods Jun 03 12:17
wwoods | sorry? Jun 03 12:17
adamw | for mdv i had separate 'modes' for pre-release and post-release for the common issues page Jun 03 12:17
jlaska | adamw: okay Jun 03 12:17
adamw | before release, i'd have a 'resolved issues' section which noted problems that had been fixed during the pre-release cycle, then that would be cleared out once we hit final release and the 'resolved issues' section would start over again Jun 03 12:17
adamw | i think that's a good model, but for f11 i'm just working in the 'final release' mode, we started too close to final release to bother with the pre-release mode i think Jun 03 12:18
adamw | so anything for which the fix goes into GA just gets taken out Jun 03 12:18
jlaska | adamw: are there steps we should consider now for the F12 common bugs process? Jun 03 12:18
jlaska | adamw: that seems fair Jun 03 12:18
adamw | not really, just create the page once we have something to write in it, copy all the boilerplate over from f11 Jun 03 12:19
adamw | i'd probably do it around the time we release a first snapshot, but it can be done earlier if anyone wants to Jun 03 12:19
* | jlaska not sure who owns it ... and when it's expected to land Jun 03 12:20
jlaska | but that can be a topic for another time Jun 03 12:20
adamw | yeah Jun 03 12:20
jlaska | okay that's it for last week ... in record time too! :) Jun 03 12:20
--- | jlaska has changed the topic to: Fedora QA Meeting | F-11-GA Prep Jun 03 12:20
jlaska | didn't want to spend a lot of time here ... just reviewing the links and "who has the ball" Jun 03 12:21
jlaska | * Blocker bugs (by component) - http://tinyurl.com/pqeq6n (currently 0 OPEN issues) Jun 03 12:21
jlaska | * Schedule - http://fedoraproject.org/wiki/Releases/11/Schedule Jun 03 12:21
jlaska | * Installation test results - QA:Fedora_11_RC4_Install_Test_Results Jun 03 12:21
jlaska | Jun 03 12:21
jlaska | we're expecting RC4 any second now (looks like content just about uploaded to the double secret password location) Jun 03 12:22
adamw | the wiki page doesn't seem to exist yet Jun 03 12:22
jlaska | with that I'll get an RC4 page posted after the meeting to highlight any changes against RC3 Jun 03 12:22
jlaska | adamw: right on, I'll pull that together after the meeting Jun 03 12:22
f13 | I'll continue to upload more bits of RC4 as they are created, mostly the live stuff Jun 03 12:23
f13 | but first I have to make jigdo of the other stuff. Jun 03 12:23
jlaska | f13: thanks ... I've got most of the content downloaded, except for x86_64+ppc DVD's Jun 03 12:23
f13 | it really sucks not having enough disc space to do it all on one system before uploading :/ Jun 03 12:23
<-- | mcepl [n=mcepl@49-117-207-85.strcechy.adsl-llu.static.bluetone.cz] has left #fedora-meeting ( ) Jun 03 12:23
jlaska | f13: as far as next steps ... the staging begins tomorrow at close of business? Jun 03 12:23
f13 | if not sooner Jun 03 12:23
jlaska | okay, the RC3 results (https://fedoraproject.org/wiki/QA:Fedora_11_RC3_Install_Test_Results) are looking fairly good (when combined with RC0, RC1, RC2) Jun 03 12:24
jlaska | perhaps we can focus on knocking out the remaining 9 MODIFIED bugs? Jun 03 12:24
jlaska | http://tinyurl.com/pqeq6n Jun 03 12:24
jlaska | I at least plan to circle around the anaconda issues again Jun 03 12:25
jlaska | that's really all I had for RC4 ... unless any quick questions or concerns, we can move on to post-mortem prep Jun 03 12:25
jlaska | wwoods: is bug#503824 a candiate for CommonBugs love? Jun 03 12:26
buggbot | Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=503824 medium, low, ---, anaconda-maint-list, NEW, x86_64 upgrade in KVM hangs (OOM) with 512MB RAM + encrypted root Jun 03 12:26
jlaska | okay, changing topics ... Jun 03 12:28
--- | jlaska has changed the topic to: Fedora QA Meeting | F-11 QA Post-mortem discussion Jun 03 12:28
wwoods | jlaska: maybe, yeah Jun 03 12:28
wwoods | I haven't reproduced on real hardware and I haven't confirmed the workaround Jun 03 12:28
jlaska | wwoods: is this sometihng where booting with mem=512 on real hardware would be enough to simulate? Jun 03 12:29
wwoods | haven't attempted Jun 03 12:30
wwoods | possibly Jun 03 12:30
jlaska | wwoods: okay, if it's something you feel > 2 people will hit ... let's get 'er on there Jun 03 12:31
adamw | i would want it to get tested on real hardware ideally Jun 03 12:31
adamw | unfortunately i have nothing left with memory chips that small =) Jun 03 12:31
adamw | i might ask on the forum Jun 03 12:31
f13 | I think there are more than one way to get anaconda to OOM with only 512 megs of ram Jun 03 12:32
f13 | doing a PXE boot install on x86_64 graphical http is likely one way Jun 03 12:32
f13 | or ppc Jun 03 12:32
adamw | btw, with my curmudgeon hat on, it's a bit sad for an installer to need 512MB of RAM. ah well. Jun 03 12:32
jlaska | f13: I haven't stumbled on it under those conditions yet Jun 03 12:32
jlaska | f13: oh you mean with under a low memory environment? Jun 03 12:33
wwoods | adamw: well, that's why I'm still retesting - there are Conditions involved Jun 03 12:33
adamw | yeah Jun 03 12:33
wwoods | like, installs from CD/DVD media might be fine Jun 03 12:33
wwoods | it might work OK on 32-bit systems Jun 03 12:33
f13 | jlaska: right, with a machine with only 512megs of ram Jun 03 12:33
wwoods | might work on real hardware Jun 03 12:33
jlaska | adamw: yeah ... there's a tangeant there :) Jun 03 12:33
adamw | yep, that's why I said it'd be nice to test on real hardware Jun 03 12:33
wwoods | and the installer itself doesn't actually require 512MB Jun 03 12:33
jlaska | roger Jun 03 12:33
adamw | x86-64 / 512MB is a combination that's only ever likely to happen in a VM Jun 03 12:33
wwoods | the installer is fine now (it used to die in depsolving) Jun 03 12:34
adamw | thinking about it Jun 03 12:34
wwoods | but the glibc-common %post kills it now Jun 03 12:34
jlaska | is there anything else to discuss aroudn this in the meeting, or should we just track this in the bz's and test results wiki? Jun 03 12:34
f13 | the problem with our minimum hardware requirements is that "It depends" Jun 03 12:34
adamw | so...actually...i guess we should either document it as a virt issue or not at all Jun 03 12:34
f13 | it depends on what type of install you're doing, and how you launched it Jun 03 12:34
f13 | but virt, where you're likely just booting kernel and initrd, and otherwise loading stage2 into memory, and doing a graphical install, 512megs on x86_64 is just Not Enough Jun 03 12:35
f13 | it's not a bug really, it's just a needed documentation Jun 03 12:35
adamw | yeah...i'm thinking this is really almost release notes / hardware requirement material rather than common bugs Jun 03 12:36
jlaska | agreed Jun 03 12:36
* | jlaska looks at the RHEL5 xen minimum requirements Jun 03 12:36
wwoods | right, like - I can specifically trigger it with virt preupgrade, but virt DVD upgrade would likely be OK Jun 03 12:36
adamw | me and rahul already have one bug open for minimum hardware requirements...https://bugzilla.redhat.com/show_bug.cgi?id=499585 Jun 03 12:37
buggbot | Bug 499585: medium, low, ---, stickster, ASSIGNED, clarify minimum hardware requirements Jun 03 12:37
adamw | i can take an action item to bug the docs team about that, and add this to it Jun 03 12:37
jlaska | oh nice Jun 03 12:37
* | stickster needs to get that reassigned! Jun 03 12:37
jlaska | adamw: yes please :) Jun 03 12:37
adamw | stickster: who should it be assigned to? Jun 03 12:38
stickster | adamw: I need to find out who's taking release notes bugs Jun 03 12:38
adamw | oh, and what would we say are realistic minimums for the virt case? 512MB for x86-32, 1GB for x86-64? Jun 03 12:38
* | stickster asks in #f-docs Jun 03 12:39
Viking-Ice | defaults as virt manager puts it Jun 03 12:39
f13 | adamw: and again it depends on how you boot the virt guest Jun 03 12:41
f13 | adamw: 512 on x86_64 is likely fine if you boot virt with a .iso image or DVD Jun 03 12:41
jlaska | so more information is needed on this topic, is there anythign else that needs to be ironed out in this meeting? Action items etc... Jun 03 12:41
adamw | right. ug. we're going to have to write an essay... Jun 03 12:41
f13 | 256 or 312 megs might be OK on x86_32 if you boot with boot.iso or dvd.iso or physical media, and do a text mode install Jun 03 12:41
adamw | i'll take the action item and try to come up with some appropriate text in conjunction with you guys Jun 03 12:42
f13 | adamw: or you could just use the "worst case" install type (pxe, graphical) and state that it may be possible to use less ram in a different scenario and leave tips on how to reduce memory requirements Jun 03 12:42
adamw | but i know jlaska wants to move on :) Jun 03 12:42
jlaska | it's a good topic, but do want to leave some time for post-mortem talk Jun 03 12:43
jlaska | adamw: thanks, we can follow-up during the week on that Jun 03 12:45
jlaska | so, I suspect (or hope) there will be a post-mortem discussion for the Fedora 11 release as a whole Jun 03 12:45
f13 | yeah, we should have that scheduled. Jun 03 12:45
jlaska | but before that gets started, I wanted to spend some time discussion what went well, and what needed improvement from a QA perspective Jun 03 12:46
jlaska | not sure of a good way to get the ideas flowing, but perhaps just opening the floor to 'things that worked well' Jun 03 12:46
adamw | everything I did worked particularly stunningly, I thought! Jun 03 12:47
jlaska | haha Jun 03 12:47
adamw | you guys sucked, though. i was totally carrying you. Jun 03 12:47
Viking-Ice | Mine even better than adamw's Jun 03 12:47
adamw | no, uh - i think for a start it was a good thing that we had the testicular fortitude to delay the release twice Jun 03 12:47
f13 | so I got a few Jun 03 12:48
Viking-Ice | it's proven it self if we delay any of the milestones we end up delaying the whole release anyway Jun 03 12:48
adamw | though it seems we maybe didn't catch some of the blocker issues early enough Jun 03 12:48
f13 | We need to be better about promoting things to the BLOCKER lists Jun 03 12:48
f13 | and allowing the review board of folks to take them off the list as necessary Jun 03 12:48
f13 | we saw too many things slip through because somebody was afraid to make it a blocker Jun 03 12:49
wwoods | the test days were awesome. part of my brain says that a) we should only accept features that we can plan test days for, and b) large-scale changes to the video drivers count a feature Jun 03 12:49
adamw | agreed...we also need to be more effective at getting other people to add things to the blocker list Jun 03 12:49
Viking-Ice | like who? Jun 03 12:49
adamw | for e.g., while i was pushing the i8x5 opengl issue, krh was working on a text corruption bug that affected all chipsets...which i wasn't aware of because he hadn't marked it blocker (and neither had anyone else) Jun 03 12:49
poelcat | review the blocker lists farther in advance of freezes and releases Jun 03 12:50
adamw | but he was _treating_ it as a blocker (working on it right up to the ga deadline) Jun 03 12:50
f13 | poelcat: good one Jun 03 12:50
adamw | so we obviously haven't yet spread the gospel of marking things as blockers Jun 03 12:50
poelcat | clarify/document the handoff process between release engineering and QA of content to be tested Jun 03 12:51
poelcat | clarify/document expectations around when said content needs to be tested by Jun 03 12:51
* | poelcat notes these are just my "brainstorming ideas"... not "must do next time" Jun 03 12:52
adamw | btw, jlaska has had to run, so he asked me to drive - i'm just leaving it open for people to throw things out there Jun 03 12:53
adamw | we're just trying to build a list of ideas at this point Jun 03 12:53
adamw | please do throw in anything in your head, doesn't have to be fully-formed Jun 03 12:53
adamw | i think we could maybe think about the release candidate distribution issue as well; f13 has a lot of reasons why it's difficult but it seems like there's a lot of genuine productive energy to try and make it happen (we actually had someone go out and build and share deltaisos - something productive rather than the usual whining about how someone else should do it) Jun 03 12:53
f13 | ideate you mean Jun 03 12:53
adamw | every time you use that word, a kitten dies Jun 03 12:53
f13 | adamw: hrm, have you not been through our design thinking sessions? Jun 03 12:54
adamw | yes, that doesn't mean i have to like the terminology =) Jun 03 12:54
f13 | ideate is part of Design Thinking, a very very productive way of approaching problem solving. Jun 03 12:54
adamw | sorry, we're sidetracking - i like the idea i just hate the word Jun 03 12:54
adamw | so, yes, ideate (shudder) - everybody ideate Jun 03 12:55
Viking-Ice | f13: is this so called FAD day of yours done ? if not would it be possible to record the session(s) Jun 03 12:55
f13 | it's not done, it's next week Jun 03 12:55
f13 | recording will be difficult. Jun 03 12:55
f13 | We asked RHT and they're going to record some of the first day for a promotional video I think Jun 03 12:55
Viking-Ice | Dam we need cams.. Jun 03 12:55
f13 | but we'll have an open conference call line, we'll be blogging, reading email, maybe even an open IRC channel Jun 03 12:56
poelcat | more transparency around the decision to slip the release or not... public discussion or at least meeting notes Jun 03 12:56
adamw | right - that's a good one Jun 03 12:56
adamw | we had generally positive response to the decision to slip but lots of people who were just guessing why we did it Jun 03 12:56
adamw | there's no reason we couldn't have had a log out there which just had the reasoning written down right there Jun 03 12:57
poelcat | try to get a better idea of how many people test each test release Jun 03 12:58
poelcat | maybe using smolt Jun 03 12:58
poelcat | or even rawhide Jun 03 12:58
poelcat | yep, i'm waiting to play that card :)better idea of how many people test each test release and rawhide Jun 03 12:59
poelcat | hm, two conversations in one :) Jun 03 12:59
adamw | do we not get numbers on users of test releases? Jun 03 13:00
wwoods | not directly Jun 03 13:00
adamw | can't we apply that "ten bazillion fedora users!" calculation? count repo hits? Jun 03 13:00
wwoods | no Jun 03 13:01
wwoods | obviously it suffers from the same problems as getting statistics of normal users Jun 03 13:01
Viking-Ice | the whole fedora stats needs rewrite infra needs to get familiar with rrdtools Jun 03 13:01
adamw | that doesn't stop us pretending :) Jun 03 13:01
wwoods | but the normal releases have two things we can count: hits to the release-specific mirror URL Jun 03 13:01
wwoods | and new smolt IDs Jun 03 13:01
wwoods | I'm not sure about counting hits on the rawhide mirror URL Jun 03 13:02
adamw | well, we can still do something with mirror numbers, right? we should have a decent 'background' number for rawhide hits when there's no test release Jun 03 13:02
wwoods | but people installing test releases rarely send in smolt stats for that machine, since it's just a test installation Jun 03 13:02
adamw | so we could derive numbers for test releases...roughly, anyway? Jun 03 13:02
wwoods | so yes, you could probably put together a very rough guess of how many people are following rawhide. Jun 03 13:02
wwoods | but is that actually the question you asked? Jun 03 13:03
adamw | i dunno, it was poelcat's question :) Jun 03 13:03
poelcat | i think it would be interesting to have a rough idea what kind of "testing community" we have and if it is growing or shrinking Jun 03 13:03
adamw | poelcat...what's the ultimate goal, here? just try and make sure we're increasing the use of rawhide and test releases? Jun 03 13:03
poelcat | or affecting the quality of our releases Jun 03 13:03
wwoods | ideally I'd like to have a notion of our usable capacity to test Jun 03 13:03
wwoods | and limit the number of accepted features based on that Jun 03 13:04
poelcat | or if we need to go to big daddy and ask for more resources Jun 03 13:04
adamw | ok... Jun 03 13:05
adamw | anyone got any more ideating to do? Jun 03 13:05
* | Viking-Ice lot's of ideas no time at the moment to explain them : ) Jun 03 13:06
f13 | better rallying around NEEDSRETESTING Jun 03 13:07
adamw | Viking-Ice: just throw a quick one-line summary, so we have a record Jun 03 13:07
Viking-Ice | As I replied to devel today maintainers need to user better the resource they have at their disposal fedora-test-list along with QA trac instance Jun 03 13:08
Viking-Ice | if they Need retesting ( reporter unavailable ) or quick bodhi passing Jun 03 13:08
adamw | so encouraging maintainers to use QA as a resource they can call upon, yep Jun 03 13:09
Viking-Ice | Coordinating with upstream testing lot of thing we need to look at ( referring to our ff testing ) Jun 03 13:10
Viking-Ice | Thats on the todo list Jun 03 13:10
wwoods | I'm confused. what are we supposed to be brainstorming about? Jun 03 13:11
adamw | wwoods: supposed to be f11 cycle post-mortem Jun 03 13:12
adamw | i think a lot of these are things that the f11 cycle made people think "it would be cool if..." Jun 03 13:12
wwoods | things specific to the F11 cycle? Jun 03 13:12
adamw | anyone have anything specific about anything we actually _did_ during f11 cycle? Jun 03 13:13
wwoods | I mentioned test days already... Jun 03 13:13
wwoods | testopia was nice while we had it Jun 03 13:13
f13 | god yes Jun 03 13:13
wwoods | (was that F11 or F10?) Jun 03 13:13
adamw | we had the semantic proposal to provide something like that Jun 03 13:13
Viking-Ice | One thing I've discussed with jlaska about is not holding a test day for something that depends on other thing that's broken like hey we schedueled a Gnome Test day but X is not working Jun 03 13:13
adamw | which is currently stalled, blocking on jlaska/infrastructure Jun 03 13:14
f13 | whatever our next test case management system is, we should code name it "Flying Car" Jun 03 13:14
adamw | Viking-Ice: did we do that? Jun 03 13:14
f13 | Viking-Ice: knowing the future would be a great boon yes Jun 03 13:14
Viking-Ice | alpha cycle test have had some test day going south due to rawhide status at the time basically bad scheduling Jun 03 13:14
f13 | (IE it's hard to know if a given component will be working or broken from update to update) Jun 03 13:15
Viking-Ice | we roughly know the shape of rawhide at the time so we could adjust.. Jun 03 13:15
wwoods | the lesson there, I think, is that we need to leave a couple of test day slots open for rescheduling Jun 03 13:15
wwoods | like leaving room in the school schedule for snow day make-up Jun 03 13:15
adamw | and try to schedule ones which rely on a more complex pile of components till a later (and hopefully stabler) point in the cycle Jun 03 13:15
Viking-Ice | one example is anaconda is borked or we cant create live cd's at the given time etc.. Jun 03 13:16
adamw | and also, make rawhide 100% stable, i think that's a clear and achievable goal ;) Jun 03 13:16
Viking-Ice | rawhide is not supposed to be stable :)= Jun 03 13:16
f13 | yes, no bugs. Jun 03 13:16
wwoods | absolutely. just redefine "stable" Jun 03 13:16
f13 | Viking-Ice: it is if you want to use it for a test day Jun 03 13:16
adamw | so, ok, we can definitely make improvements with the scheduling to try and be more flexible for brokenness Jun 03 13:17
adamw | and we should continue to start trying to build live CDs several days before the test day, so we're not screwed if it happens to be broken just for one day Jun 03 13:17
Viking-Ice | one thing we should be working at is to make everyday a test day Jun 03 13:17
adamw | alright, getting too general again! Jun 03 13:17
adamw | think f11 follow up :) Jun 03 13:17
wwoods | we were supposed to get autoqa stuff going for F11 Jun 03 13:18
wwoods | and instead we had to choose between doing autoqa or testing F11 Jun 03 13:18
Viking-Ice | which reminds me have any taken a look at litmus? Jun 03 13:18
adamw | that sounds pretty much like the old classic everything takes longer and needs more people than you think it will, though Jun 03 13:18
Viking-Ice | https://wiki.mozilla.org/Litmus Jun 03 13:19
adamw | it shouldn't be a recurring issue, right? by the time we hit the rush phase for f12, lots of autoqa stuff should be up and humming over nicely Jun 03 13:19
wwoods | adamw: right Jun 03 13:19
f13 | wwoods: we did get some progress on autoqa Jun 03 13:19
wwoods | yeah Jun 03 13:19
wwoods | and we should probably have actually stuff running automatically very soon Jun 03 13:19
adamw | right Jun 03 13:19
wwoods | err, actual Jun 03 13:19
adamw | so that was just a "it's unfortunate but crap happens" moment, i think...not one we can learn any specific lessons from... Jun 03 13:20
wwoods | "more autoqa" would be very, very good Jun 03 13:20
adamw | yeah, definitely Jun 03 13:20
adamw | just to take autoqa on its own - autoqa is awesome and we need as much of it as possible :) Jun 03 13:20
wwoods | people do use what scant reports we produce Jun 03 13:20
wwoods | so doing more can't possibly hurt Jun 03 13:20
adamw | Viking-Ice: i don't think anyone's looked specifically into litmus, btw, no - i don't remember it coming up in our previous discussions about test case management...we can put the topic on next week's meeting agenda Jun 03 13:21
Viking-Ice | Great Jun 03 13:21
Viking-Ice | What ever we decide to use we need to spread the word and get other high profile upstream project to use so test cases can be exported/imported Jun 03 13:22
f13 | little steps Jun 03 13:22
adamw | right...for this meeting, we can say, we would definitely benefit from proper test case management, the f11 cycle did show us that. Jun 03 13:22
adamw | ok, are we all done ideating? Jun 03 13:24
Viking-Ice | One other thing which F11 cycle ( an previous ones ) wiki is not that good in many things like test results etc.. Jun 03 13:24
Viking-Ice | we need to come up with some replacement hopefully in the F12 cycle Jun 03 13:25
adamw | well, we already noted for test cases...did you have anything else you specifically thought the wiki was a bad way to handle? Jun 03 13:25
Viking-Ice | stats :) Jun 03 13:25
adamw | which stats? Jun 03 13:25
Viking-Ice | https://fedoraproject.org/wiki/Statistics <-- HELLO Jun 03 13:26
adamw | ok, i see Jun 03 13:26
adamw | does QA own that, though? Jun 03 13:26
Viking-Ice | charts people charts. . Jun 03 13:26
adamw | if not, it's not really our problem (to take a cynical organizational view of things :>) Jun 03 13:27
Viking-Ice | QA <-- Jun 03 13:27
adamw | i think that's coming from stickster with his pr hat on - it may be something you could talk to him about Jun 03 13:27
Viking-Ice | infra web space simple html page with autogenerated stats on charts Jun 03 13:28
adamw | stickster: is that right? what would be the group to talk about improving how those statistics are presented? Jun 03 13:28
adamw | well...stickster's not around. but yeah, i think that goes under the 'not-QA' heading Jun 03 13:29
* | stickster gets back, sorry Jun 03 13:29
adamw | stickster: quick answer to the above? Jun 03 13:30
stickster | I would say the Websites team Jun 03 13:30
stickster | But the answer will be, "Who's going to write that autogeneration material?" Jun 03 13:30
* | stickster puts on Great Carnac prognostication hat Jun 03 13:30
Viking-Ice | the one that controls the logs Jun 03 13:30
adamw | alright, so, talk to the websites team and bring patches =) Jun 03 13:31
adamw | it does sound sensible to me though Jun 03 13:31
adamw | surely writing a quick auto-generation script is preferable to some mug running it manually and updating the wiki page every week Jun 03 13:31
stickster | I don't think anyone "controls" the logs per se, but I think anyone's entitled to look at them and help write stuff Jun 03 13:31
* | stickster has some really simple bash+cron stuff he does to get those numbers Jun 03 13:31
stickster | and I'm happy to share those Jun 03 13:31
stickster | It takes me about 5 minutes every Tuesday to add them to the wiki page Jun 03 13:31
stickster | But I'm all for eliminating that step in favor of something shinier. Jun 03 13:32
adamw | alright Jun 03 13:32
adamw | we're definitely off QA topic Jun 03 13:32
stickster | I think we're leaving the QA space. Jun 03 13:32
adamw | and we're over time already Jun 03 13:32
stickster | thanks adamw, sorry. Jun 03 13:32
adamw | so, let's cut that one there :) Jun 03 13:32
adamw | alright...i guess we're done with the post-mortem f11 topic for now Jun 03 13:32
adamw | so, that's the last topic on the list - which leaves us at open floor Jun 03 13:32
adamw | anyone have anything else to bring up? Jun 03 13:33
Viking-Ice | I think we can just wrap it up right ( been kinda mixed post mortem open floor discussions ) Jun 03 13:33
adamw | well, let's see (maybe everyone else left already :>) Jun 03 13:33
adamw | going...going... Jun 03 13:34
adamw | alright, end of meeting! thanks everyone. Jun 03 13:34
* | Viking-Ice smurf... Jun 03 13:35

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!

Retrieved from "https://fedoraproject.org/w/index.php?title=QA/Meetings/20090603&oldid=108636"