Fedora 17 QA Retrospective

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 17. The feedback will be used as a basis for identifying areas for improvement for Fedora 17 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

• Gwjasu - I like ____ about the new ____ process

Adding feedback is fairly straight forward. If you already have a Fedora account ...

1. Login to the wiki
2. Select [Edit] for the appropriate section below.
3. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

4. When done, Submit your changes

Anonymous

If you do not have a Fedora account, follow the instructions below to submit anonymous feedback. Please note, mediawiki records the IP address associated with any anonymous page edits.

1. Select the appropriate page for your feedback...
   • Something that worked well
   • Something that didn't work well
   • Anything on your QA wishlist
2. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

3. When done, Submit your changes

Feedback

Things that went well

• Adamwill - Pushing back the TC dates continued to help a lot with getting validation done on time
• Bruno - The gcc 4.7 rebuild got done early enough, that we didn't have blocker issues caused by it

Could have been better

• Adamwill - noloader landing between Beta TC1 and Beta TC2 was way too late and caused extensive havoc
• Adamwill - maybe just because we have longer for TCs now, but RC period feels very short: RC1 date is a Thursday, Go/No-Go is the following Wednesday, that's 6 days but only 4 business days of validate / fix / respin
• Adamwill - the ability to add stuff to comps during freezes finally bit us on the ass, as I knew it eventually would: Bugzilla: #807879. Adding gnome-boxes to comps during Beta freeze broke networking in KVMs
• Adamwill - desktop team was sad that we couldn't co-ordinate Beta release with the GNOME 3.4 final release so 3.4 final made Beta
• Adamwill - pulling livecd-tools 17.7 as NTH for RC4 to 'fix' https://bugzilla.redhat.com/show_bug.cgi?id=810104 turned out to be the wrong call, it didn't fix that and it broke https://bugzilla.redhat.com/show_bug.cgi?id=811412
• Adamwill - we definitely need to revisit the schedule; we need to find out exactly what has to happen between a 'go' call and the release, figure out how long that actually takes, and move the go/no-go and release readiness meetings to account for it. the day of dead time between go/no-go and release readiness is a complete waste of time
• Kparal - Blocker bug meetings should not be on Friday. We lose a lot of Europe guys presence this way, because it's Friday evening for them (me included). 17.00 UTC is fine, but it must not be the Friday.
• Bruno - The F17 branch of the spin-kickstarts repo kept getting changed during freezes. It would be nice to have a way to stage this during freezes so that the spin-kickstarts package included on the install disk matches what we use for building. This is important for GPL compliance.
• Kparal - Fedora 17 upgrade fail. Both major approaches (DVD, preupgrade) were broken. We need to do better.

Wishlist

• tflink - it would be nice to have a better system for tracking current blockers. While the current wiki-based system is great, it's only updated once per hour and we don't always notice right away when the wiki updates start failing.
• Bruno - I think it would be useful to review the blocker bugs to see if we can spot any trends. Just looking at the component counts might be useful. even if we don't do anything more.
• Bruno - It seemed like install / upgrade testing wasn't getting done early early enough. People (like me) running the branched release don't see these issues in routine use. We may want to find some way to encourage or automate testing on installs and upgrades in order to get earlier warning of problems.
• pschindl - It would be great if there was better communication among the teams (QA, anaconda, FESCo etc.). Nobody knows which features should work and every team has it's own vision about it. Better communication could save us a lot of time spent on arguing.
• pschindl - We should reviewed release criteria, dismiss all which are obsolete and maybe add some new (like we did for PXE). This step should be done in cooperation with other teams (like anaconda and FESCo).
• Robatino - Something needs to be done about the false file conflicts between samba3/samba4 reported by the potential_conflict.py script - see https://lists.fedoraproject.org/pipermail/test/2012-May/108123.html . The script itself lists Seth Vidal and Jesse Keating as authors. I sent email to their fedoraproject.org emails and Seth responded suggesting that I talk to someone in QA. If this is not fixed, it will become difficult to untangle the false conflicts from the real ones during 18 development.
• Robatino - Releases/16/Spins exists, but Releases/17/Spins and Releases/18/Spins do not. These are referred to in QA:Testcase_Mediakit_ISO_Size. I used the 16 page for testing during 17 development.

Recommendations

After enough time has been given for feedback, the QA team will discuss and make recommendations on changes to prioritize for Fedora 18. This section organizes and lists the recommendations.

In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the Fedora 18 roadmap in the QA TRAC instance.

Release Engineering

1. rel-eng ticket#5215 - Stage comps and spin-kickstarts during freezes

   This will avoid the problems with comps and spin-kickstarts being changed during freezes.

Release criteria

1. fedora-qa ticket#297 - Review release criteria for Fedora 18

   Check if any old criteria can be dropped and if any new are required by recent features.

Process

1. fedora-qa ticket#296 - Fix blocker bug tracker page and replace it with a static HTML page

   It got broken by the Bugzilla upgrade, of course we need it fixed.