Fedora 15 QA Retrospective

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 15. The feedback will be used as a basis for identifying areas for improvement for Fedora 15 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

• Gwjasu - I like ____ about the new ____ process

Adding feedback is fairly straight forward. If you already have a Fedora account ...

1. Login to the wiki
2. Select [Edit] for the appropriate section below.
3. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

4. When done, Submit your changes

Otherwise, if you do not have a Fedora account, follow the instructions below ...

1. Select the appropriate page for your feedback...
   • Something that worked well
   • Something that didn't work well
   • Anything on your QA wishlist
2. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

3. When done, Submit your changes

Feedback

Things that went well

1. jlaska - Mailing list - Seeing an increased level of mailing list traffic discussing test issues with systemd and gnome3. This is pretty exciting and exactly the type of communication I'd love to see on the list. Testers helping testers identify and triage issues.
2. jlaska - Beta - Because of the FESCO approval for the late arriving NM-0.9 changes (refer to [https://fedorahosted.org/fesco/ticket/572 FESCO#572), QA held firm that the schedule must slip one week. At the time of the decision, 2 test milestones were missed (pre-beta acceptance, TC1). Attempting to absorb the late changes, and the bugs introduced by those changes, would have negatively impacted the release and definitely resulted in a slip regardless. It was the right decision to slip the release after accepting the late NM-0.9 change.
3. rhe - boot.iso for anaconda updates - it's a great idea to post boot.iso with new anaconda version for testing before composing it to the next TC or RC versions.

Could have been better

1. jlaska - Alpha-TC wasn't branched - F15 Alpha test compose was created using pre-branched content. This was unexpected, but it appears that there is no clear guidance/documentation on what is expected (see https://fedorahosted.org/rel-eng/ticket/4399)
2. jsmith - Mass rebuild not included in schedule - The mass rebuild timing wasn't ideal, it landed at the same time of the branch. While release engineering did an outstanding job resolving rebuild issues in a timely manner, the schedule didn't account for the mass rebuild, and test composes were delayed.
3. jlaska - insufficient live testing - Because of RHBZ #672265 and RHBZ #676904 we didn't get a lot of testing on the live images prior to, and during the Alpha test composes. As a result, once those issues were fixed, we found 3 additional Alpha blockers
   • RHBZ #679107 - TypeError: argument 2 to map() must support iteration
   • RHBZ #663294 - RuntimeError: XOpenDisplay failed }} - TypeError: argument 2 to map() must support iteration
   • RHBZ #672030 - AttributeError: 'NoneType' object has no attribute 'format' }} - TypeError: argument 2 to map() must support iteration
4. jlaska - missed first 2 blocker review meetings - Not sure what else to say ... we forgot about these meetings. With FUDCon and PTO, the first two completely fell off my radar.
5. jlaska - i18n - i18n/l10n continues to highlight problems early on in the release. F-15-Alpha slipped one week due to RHBZ #676827 (keyboard with german layout doesn't work in gdm). We need release criteria for LANG/keymap issues.
6. jlaska - Missing criteria - When proposing and publishing release criteria, use caution when making changes. Particularly when release criteria pages are already available for the next release. You must be sure that any changes made to 42 are copied forward to 43. This was problem for Fedora 15 Alpha with the desktop artwork criteria. The artwork criteria was added to the Fedora 14 Alpha criteria page, but the Fedora 15 Alpha criteria page was already created. The new criteria were not carried forward, and Fedora 15 Alpha released with the incorrect artwork. As a result, a bug was filed, but not escalated.
   • RHBZ #677080 - 'F14' artwork is shown during F-15 installation
   • Also, the f15 design schedule does not have a task to update fedora-logos with Alpha installer artwork.
7. Rhe 06:50, 14 March 2011 (UTC) - VNC test - Add password test for VNC to current test run since the bug was found:
   • RHBZ #678150 - VNC install w/ password, fails to establish password for VNC session
8. vhumpa - GNOME 3 Test Day attendance - for some reason, for this #2 testing event, significantly less testers showed up: 19 compared to over 50 for #1. Still - a nice number of bugs have been encountered, 1.58 bugs per tester at #2 compared to 1.28 at #1, which is due to a larger number of tests this time (25 vs. 16). Although a test day was finally a success, it might be useful to reason about the lower turnout. Here are some ideas, but please note that they are just speculations:
   • Generally, there might have been less simply curious people as they'd already get the curiosity satisfied by #1
   • At the last moment, we wanted to add some of the latest updates that were not yet available in repositories, which caused:
     • We weren't sure if we wouldn't actually want to move the event, so perhaps notifications were sent a little late
       • I have no data on this - but perhaps also less announce channels were used this time
     • We couldn't use the nightly composes as previously planned and had to make custom images - which (as Adamw's sleepless nights confirm) brings trouble and takes time. Particularly the i386 image wasn't available until the late Test Day, which could have driven away some of the testers
   • A significant number of test cases could also have discouraged some testers with less time. We had to divide the tests in two sections/tables and majority of the testers did go through just the first part - which can be seen as a little proof of this.
9. jlaska - Beta - Missed the 2011-03-17 - F-15 pre-beta acceptance milestone (https://fedorahosted.org/rel-eng/ticket/4530)
   • Some nasty glibc change caused an anaconda failure - (see https://bugzilla.redhat.com/show_bug.cgi?id=684742)
   • Delays in landing NetworkManager-0.9 resulted in anaconda build failures (https://fedorahosted.org/fesco/ticket/572)
10. jlaska - Beta - Slipped release 1 week due to decision by FESCO (refer to FESCO#572) to accept late NetworkManager-0.9 changes which impacted *all* desktop environments. QA felt strongly that the right decision was to slip the release one week and not attempt to absorb the changes given that we already missed 2 test milestones (pre-beta acceptance and beta TC1). Could the late arriving changes have been seen/predicted earlier? Ticket filed on 03/10/11
11. Rhe - btrfs - Btrfs file system was added for partitioning, need make sure if it's supported and then add test case for it.
12. We had a complaint that "Release Candidate" was confusing when applied to Alpha and Beta releases. A suggestion was made to start calling these "Alpha Candidate" and "Beta Candidate".
13. AdamW - the introduction of the /run directory was done very late in Beta timeframe, and this caused most of the trouble we had getting the Beta out on time. The impact of the change was somewhat misrepresented (at first devel list was told that /var/run would not be made a symlink to /run in F15 timeframe, but in fact it *is* being made a symlink on new installations).
14. jlaska - 'Install ISOs for updates-testing' - building custom boot.iso images to supply anaconda karma seems broken. This should be enabled by default. Why aren't we also providing nightly updates-testing live images for consumption?
15. Rhe - nfsiso repo - Clumens recommended that the nfsiso repository should only have the correct ISOs the anaconda points to instead of a set of different isos. If so, the test case/documentation need be modified to reflect it. See RHBZ #695280.

Wishlist

1. AdamW - More package specific test cases - We need more package specific test cases (NetworkManager and kernel).
2. robatino - Proposed regular naming scheme for blocker and NTH tracker bugs (https://fedoraproject.org/wiki/Talk:BugZappers/HouseKeeping/Trackers)
3. robatino - Possibility of using "Install" instead of "Fedora" as the directory name for the install images (http://lists.fedoraproject.org/pipermail/devel/2011-April/150370.html and https://fedorahosted.org/rel-eng/ticket/4654)