Fedora 15 QA Retrospective

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 15. The feedback will be used as a basis for identifying areas for improvement for Fedora 15 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

Adding feedback is fairly straight forward. If you already have a Fedora account ...

1. Login to the wiki
2. Select [Edit] for the appropriate section below.
3. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

4. When done, Submit your changes

Otherwise, if you do not have a Fedora account, follow the instructions below ...

1. Select the appropriate page for your feedback...
   • Something that worked well
   • Something that didn't work well
   • Anything on your QA wishlist
2. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

3. When done, Submit your changes

Feedback

Things that went well

1. jlaska - Mailing list - Seeing an increased level of mailing list traffic discussing test issues with systemd and gnome3. This is pretty exciting and exactly the type of communication I'd love to see on the list. Testers helping testers identify and triage issues.
2. jlaska - Beta - Because of the FESCO approval for the late arriving NM-0.9 changes (refer to [https://fedorahosted.org/fesco/ticket/572 FESCO#572), QA held firm that the schedule must slip one week. At the time of the decision, 2 test milestones were missed (pre-beta acceptance, TC1). Attempting to absorb the late changes, and the bugs introduced by those changes, would have negatively impacted the release and definitely resulted in a slip regardless. It was the right decision to slip the release after accepting the late NM-0.9 change.
3. rhe - boot.iso for anaconda updates - it's a great idea to post boot.iso with new anaconda version for testing before composing it to the next TC or RC versions.

Could have been better

1. jlaska - Alpha-TC wasn't branched - F15 Alpha test compose was created using pre-branched content. This was unexpected, but it appears that there is no clear guidance/documentation on what is expected (see https://fedorahosted.org/rel-eng/ticket/4399)
2. jsmith - Mass rebuild not included in schedule - The mass rebuild timing wasn't ideal, it landed at the same time of the branch. While release engineering did an outstanding job resolving rebuild issues in a timely manner, the schedule didn't account for the mass rebuild, and test composes were delayed.
3. jlaska - insufficient live testing - Because of RHBZ #672265 and RHBZ #676904 we didn't get a lot of testing on the live images prior to, and during the Alpha test composes. As a result, once those issues were fixed, we found 3 additional Alpha blockers
   • RHBZ #679107 - TypeError: argument 2 to map() must support iteration
   • RHBZ #663294 - RuntimeError: XOpenDisplay failed }} - TypeError: argument 2 to map() must support iteration
   • RHBZ #672030 - AttributeError: 'NoneType' object has no attribute 'format' }} - TypeError: argument 2 to map() must support iteration
4. jlaska - missed first 2 blocker review meetings - Not sure what else to say ... we forgot about these meetings. With FUDCon and PTO, the first two completely fell off my radar.
5. jlaska - i18n - i18n/l10n continues to highlight problems early on in the release. F-15-Alpha slipped one week due to RHBZ #676827 (keyboard with german layout doesn't work in gdm). We need release criteria for LANG/keymap issues.
6. jlaska - Missing criteria - When proposing and publishing release criteria, use caution when making changes. Particularly when release criteria pages are already available for the next release. You must be sure that any changes made to 42 are copied forward to 43. This was problem for Fedora 15 Alpha with the desktop artwork criteria. The artwork criteria was added to the Fedora 14 Alpha criteria page, but the Fedora 15 Alpha criteria page was already created. The new criteria were not carried forward, and Fedora 15 Alpha released with the incorrect artwork. As a result, a bug was filed, but not escalated.
   • RHBZ #677080 - 'F14' artwork is shown during F-15 installation
   • Also, the f15 design schedule does not have a task to update fedora-logos with Alpha installer artwork.
7. Rhe 06:50, 14 March 2011 (UTC) - VNC test - Add password test for VNC to current test run since the bug was found:
   • RHBZ #678150 - VNC install w/ password, fails to establish password for VNC session
8. vhumpa - GNOME 3 Test Day attendance - for some reason, for this #2 testing event, significantly less testers showed up: 19 compared to over 50 for #1. Still - a nice number of bugs have been encountered, 1.58 bugs per tester at #2 compared to 1.28 at #1, which is due to a larger number of tests this time (25 vs. 16). Although a test day was finally a success, it might be useful to reason about the lower turnout. Here are some ideas, but please note that they are just speculations:
   • Generally, there might have been less simply curious people as they'd already get the curiosity satisfied by #1
   • At the last moment, we wanted to add some of the latest updates that were not yet available in repositories, which caused:
     • We weren't sure if we wouldn't actually want to move the event, so perhaps notifications were sent a little late
       • I have no data on this - but perhaps also less announce channels were used this time
     • We couldn't use the nightly composes as previously planned and had to make custom images - which (as Adamw's sleepless nights confirm) brings trouble and takes time. Particularly the i386 image wasn't available until the late Test Day, which could have driven away some of the testers
   • A significant number of test cases could also have discouraged some testers with less time. We had to divide the tests in two sections/tables and majority of the testers did go through just the first part - which can be seen as a little proof of this.
9. jlaska - Beta - Missed the 2011-03-17 - F-15 pre-beta acceptance milestone (https://fedorahosted.org/rel-eng/ticket/4530)
   • Some nasty glibc change caused an anaconda failure - (see https://bugzilla.redhat.com/show_bug.cgi?id=684742)
   • Delays in landing NetworkManager-0.9 resulted in anaconda build failures (https://fedorahosted.org/fesco/ticket/572)
10. jlaska - Beta - Slipped release 1 week due to decision by FESCO (refer to FESCO#572) to accept late NetworkManager-0.9 changes which impacted *all* desktop environments. QA felt strongly that the right decision was to slip the release one week and not attempt to absorb the changes given that we already missed 2 test milestones (pre-beta acceptance and beta TC1). Could the late arriving changes have been seen/predicted earlier? Ticket filed on 03/10/11
11. Rhe - btrfs - Btrfs file system was added for partitioning, need make sure if it's supported and then add test case for it.
12. We had a complaint that "Release Candidate" was confusing when applied to Alpha and Beta releases. A suggestion was made to start calling these "Alpha Candidate" and "Beta Candidate".
13. AdamW - the introduction of the /run directory was done very late in Beta timeframe, and this caused most of the trouble we had getting the Beta out on time. The impact of the change was somewhat misrepresented (at first devel list was told that /var/run would not be made a symlink to /run in F15 timeframe, but in fact it *is* being made a symlink on new installations).
14. jlaska - 'Install ISOs for updates-testing' - building custom boot.iso images to supply anaconda karma seems broken. This should be enabled by default. Why aren't we also providing nightly updates-testing live images for consumption?
15. Rhe - nfsiso repo - Clumens recommended that the nfsiso repository should only have the correct ISOs the anaconda points to instead of a set of different isos. If so, the test case/documentation need be modified to reflect it. See RHBZ #695280.
16. Adamw - BetaNag - https://bugzilla.redhat.com/show_bug.cgi?id=703815#c8, we should have criteria that include removal of the betanag for Final release.
17. Rhe - test case update - initrd.img is not in gzip format from F-15-Final-RC1, thus the QA/TestCases/KickstartKsFilePathKsCfg is not applicable to include ks.cfg in it. Test case need be updated.
18. jlaska - F-15-Final-RC1 blockers -
    • RHBZ #704030 - package conflicts in 15 Final RC1 - the generic* packages made it into RC1, resulting in broken dependencies. How'd this get into RC1, and no other interim milestone tree? adamw: releng inadvertently left the necessary exclude statements out of the repo definition when transferring it from tc1 to rc1 build
    • RHBZ #704042 - X fails to start after firstboot in 15 Final RC1 - dconf-0.7.5-1 from updates-testing appears to fix it - root cause? How did dconf-0.7.4-1 get into RC1, how did dconf-0.7.5-1 not get into RC1? adamw: 0.7.4 landed before freeze, and got positive karma: the bug is only apparent when tested with a new user account, so it's not surprising that no-one hit it in normal 'update and see if everything works' testing. A bug was filed but not proposed as a blocker, and the 0.7.5 update was not marked as fixing any particular bug, which made it a bit harder to see how important it was at a casual glance. Probably reasonable to think desktop team might have marked this as blocker: we could talk to them about flagging up significant fixes like this
    • RHBZ #699113 - During boot, message 'mkdir: cannot create directory /run, file exists' appears on virt. console 1 - root cause? Could this have been discovered (or resolved) earlier? adamw: the actual blocker here was caused by the *fix* for this bug: the fix to make the /run error go away was incomplete and caused live boots to fail. Could probably have been discovered earlier as nightlies were failing this way for a while. May be one where we should be stricter on NTH; the error was entirely cosmetic (though likely to confuse users, which is why we accepted it as NTH)
    • RHBZ #704415 - abrt-dump-oops breaks live media creation - root cause? Could this have been discovered (or resolved) sooner? adamw: root cause - badly-written rpm scriptlet. like most of the others, this was not noticeable in typical update-and-use testing, only noticeable in composing live images. i think we actually tracked this one down reasonably quickly given the subtlety of the bug
19. jdulaney - Glibc update dropping support for dep that is needed by multiple packages early in the Final process, causing all manner of trouble. This should not happen. Maybe setup a process that must be completed before some feature that may be a dependency is removed. Does such a thing exist?
20. jdulaney - In Bugzilla I haven't got permissions to set blocker status. This came up as a hindrance during the F15 process.
21. jlaska - Bugs missed ...
    • RHBZ #706122 - USB overlay (persistent storage) not being mounted - Use of overlay not explicitly called out in any test matrix
    • RHBZ #706542 - AttributeError: 'NoneType' object has no attribute 'rfind' - FIXME
22. pbrobinson - NetworkManager 0.9 was a major change to multiple features/desktops and should have been feature, and landed prior to the Feature Freeze and advertised appropriately. Instead it landed just before the beta and was only discussed on a non advertised bug in a fedora hosted trac instance (sorry can't remember which one). It also affected anaconda (points 9/10 above). Changes like this need to be landed as per policies or pushed to the next release, or what is the point of having policies if core components are allowed to completely ignore them!
23. Bruno Wolff - There were a few 3+ hour blocker review meetings. It's hard to get people to volunteer to participate in those.
24. Rhe - Test for pre-release screen check - There should be a test case(and release criteria) checking pre-release warning screen removed or not from installer before it's released. Propose the priority to 'Final'.
25. Rhe - Site Error - Site Error on Wiki result page saying 'Cite error: Invalid < ref > tag; name cannot be a simple integer. Use a descriptive title'.

    This is due to the recent mediawiki upgrade. I saw this error in staging, and changed the template in staging, but completely forgot to transition that change to production. Well, I thought I did, but I must have changed the template in staging instead. Thanks for fixing!

Wishlist

1. AdamW - More package specific test cases - We need more package specific test cases (NetworkManager and kernel).
2. robatino - Proposed regular naming scheme for blocker and NTH tracker bugs (https://fedoraproject.org/wiki/Talk:BugZappers/HouseKeeping/Trackers)
3. robatino - Possibility of using "Install" instead of "Fedora" as the directory name for the install images (http://lists.fedoraproject.org/pipermail/devel/2011-April/150370.html and https://fedorahosted.org/rel-eng/ticket/4654)
4. robatino - I'd like to implement https://fedorahosted.org/fedora-infrastructure/ticket/2241 by keeping the disos for TCs/RCs at http://alt.fedoraproject.org/pub/alt/stage/deltaisos/ indefinitely instead of deleting them after each development cycle. I have disos for all the F14 TCs and RCs which in total take about 7 GB. The filesystem where the disos live is the same one used by the stable release archive, and currently has about 6 TB available out of 9 TB total. The elimination of CDs in F15 frees up roughly the same amount of space as needed by the disos. So it seems that space should not be a problem. The disos between Alpha/Beta/Final can be deleted eventually since they are not necessary for recovering the TCs/RCs. As long as there is one diso for each TC/RC from an earlier one, all of them are retrievable. Update: Implemented with archive/ subdirectory (not mirrored so visible only on alt). Is it possible that having a large set of install images with known problems (from the test matrices) could help in developing automated tools for testing these images?
5. jlaska - Add artwork test case to the desktop and installation matrix?
6. jlaska - fedora-release - We always seem to forget to request an updated fedora-release package near the end of the release. It's in the SOP for rel-eng to update fedora-release, but that doesn't always remove the betanag. We should add a test to confirm the betanag is removed for final, and another test to confirm the betanag is present anywhere else.

Recommendations

After enough time has been given for feedback, the QA team will discuss and make recommendations on changes to prioritize for Fedora 16. This section organizes and lists the recommendations.

In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the Fedora 16 roadmap in the QA TRAC instance.

Release Engineering

1. Create fedora-qa ticket - Update rel-eng 'test composes' SOP to provide live images

   Live images must be included in Test Composes and Acceptance runs, for the past two releases we didn't actively test live image creation and use until TC1, which often resulted in discovering and fixing bugs late.

2. fedora-qa ticket#220 - Review RHBZ #704030 with rel-eng to identify cause and make appropriate SOP changes to ensure issue is not repeated

   RHBZ #704030 resulted in F-15-Final-RC1 being broken on arrival

   Bug occurred due to miss-paste; it should not occur again.

3. Create fedora-qa ticket - Decide on milestone naming format and update appropriate documentation

   This is related to the concern about the terminology Release Candidate

   One proposal suggested Alpha Candidate, Beta Candidate, and Final Candidate

Test Coverage

1. Create fedora-qa ticket - Update QA acceptance test plan to include live image tests

   See also #alpha-live-image

2. fedora-qa ticket#210 - Update existing VNC test case, or create new tests, to test vncpassword= and vncconnect=

   No explicit testing for VNC password or vncconnect was performed and an issue was encountered related to vncpasswd not working (RHBZ #678150)

3. fedora-qa ticket#209 - Add tests to cover partitioning using root on btrfs, xfs and other non-ext fstype's

   There are no tests to cover partitioning using non-ext file systems, such as btrfs/xfs etc..

4. fedora-qa ticket#193 - Update HD-ISO and NFS-ISO test instructions to use *only* the ISO image under testing

   The installer is sensitive when doing HD-ISO or NFS-ISO installs when multiple ISO images are present. Update test instructions to include *only* the desired ISO image when testing

5. fedora-qa ticket#208 - Update ks=file:/ks.cfg to provide correct instructions for use with LZMA

   Due to a late lorax change, the initrd.img is no longer a gzip compressed CPIO ball. Instead it is now a LZMA compressed CPIO ball.

   $ lzma --format=xz -c -d /media/images/pxeboot/initrd.img | cpio -id

6. Create fedora-qa ticket - Review RHBZ #704042 to determine whether dconf test cases are needed to account for "test with new user account"

   RHBZ #704042 affected the final release criteria, an update that resolved the bug was available in updates-testing, but it was not identified as a critical update.

7. fedora-qa ticket#204 - Add artwork (installer, firstboot and desktop background) artwork tests designed to validate artwork release criteria

   We have artwork related release criteria, but there are no artwork related test cases. Once created, perhaps add this to the desktop and installation matrix, or create a new artwork test matrix?

8. fedora-qa ticket#151 - Current install and desktop test matrices don't fully cover release criteria

   Identify test gaps, this would be release criteria for which there is not explicit test coverage in either the desktop or install matrix

   Consider creating an additional test matrix to account for non-desktop and non-install related criteria (artwork?)

Release Criteria

1. Create fedora-qa ticket - Review i18n/l10n release criteria to ensure properly handling for RHBZ #676827

   For several releases now it's been unclear how i18n/l10n bugs should affect the release criteria. We need someone to propose i18n/l10n release criteria. Ideally, something that starts somewhat relaxed, but gets more specific and restrictive for the Beta and Final releases.

2. Create fedora-qa ticket - Draft and propose criteria to handle betanag removal for the final release

   There are no criteria to cover the removal of the betanag for the Final release ... while it's an obvious needed change, it's not clear how to escalate this issue as it does not impact the current criteria in any way.

3. Create fedora-qa ticket - Discuss possible criteria adjustments (and test matrix additions) to account for USB overlay

   USB overlay (persistent storage) was not mounted with Fedora 15 live media.

Schedule

1. Create fedora-qa ticket - Review design schedule for Alpha installer/firstboot artwork coverage

   Design schedule does not have a task to provide Alpha artwork. It wasn't obvious that non-F14 installer artwork was needed at the Alpha. Discuss with the design-team and rbergeron if any changes are needed on the schedule to ensure coverage.

2. Create fedora-qa ticket - The Fedora 15 mass-rebuild was not on the schedule

   The rebuild accounted for delays in generating the Alpha Test compose images, which contributed to a slip. What is needed to ensure mass rebuilds are identified and planned for ahead of time?

3. Create fedora-qa ticket - Alpha continues to slip each release

   The Alpha TC1 and the Rawhide branch dates are on the same day ... this has led to a slip for the last 2 Alpha's

   Should we frontload more testing prior to the TC1? For example, should the pre-alpha acceptance test runs prior to the Branch be more thorough?

   Is there value in testing prior to the branch date ... won't everything just completely change/break?

Process

1. fedora-qa ticket#221 - Brainstorm for ways to reduce the blocker review meeting length

   The meetings are required, effective but far too long. How can we encourage more offline/asynchronous bug processing

2. RHBZ #707252 - Work with Red Hat Bugzilla team to allow users to edit the whiteboard and bug dependency information for Fedora bugs

   Special permissions are needed by bugzilla to change the whiteboard or blocker fields. This restriction wasn't known and affects who can escalate blocker bugs for review in Fedora.

3. Create fedora-qa ticket - Discuss RHBZ #704042 with desktop-team to ensure understanding of bug escalation procedures and release criteria

   RHBZ #704042 affected the final release criteria, an update that resolved the bug was available in updates-testing, but it was not identified as a critical update.

4. Create fedora-qa ticket - Review proposed blocker and NTH alias changes, and adjust SOP's accordingly

   Robatino recommended a more standard format for blocker and NTH tracker bugs (see https://fedoraproject.org/wiki/Talk:BugZappers/HouseKeeping/Trackers)