From Fedora Project Wiki

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 16. The feedback will be used as a basis for identifying areas for improvement for Fedora 16 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

Adding feedback is fairly straight forward. If you already have a Fedora account ...

  1. Login to the wiki
  2. Select [Edit] for the appropriate section below.
  3. Add your feedback using the format: 
    * ~~~ - I like ____ about the new ____ process
  4. When done, Submit your changes

Anonymous

If you do not have a Fedora account, follow the instructions below to submit anonymous feedback. Please note, mediawiki records the IP address associated with any anonymous page edits.

  1. Select the appropriate page for your feedback...
  2. Add your feedback using the format: 
    * ~~~ - I like ____ about the new ____ process
  3. When done, Submit your changes

Feedback

Things that went well

Could have been better

  1. jlaska - Pre-release acceptance - the Rawhide Acceptance milestones continue to sneak up on us ... and it takes a bit to get images lined up. It's better this release with twu owning the QA side of the process, but it still feels like there is more room for transparency/communication around the process.
  2. adamw - when there are showstoppers in anaconda early we tend to just sit around and wait for them to get fixed, but without much urgency; which means we're getting no idea of what bugs are lurking behind the showstoppers, and we wind up trying to fix a lot of blockers in a small amount of time once the showstoppers are finally fixed
  3. jlaska - Alpha TC1 - Missed the Alpha-TC1 milestone by 1 week, little to no communication regarding status. With only two weeks Alpha ISO test time, losing 50% of time almost certainly results in a slip. See rel-eng ticket#4844 . None of the Alpha rel-eng release tickets have been filed at this time.
    • adamw - perhaps releng image compose SOP should require updating of ticket with progress info, especially when image compose / delivery is delayed?
  4. jlaska - Alpha branch - Unclear when the F16 branch was completed as not all tasks in the Mass_Branching_SOP were completed. The SOP either needs to be followed or updated.
    • MirrorManager was not updated to add a fedora-16 repo tag, therefore, TC1 installations could not find a repository to install from (RHBZ #727428)
    • TC1 images were built with Version=16-Alpha.TC1 specified as the --version argument to pungi. This results in a file /.buildstamp in the install image with an incorrect version (Version=16-Alpha.TC1) that is used as the $releasever variable for yum. The net result is that there are no yum repos that match repo=16-Alpha.TC1&arch=x86_64. The version needs to be an integer, or MirrorManager redirects need to be established prior to compose.
    • The fedora-release package was not properly updated to include [fedora] with enabled=1. As a result, TC1 images included a fedora-release package that didn't enable the Fedora 16 repos.
    • The fedora-release package was not updated to include the Fedora 16 release name Verne. This is a minor issue, but results in login prompts claiming the system is Rawhide. Recommend updating (or creating) the existing Mass_Branching_SOP to document expected results when updating fedora-release
    • adamw - seems branching may not actually have been completed by time of TC1 delivery, even. Perhaps appropriate releng SOPs should be updated to clarify that branching - or at least some specified subset of branching tasks - must be completed before Alpha TC1 delivery
  5. jlaska - Alpha RC2 - RC2 was composed to resolve the following blocker bugs...
  6. jlaska - Alpha RC3 - RC3 was composed to resolve the following issues ...
  7. jlaska - Alpha RC4 - RC4 was composed to resolve the following issues ...
  8. jlaska - Alpha RC5 - RC5 was composed to resolve the following issues ...
  9. adamw - it wasn't an A+ idea for both rpm maintainers to be on vacation at the same time, with no cover in place, during a critical freeze period
  10. adamw - when we hit an obvious showstopper we tend to focus in on it exclusively until it's fixed, iterate, and then be surprised when there are bugs hidden behind it; we should work harder to try and workaround showstoppers in a way that has as small an impact as possible, and continue testing, to avoid this problem. e.g. RHBZ #730863 hid behind RHBZ #729563, but we could have exposed it by editing /etc/selinux/config in the installed system prior to rebooting from the installer
  11. robatino - Alpha and Beta torrents often have unsigned checksum files - this just happened with 16 Alpha. It also happened during F13 and F14 - see http://lists.fedoraproject.org/pipermail/test/2010-April/090106.html for F13 Beta, and https://bugzilla.redhat.com/show_activity.cgi?id=638738 for F14 Alpha and Beta. When it's pointed out, we're told that it's not feasible to change them once people have started downloading, but if this is true, there needs to be better checking (at least as much as for mirror content, which can be changed) to make sure the torrents are correct before being posted. Filed https://fedorahosted.org/fedora-qa/ticket/237 and https://fedorahosted.org/rel-eng/ticket/4906 . No response from releng as of yet. Unless QA is given access prior to public posting, it is powerless to prevent this.
  12. robatino - Also, it's possible that the checksum files can be signed more than once. There are two versions of the signed checksum files for F15, one on the torrents (the original ones) and a different one on the mirrors. See http://robatino.fedorapeople.org/checksums/15-Final/ for the two versions - for example, Fedora-15-i386-CHECKSUM.orig and Fedora-15-i386-CHECKSUM. While not nearly as serious as the above issue (since the checksums are the same and both signatures are good), it could be confusing.
  13. adamw - we completely and utterly whiffed on EFI testing for Alpha and Beta. Need to sort things out so at least two QA team members have EFI capable hardware and test EFI installs at all release points.
  14. adamw - NFS testing instructions may need to be more precise to avoid pilot errors, and may need to account for NFSv4 quirks.
  15. adamw - need to do more direct personal pinging of maintainers who seem unresponsive to blockers; some respond when contacted directly but do not appear to place a high priority on bug reports
  16. adamw - Install test plan was mostly neglected this cycle; might be a useful framework for working out plans for stuff like grub2. in f17 we could use it for anaconda re-design
  17. tflink - We really didn't do a whole lot with EC2 testing for F16. Since EC2 issues are now release blocking issues, we should probably be coordinating something or working with the Cloud SIG to make sure that the testing gets done.
  18. mkrizek - We should have tested if transition between SysV and systemd init scripts went fine. For instance cups was not enabled when upgrading from F15.

Wishlist

  1. robatino - Proposed regular naming scheme for blocker and NTH tracker bugs (https://fedoraproject.org/wiki/Talk:BugZappers/HouseKeeping/Trackers)
  2. adamw - we need to extend our coverage of various methods of booting the installer: we need to check all the combinations of EFI / BIOS boot with installer written to spinning disc or USB, using dd or livecd-iso-to-disk or liveusb-creator. We may need to do this by extending the columns in the matrix rather than by adding test cases; so we'd have more results columns for some of the test cases. Particular problem cases that came up in F16 were different bugs in EFI booting with different images, and various issues that show up when you boot the installer from USB rather than a spinning disc.
  3. tflink - Maybe it was due to the high number of TC/RC spins that we did for F16 but it would be awesome to have some more automation around image generation for testing. I'm not talking about any rel-eng processes but it would be nice to be able to request generation of a boot.iso made from custom packages and packages in koji in order to do testing. I can see this being helpful for test day coordination as well but there are issues with complexity and hosting so this is a bit of a "pony list" item.
  4. tflink - Since qemu/kvm VMs are all legacy BIOS emulation, it would be awesome to do EFI emulation. There are some projects out there on using EFI in qemu VMs but it is unknown whether they actually work. Either way, it might be good to look into them in order to evaluate whether their use is practical
  5. adamw - wider upgrade coverage: upgrade with KDE installed and EFI upgrade are two areas that probably need testing
  6. cwickert - wider upgrade coverage: Test all supported grub locations, not only MBR but also first sectors of /boot
  7. cwickert - ChangeLog in the wiki to seee which packages were updated and which bugs are supposed to be fixed by a new compose.
  8. cwickert - More testers for spins, otherwise it's hard to catch up with the speed the new composes are done.
  9. adamw - might need improved / more Xen test cases

Recommendations

After enough time has been given for feedback, the QA team will discuss and make recommendations on changes to prioritize for Fedora 17. This section organizes and lists the recommendations.

In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the Fedora 17 roadmap in the QA TRAC instance.

Release Engineering

  1. rel-eng ticket#4985 - Extend test image creation SOP to require notification of delays
    This will mean that at least everyone knows why a TC/RC drop is delayed, and what the ETA is
  2. rel-eng ticket#4986 - Create a 'Deliverables' page / SOP listing exactly what images, torrent files, signatures, checksums etc should be provided with (pre)-releases
    This should ensure consistency and correctness in what images are generated at what release points, and the metadata that accompanies them
  3. rel-eng ticket#4994 - Make it clear in releng SOPs that branching must be complete prior to Alpha compose
    This should address the various issues we had in Alpha TC1 caused by branching tasks not being completed prior to the compose

Test Coverage

  1. fedora-qa ticket#256 - Add EFI data to installation / base results matrices
    We need to get more EFI testing done; the ticket notes this can be done either as added test cases or a re-jigging of the results matrix
  2. fedora-qa ticket#258 - Create EC2 test cases and integrate into validation matrices
    This is an increasingly important use case now in the criteria, which we should cover
  3. fedora-qa ticket#260 - Cover USB-written images better in installation validation testing
    Writing images to USB is becoming more and more popular and we hit various issues with it
  4. fedora-qa ticket#261 - Revise upgrade test case set
    We don't really cover enough upgrade scenarios; we can never cover them all, but we should cover more
  5. fedora-qa ticket#263 - Expand Xen test case set
    Xen is now in the release criteria and has increased interest. Reached out to interested parties to see if we should add more test cases; response was positive, so ticket is now open with details of the additional tests that would help

Brought forward from Fedora_15_QA_Retrospective

  1. fedora-qa ticket#204 - Add artwork (installer, firstboot and desktop background) artwork tests designed to validate artwork release criteria
    We have artwork related release criteria, but there are no artwork related test cases. Once created, perhaps add this to the desktop and installation matrix, or create a new artwork test matrix?
  2. fedora-qa ticket#151 - Current install and desktop test matrices don't fully cover release criteria
    Identify test gaps, this would be release criteria for which there is not explicit test coverage in either the desktop or install matrix. This is done for Alpha and Beta

Release criteria

Brought forward from Fedora_15_QA_Retrospective

  1. fedora-qa ticket#262 - Draft and propose criteria to handle betanag removal for the final release
    There are no criteria to cover the removal of the betanag for the Final release ... while it's an obvious needed change, it's not clear how to escalate this issue as it does not impact the current criteria in any way.

Schedule

  1. Drop most RATS points and replace them with earlier TC drops
    This was done experimentally for F16 Beta and Final and seemed to produce positive results. Already discussed in a mailing list thread and sent as a request to rbergeron

Brought forward from Fedora_15_QA_Retrospective

  1. Create design-team ticket - Review design schedule for Alpha installer/firstboot artwork coverage
    Design schedule does not have a task to provide Alpha artwork. It wasn't obvious that non-F14 installer artwork was needed at the Alpha. Discuss with the design-team and rbergeron if any changes are needed on the schedule to ensure coverage.
    Mail sent to mairin, rbergeron and jlaska 2011-11-29 to clarify current status of this item: if action is needed, will file a design-team trac ticket

Process

  1. fedora-qa ticket#257 - Create and maintain F17 install test plan
    We neglected the install test plan for F16 after rhe left, need to maintain it for F17
  2. fedora-qa ticket#259 - Review Fedora 17 feature list and co-ordinate with owners of significant features to create test plans
    This should help us avoid problems like the ones we had with the SysV -> systemd service transition in F16
  3. fedora-qa ticket#273 - Improve browseability of validation matrix pages
    Make it easier to find the validation results page you need to look at
  4. Provide notification of changes between TCs and RCs
    Already completed

Brought forward from Fedora_15_QA_Retrospective

  1. Create fedora-qa ticket - Review proposed blocker and NTH alias changes, and adjust SOP's accordingly
    Robatino recommended a more standard format for blocker and NTH tracker bugs (see https://fedoraproject.org/wiki/Talk:BugZappers/HouseKeeping/Trackers)
    Asked Andre 2011-11-29 to bring his proposal to test mailing list where it is more prominent, if it's accepted, will file a ticket to track implementation

References

Retrieved from "https://fedoraproject.org/w/index.php?title=Fedora_16_QA_Retrospective&oldid=291591"