Fedora 19 QA Retrospective

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 19. The feedback will be used as a basis for identifying areas for improvement for Fedora 19 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

• Gwjasu - I like ____ about the new ____ process

Adding feedback is fairly straight forward. If you already have a Fedora account ...

1. Login to the wiki
2. Select [Edit] for the appropriate section below.
3. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

4. When done, Submit your changes

Anonymous

If you do not have a Fedora account, follow the instructions below to submit anonymous feedback. Please note, mediawiki records the IP address associated with any anonymous page edits.

1. Select the appropriate page for your feedback...
   • Something that worked well
   • Something that didn't work well
   • Anything on your QA wishlist
2. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

3. When done, Submit your changes

Feedback

Things that went well

• Adamwill (talk) - Closest release to planned schedule ever(?) Our efforts to start testing earlier and be more aggressive in finding all blockers as early as possible seem to be helping.
• Adamwill (talk) - Testing load seems to be spreading out again after being concentrated on a few people for some recent releases.

Could have been better

• Adamwill (talk) - We forgot to test Secure Boot until very late.
• Adamwill (talk) - We nearly missed a significant blocker in text mode due to not having enough coverage for it. It's nearly as fully-featured as GUI mode now: we need to find a way to test it more extensively.
• Adamwill (talk) - Test cases are starting to bit rot, many are written around old components or oldUI.
• Adamwill (talk) - Some release criteria are missing enforcing test cases.
• nonamedotc - Anaconda text install test cases could be more elaborate.
• mattdm (talk) - Since cloud images are supposed to be first-class citizens this release, cloud sig should have worked with qa to actually make release criteria for the images.
• nonamedotc - Will it be possible to have the QA meetings at two times on alternate weeks? Week 1 - current time, week 2 - evening, week 3 - current time and so on.
• Adamwill (talk) - we may have made the wrong FE call on #974543, that was a fairly large behaviour change and it landed very late. See #979205
• Bruno - It was a pain to try to build a correct spin-kickstarts package when the repo changed after the commit we wanted to appear in the package. This has been fixed for f20. This also makes it easier to have a private branch for use with builds. When the private branch is merged back in we can tag the appropriate commit and use it for a build without worrying about other changes that happened during that time.
• Bruno - It would have been nice to have gotten a tc for all of the spins at the beginning of the freeze. I know rel-eng isn't too happy with the spin situation and don't want to spend a lot of extra resources on them, but just doing the minor spins at rc time gives little chance to recover from problems and cost us the scientific spin iso this time.

Wishlist

• Adamwill (talk) - We're only ever going to get more and more drudge tests, boring tests we have to run: we really need to get some automation going to help with comprehensive test coverage.

Recommendations

After enough time has been given for feedback, the QA team will discuss and make recommendations on changes to prioritize for Fedora 19. This section organizes and lists the recommendations.

In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the Fedora 19 roadmap in the QA TRAC instance.

Process

1. fedora-qa ticket#395 - Consider how to improve testing of different installer interfaces

   Related to the 'text interface' feedback items: how can we test it better?

2. fedora-qa ticket#396 - Revise partitioning test cases to reflect newUI design and cover its intended use cases

   Partitioning test cases are among the most 'bit-rotten'

3. fedora-qa ticket#397 - Create test cases for release criteria that are currently missing them
4. fedora-qa ticket#398 - Add Secure Boot test case