Fedora 35 QA Retrospective

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 35. The feedback will be used as a basis for identifying areas for improvement for Fedora 36 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to test@lists.fedoraproject.org.

Providing feedback

• Gwjasu - I like ____ about the new ____ process

Adding feedback is fairly straight forward. If you already have a Fedora account ...

1. Login to the wiki
2. Select [Edit] for the appropriate section below.
3. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

4. When done, Submit your changes

Anonymous

If you do not have a Fedora account, follow the instructions below to submit anonymous feedback. Please note, mediawiki records the IP address associated with any anonymous page edits.

1. Select the appropriate page for your feedback...
   • Something that worked well
   • Something that didn't work well
   • Anything on your QA wishlist
2. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

3. When done, Submit your changes

Feedback

Things that went well

• adamwill - We slipped both Beta and Final appropriately and did not rush the releases.
• adamwill - The final RC was built plenty of time before Go/No-Go, allowing for a good amount of testing.
• adamwill - We were able to resolve all the KDE-related blockers with good co-operation from Fedora and upstream KDE folks, especially Aleix Pol who resolved a lot of the Discover blockers.
• Jpbn (talk) I like the James Laska's old list of leading questions to prompt feedback, because it's still pretty good: it shows the way to improve my knowledge about the test proces.
• Ahmedalmeleh - I only started contributing to the QA team for Fedora 35 Final, I enjoyed submitting my viewpoints in IRC and joining the Go/No-go meeting. I know how to use the blocker bugs tool and Bodhi quite well (Fedora Updates System). I am glad we took our time to perfect the Final Fedora 35 release and overcame any obstacles (blocker bugs) that stood in our way. By the way I used some of "James Laska's old list of leading questions to prompt feedback" to answer some of this, it still works.
• Mattdm (talk) - Overall release is very solid and well-received. I feel confident that we're doing a good job in making sure upgrades (and new install first impressions) are pleasant for users.
• sumantrom (talk) - Fedora CoreOS aligned with our release cycle and we got time to capture bugs in the test week. Read More
• sumantrom (sumantrom) - There is a growth in QA testers in terms of test days. The test days app hasn't gone down even ONCE *KUDOS frantizek*

Could have been better

• adamwill - It took a long time to work through all the iterations of the fedora-third-party/SELinux bug. Seems like we could have tightened the loop on that somehow.
• adamwill - Movement on the KDE blockers was initially slow, if we could have resolved them earlier we would have had more time to find and fix the bugs that eventually delayed Final.
• Ahmedalmeleh - Bugzilla is challenging to me and still getting used to it. I wish I was able to learn how to operate OpenQA's automated tests, in the given timeframe and was given guidance sooner.
• kparal - After F34, we talked in our team about doing a selected compose testing in a complete full run (all test cases), way ahead of the milestones (Beta and Final), and as a group effort to encourage people to really fill out all test cases. Yet we never got to it in F35. Some of the bugs might've been discovered earlier this way.
• Mattdm (talk) - Seconded on the KDE blockers. As I understand it, KDE SIG is working on getting KDE upstream folks directly involved and aware of potentially-blocking issues sooner, which will really help.
• Mattdm (talk) - The timing of KDE Plasma releases is not well-aligned with our schedule. We were trying to stabilize with 5.22, when upstream was really on to 5.23 (which is now in our updates-testing.) With F36, we're probably okay, with the 5.24 release coming the same day branch and continuing with 5.24.z bugfixes right through shortly after our release day. But with F37, 5.26.0 drops just a week before our "Early' Final target date.
• Mattdm (talk) - Lots of people (as non-scientifically measured by my personal impressions of help requests I'm seeing) are hitting the wireplummer-not-enabled problem. Maybe we should have blocked on that, actually.
• copperi agree on weird sound issues could have been tested more.
• sumantrom (talk) - Sending some testing event updates to our "users" list. I see surge in contributors when there is an article posted in Fedora Magazine which means Fedora's user base wants to contribute/ try out things. Posting things in the user-list will help.
• sumantrom (talk) - Onboarding session on Bugzilla and test case writing.
• frantisekz - Some testing could have been conducted earlier in the development, especially KDE/Discover blockers and bugs have been present for a long time before the release.

Wishlist

• Ahmedalmeleh - For Fedora 36 Beta I wish documentation on how to use OpenQA and Bugzilla was more easily accessible and we would give new participants even more range of tools and knowledge from the start.

Recommendations

After enough time has been given for feedback, the QA team will discuss and make recommendations on changes to prioritize for Fedora 36. This section organizes and lists the recommendations.

In order to coordinate efforts, and measure effectiveness of recommendations, please record and track any action taken in the QA issue tracker.

• Be more proactive about co-ordinating complex SELinux fixes between SELinux and component maintainers in future
• Keep in touch with newer KDE contributors from the Fedora 35 cycle so we can fix future KDE blockers promptly
• Make a Bugzilla how-to video
• Improve Bugzilla instructions
• Fedora 36 Beta validation test event
• Onboarding session on Bugzilla and test case writing
• Create test cases for graphical package management
• Automate graphical package manager test cases
• Update QA/SOP_Test_Day_management "Promote the Test Day" section - done