Fedora 12 QA Retrospective

Introduction

This page is intended to gather feedback from the Fedora QA community on things that worked well and things that could have been better with the testing of Fedora 12. The feedback will be used as a basis for identifying areas for improvement for Fedora 13 testing. Any thoughts, big or small, are valuable. If someone already provided feedback similar to what you'd like to add, don't worry ... add your thoughts regardless.

For any questions or concerns, send mail to fedora-test-list@redhat.com.

Providing feedback

Adding feedback is fairly straight forward. If you already have a Fedora account ...

1. Login to the wiki
2. Select [Edit] for the appropriate section below.
3. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

4. When done, Submit your changes

Otherwise, if you do not have a Fedora account, follow the instructions below ...

1. Select the appropriate page for your feedback...
   • Something that worked well
   • Something that didn't work well
   • Anything on your QA wishlist
2. Add your feedback using the format:

   * ~~~ - I like ____ about the new ____ process

3. When done, Submit your changes

Feedback

Things that went well

Process

• Liam Li - The schedule of Fedora 12 Quality Tasks is accurate. From there, we can exactly know when to start the install testing. 2. The ticket to request Compose or media is very convenient and has high performance
• jlaska - Having QA/SOP_Test_Day_management was invaluable and resulted in a consistent look'n'feel for Fedora 12 test days
• Viking-Ice - Increase in how to debug <component> for reporters to follow and provide better bug report.
• Viking-Ice - Consistency in how to debug <component> pages for better look and feel.
• He Rui - Common f12 bugs page
• He Rui - The SOP management of both test days and install tests
• jlaska - Care and feeding of the Test Day live image creation wiki (QA/Test_Days/Live_Image) ... and acceptance as an official spin

Communication

• jlaska - The creation of the test-announce@lists.fedorahosted.org mailing list made it much easier to announce test events. With a single mailing list for announcements, it's harder to miss any events.
• Viking-Ice - Improvement in advertisement testing/test day's
• Viking-Ice - Better collaboration and participation from developers/maintainers with QA.

Test Planning

• jlaska - Automated test cases provided for the Power Management test day == awesome!
• Viking-Ice - Increase in SIG and maintainers for hosting and holding their own test days and specific application testing.
• Liam Li - [The number of] People who participate in install test are increasing
• Adam Williamson - test days went well again. it was nice to see how many independent test days there were.

Test Execution

• jlaska - Consistent announcement of ISO test events from Liam was great to ensure a consistent testing experience (see User:Liam/Draft_Install_Test_SOP).
• jlaska - We had more ISO install test runs, and they were earlier in each milestone
• jlaska - F-12 ISO Install testing solicited more testing from outside the core install test group. 27 different people contributed test feedback on the wiki test matrix during F-12. . 19 different people contributed test feedback on the wiki test matrix during F-11.
• jlaska - Community initiated testing of XFCE spin -- See https://www.redhat.com/archives/fedora-test-list/2009-August/msg00443.html

Infrastructure

• Viking-Ice - Nightly-composes of rawhide test images.
• Viking-Ice - Increase in request from sig/maintainers in Fedora QA Trac system.
• He Rui - Traceback can be saved or reported automatically
• He Rui - Tickets could be created and traced in Trac system
• Viking-Ice - Automatic bug reporting tool develop to help provide better bug reporting and allow technical challenge people to participate report and provide feedback.

Could have been better

• anonymous

uname -a hera.localdomain 2.6.31.12-174.2.3.fc12.x86_64 #1 SMP Mon Jan 18 19:52:07 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux language brazilian portuguese

this works for me ls -l *.rpm

look at "./" preceding the char "*" this works for me ls -l ./*.pps

but

this does not works for me

ls -l *.pps ls: invalid option -- 'P' Experimente "ls --help" para mais informações.

I think this is a bug

• jlaska - it may be, but I believe you might want to confirm what the asterix is expanding to in the directory you are running the command ls -l *.pps. It's possible it is expanding to a filename that has -P as the first 2 characters, which confuses the ls command. What does echo *.pps yield? You may wish to enclose the command in quotes, ls -l "*.pps". If problems remain, please raise the issue on the users or test list.

Process

• jlaska - The blocker bug meetings during the Final milestone were too long.
• He Rui - Post-event review, python could be used for install test like in test day

Communication

• Kparal - test-announce wasn't used for announcing RC candidates IIRC, we should make sure all important events are mentioned there
• He Rui - Test days/Install test events could be put up obviously on wiki so that people who wanna contribute can find it easily from wiki except from mail lists
• He Rui - Create a main page for installation test, where it tells what, when and where are the install test events. And it's better if this page can be put in fedora qa activities
• Rahul Sundaram - IMO, RCs needs to advertised loudly. We need all the testing we can get and more alpha or beta snapshots would help as well, I think.
• Liam Li - The bug maintenance takes lots of time.Can we build a bug report instructions about what logs need to attach when report a bug for specified component.or this can build to bugzilla? when selected a component,some logs/information are recommended to attach to save the communication time between the developers and testers. By this way,at least some necessary information will not be missing because of the carelessness of testers, and the reproduce time will be reduced when some necessary information missed

Test Planning

Installer/Preupgrade

• Kparal - The preupgrade could receive more testing/test cases
• Matthias Clasen - From my perspective, the two main avenues to a new Fedora release are the live installer and preupgrade, and those two should get all the attention they can get.
• Adam Williamson - there was clearly a lot of uncertainty about RAID issues; it's something we obviously don't as a team test well enough (and some of us personally don't understand enough :>). In the end there didn't turn out to be any horrible issues, but the confusion was evident, and we did miss the Intel BIOS RAID stuff-ups. For F13 we should have better RAID testing both in Test Days and in pre-release test cycles.
• Rahul Sundaram - Lot more users are trying Preupgrade and QA needs strong focus on the upgrade story including test days directed towards it. The small /boot is a major issue and another common bug (not listed in the wiki but I think needs to be added) is https://bugzilla.redhat.com/show_bug.cgi?id=538118 which makes the problem worse. We really need to make sure Anaconda creates a bigger /boot for Fedora 13, preupgrade is explicitly tested well and has solid workarounds for the small /boot case.
• Michal Jaegermann - It seems to me that this time there are way too many "unupgradable" packages. I mean by this that when upgrading from a maintaned Fedora 10 or 11 installation packages "on hand" have higher version than even _updates_ for Fedora 12 and are turned into pseudo-orphans. With broken 'package-cleanup --problems' (see https://bugzilla.redhat.com/show_bug.cgi?id=541551) that adds an extra excitement and some of these "lefovers" break due to missing dependencies.
• Pingram - Most of the preupgrade process was well automated and went smoothly for a standard client with no major package modification but I faced some minor issues on my main desktop. I think there would be benefit in being able to customise repositories through the GUI as modifying /usr/share/preupgrade/releases.list still resulted in preupgrade downloading the relatively large stage1/2 images from my nearest mirror instead of my local repo. Also, having a report of dependency issues and possibly a way to resolve/remove/ignore them before starting the preupgrade proper may help overcome issues on next boot.
• Pingram - Post upgrade I faced the arduous task of using yum to remove earlier version packages that didn't have current dependencies. Can preupgrade remove, for example, f11 packages that aren't needed for f12? And, is there a way of reporting on .rpmnew and .rpmsave diff's in some sort of postupgrade to assist in finalising an installation? On my system I have an 802.1q trunked connection which AFAIK NetworkManager can't handle, if all packages are downloaded to the local machine is a network connection still needed?

Xorg

• Adam Williamson - We weren't completely on top of X.org bugs for this release. The ones that wound up getting promoted to release blocker level were kind of an arbitrary selection. I think we got nouveau mostly right as I had a reasonable grip on nouveau triage, but we just had too few people to triage server / intel / ati bugs during the cycle, so when we hit beta / RC stage, we didn't have the whole bug set well enough triaged to be able to be sure we picked the most important bugs as blockers. For F13 we should stay on top of triage better so we can do blocker identification accurately. happily, matej is more active on X triage again now and we have some more assistance from Chris Campbell (thank you Chris!) further volunteers would be great. I will try to stay on top of nouveau, again.
• Rahul Sundaram - KMS is still flaky in some cases. In particular, a few Intel users seem to be reporting lower resolution by default without nomodeset and ATI performance seems to have regressed. Although I am no fan of proprietary drivers, I must note that installing the proprietary Nvidia driver has become a bit more of a hassle.

Security

• Adam Williamson - we have the big security thing (aka lack of a defined security policy and test plan) to deal with. I did start a thread on fedora-devel-list about that.
• Rahul Sundaram - PackageKit signed install policy was a major issue and although QA was not really responsible for it, the team does need to do it all can do to avoid anything like this in the future . Signing Rawhide packages automatically would have caught this. Not many users in forum or fedora-list complained about it however.

General

• Adam Williamson - in the end, we focused heavily on just three component areas for testing: anaconda, kernel, and X.org. This is primarily a function of the fact that these are the most vital bits; it feels like we're still at the point of doing let's make sure it's not totally broken testing than let's make sure it's really good testing. We didn't do stuff like making sure the desktop was polished.
  • Matthias Clasen - I agree with this assessment. I would go even further and say it is to a large extent 'lets make sure the installer is not totally broken for exotic cases' testing.
• Jim Haynes - I've learned it is necessary to install foomatic-db to have my particular printer show up in system->administration->printing. According to Adam Williamson, This has been documented in common bugs, but it does seem like something we ought to have caught - thanks for bringing it up. Perhaps we need more testing of common basic peripherals like printers.

Test Execution

• CAI Qian - Power Management Test Day is boring, because it provides a automated scripts to run, and not sure how to inspect that data from the testers. As the results, it lose the fun to hunt bugs. It is important to have some instructions to inspect the data we received in the future, so we can see if there any regression or issues.
• He Rui - Transfer from manual-installation to Auto-installation test
• Liam Li - Install test on ppc platform is not sufficient. I mean some cases can not be executed comparing with i386/x86_64
• Liam Li - The install time is very long,most of the time is spent on package install stage,but most of the bugs are not occurred on packaging install stage,can we build a CD which is very similar with LiveCD, but it just for install test,only install gnome and other basic packages?By this way, we can find anaconda bugs, but the size of this media is less then 700M. Or we can add this function to LiveCD,there is an option to start install on boot stage,not to start install after login desktop like liveCD.

Infrastructure

• Scott Robbins - Although it's not much of a factor in the US, apparently bandwidth limits are quite common in Australia--there were several who seemed rather disgruntled about the somewhat confusing SHA1/SHA256 sum issue. Although it's now clear in the release notes, those who verify, as a rule, have done it before with other distributions, and are used to downloading an ISO, looking at the sums file, and seeing that it's md5 (if anyone's still using it), or SHAwhatever. Upon getting what seems to be a bad checksum, they download again, which in fairness, if you get one bad checksum, is probably the logical thing to do.
• Liam Li - Speed of test Media for download is snow,people are unwilling to download DVD to test.could we put this to mirror?or think about other way to solve this problem?

Wishlist

• 66.187.233.202 - I would like a pony
• jlaska - I also want a pony

Process

• Kparal - Searching through our wiki I could not find a page named "Fedora 12", "Fedora 13", etc. Simply a page summarizing (linking) everything important about a release - schedules, features, current progress, (release notes). It's scattered in pages like these and these, but no page is standing out as an "outpost" of that particular release. I imagine if we have some page like that, we could add there links about the current testing efforts, so people would be easily navigated exactly where we need them to help ("current QA efforts: Fedora 13 RC2 testing"...). This "Fedora 13" overview page would of course be referenced from main wiki page, so people can easily find it.

Communication

• Kparal - People attending Test Days presentation asked which sources are most attractive for Test Day announcements and allure many people. Where certainly not to forget to advertise when arranging a test day.
• Kparal - People attending Test Days presentation asked if test day results are presented and published externally, like in magazines, etc.
• Kparal - People attending Test Days presentation asked if Fedora Test Days are on Facebook and Twitter, which could attract larger audience.
• Liam Li - Have more method and more broadcast way to let more people know our testing
• Kparal - I would like to see more people in beta/RC testing. New templates are going to solve the practical part (display multiple results easily), now the issue is to attract more people. I would like to visit get-fedora page and see there "Fedora 13 Release Candidate available! Download <<here>>, help with testing <<here>>". And links to our wiki pages describing how to contribute on testing. I think Ubuntu is doing similar thing (large banners "Beta available" on their front page). That could bring a lot of new people.
• He Rui - More Asian testers could participate in the test events. (could inform them by forwarding announcements to their mail list/irc in native language)
• Jim Haynes - Over all this is the easiest Fedora version upgrade I have ever done; and when I did have problems I got quick and correct answers from this list (fedora-test-list@redhat.com) that I was not getting anywhere else. (Maybe that suggests we should have a current-version-install ombudsman list, where anybody can present questions but only authoritative people can give answers.)

Test Execution

• He Rui - 'Edit' on wiki could be more automatically by filling in and submitting a form without wiki syntax.

Recommendations

After enough feedback has been collected, the QA team will discuss and make recommendations on changes for Fedora 13. This section will be filled in at that time.

Process

• Move User:Liam/Draft_Install_Test_SOP out of draft status
• Move User:Poelstra/blocker_bug_meeting_sop out of draft status
• Draft a SOP (or extract from existing BugsAndFeatureRequests page) that describes how to document bugs (e.g. common_bugs or release notes or install guide etc...)
• Promote install testing as a QA activity (includes updates to QA/Join and QA)
• Draft/promote a recommended test checklist for rawhide testers (for example see User:Jlaska/Draft2)
• 'How to debug <component> problems?' -- Determine whether guides can be offered by Fedora QA as a result of a test day/feature process.
• Closer engagement between install test and BugZapper traige
  • Review and refresh as needed How_to_debug_installation_problems (needs remote debugging tips)
  • Increased participation in triaging incoming bugs
• Continue with existing Test Day definition process

Test Planning

• Maintenance of the Fedora Install Test Plan
  • Remove duplicate content use cases
  • Adjust priority according to revised release criteria
  • Adjust preupgrade test cases to accomodate late F-12 preupgrade bugs
• Draft a more formal, but basic and lightweight, security checklist/test plan
• Draft a test plan to cover the desktop verification from Fedora_Release_Criteria
• We have no formal Xorg-x11-drv test plan ... either create one, or develop proposal for gathering the same feedback through other means (smolt, bugzilla)
• Proposed Test Days
  • Another round of X test events (these are always popular and successful)
  • Another sound event?
  • Several installer events (partitioning ui focus, libcurl, raid again)
  • SSSD by default
  • Rehash of gnome-disk-utility?
• Fit'n'Finish?

Test Automation

• Deploy AutoQA 'is rawhide broken?' front-end into fedora infrastructure
• Deploy build-time sanity checking mechanism for package maintainers (rpmguard, rpmlint etc...)
• Develop proof-of-concept for automating install test matrix -- see Is_anaconda_broken_proposal for details

Communication

• Continue looking for ways to identify and lower contribution barriers for install testing
• Pursue more methods for communicating test events
  • Fedora QA social networking presence (facebook/twitter pages)
  • Provide link to how to get involved in testing on http://fedoraproject.org/get-prerelease
• Need some sort of Xorg adapter test dashboard -- something simple using existing tools to start with (wiki or smolt)?
• Reduce the Blocker Bug Review in-meeting time
  • Either schedule more meetings
  • Or, continue documentation efforts to help testers self-diagnose severity of failures out of meeting

References

• See Fedora_11_Retrospective_Notes