QA/SOP Test Day management

This page explains how to go about arranging a Test Day event. If you have any questions or concerns about the Test Day process, please contact the QA group.

Get people on board

Make sure all involved parties are aware of the Test Day. Usually, you would want the maintainer(s) of the affected component(s) to be aware of the Test Day and, ideally, present when it happens. If you know of experienced users of the component(s) who will likely be able both to provide useful feedback at the Test Day and bring in other users, it will help to bring them on board in advance, too. If the affected component has a triager - see the BugZappers team's list of components and triagers - bring them in too.

Set a date

Aim for a day when:

• Most involved people (organizer, maintainers, experienced users, triagers, QA team) can be available
• The feature will be close enough to complete for the testing to mean something, but outside of any deep related freeze periods
• No other event which would be likely to draw away users with an interest in the Test Day is happening - including other Test Days!

Try to be flexible when it comes to the time the Test Day actually happens: consider timezone issues, especially if many users of the feature are in a particular geographical area. Ideally, aim to have developers or experienced users present throughout the entire period when it is the date of the Test Day somewhere in the world.

Create the Wiki page

This will be the central source of information regarding your Test Day: make sure it's complete, accurate and up to date at all times. Use this Test Day template as a base: copy and paste the source of that page and fill it in for your event. Your page should be given a name of the form Test_Day:(DATE)_(TOPIC) - e.g. Test_Day:2014-09-25_Virtualization . It should be in the category for Test Days for the pending release - for instance, Category:Fedora 42 Test Days. You can add your event to the Test Day schedule for the appropriate release. Release Test Day schedules are linked to from the main Test Days page.

As a goal, your Wiki page should provide enough information that someone could perform all the testing and provide their results without any other reference or help.

Create test cases

A test case is simply a precise set of instructions which cause a reliably repeatable operation to occur which allows you to determine some kind of information from the results of the operation. There is a template to be used for creating test cases: if you visit that page, you will see a link which explains how to use this template to create a test case. You can see example test cases in the test case category page. When creating a test case, try to make sure the instructions are explicit enough that anyone who may want to participate in the test day can follow them, and entirely unambiguous so that the user cannot perform the test incorrectly. Don't cram multiple operations into a single test case: create separate test cases for each different operation you wish to test. If possible, try to create the Test Case in such a way that it will be re-usable in the future (ask yourself if someone running an edition of Fedora from five years after you write the test case would still be able to follow it), although this is not always possible.

Link to each of the test cases from the How to Test section of the Wiki page.

Set up the TestDayApp

The results for test cases are stored and managed in the TestDayApp. The TestDayApp takes metadata from your test day's metadata page and creates an easy to use interface for submitting test results. An explanation of the required metadata is on this page while an example can be found here.

Step 1

Create the required metadata for the test day. Go to the create/update page, enter the full url for the metadata of your test day and click 'Submit.' Your browser will reload and new entry in the TestDayApp will be created.

Should you update the metadata, once again visit create/update page, enter the full url for the metadata and click 'Submit'. As long as you keep the URLs for testcases the same, the results will be intact.

Step 2

Once the test day is completed you should export the results from the TestDayApp. Go to the export page, enter the full url for the metadata of your test day and click 'Submit.' Your browser will reload and you'll be presented with the results of your test day in wiki format.

Step 3

Copy the wiki-formatted results from step 2 and paste them into your Test Day wiki page. Use the "Show Preview" button to check your formatting.

Build a live CD

If some or all of the test cases can be successfully performed from a live boot, it is highly recommended to provide a live CD image. Instructions on doing so can be found here. Remember to customize the spin to include whatever application you want to test, and any utilities that are needed for testing it. Also make these available through a repository if they're not in rawhide, for people who want to test from their rawhide system rather than the live CD. Put the live CD image up in a fedorapeople space; contact the Infrastructure Team to extend your quota if necessary.

Promote the Test Day

• Blog about it (and make sure your blog's on Planet Fedora). Multiple blog posts are good! Don't just have one person blog - have as many as possible, and you can blog more than once if you like (once two days ahead of the event and once on the day itself, for instance)
• Post to social media - Diaspora, Twitter, Google+, Facebook, the more the merrier - using the #fedora hashtag, and the Fedora and/or Fedora QA groups for the platforms that have them
• Send an announcement mail to the test-announce mailing list
• Contact the Fedora Magazine team and ask them to post about the Test Day - link to your blog and/or social media posts about the event. You can use their contact form, but it might be faster to contact the marketing mailing list or the #fedora-mktg^[?] IRC channel
• If it affects a specific country, contact the Ambassador(s) for that country. If it affects a specific community, contact the places where that community gathers - IRC, forums, whatever
• If it's possibly of interest to a wider community than just Fedora users (make sure you have a live image, for this one!), submit it to some general interest news sites. Don't be scared - news sites love content. They live off it. They have a very low bar for it. Try OS News, Linux Today, LWN, and LXer for everything, and Phoronix for anything that a home enthusiast community is likely to be interested in. You should also get a post in the Fedora Forums news section, but you'll need someone with posting privileges to do that
• If you are a Red Hatter, send the announcement also to the internal QE mailing list.

It's probably best to announce the test day about a week in advance, and do a 'quick reminder' post the day before.

Pre-event reminder

Remind the involved parties - maintainers, experienced users, triagers, QA team - about the event shortly before it happens, to make sure they show up. And remember to show up yourself!

Update Test_Day:Current

Update the Test_Day:Current wiki redirect. The Test_Day:Current page is used to conveniently direct testers to the most current test day. To update the page, login to the wiki and click here.

Do a post-event review

Doing a review of the Test Day once it's finished helps make it clear what the Test Day achieved, acts as a useful reference for issues discovered, and helps to promote the Test Day process to other users. A good way to write the review is to base it around a generated list of the bugs reported as a result of the Test Day. You can generate such a list by running this command:

curl "https://fedoraproject.org/wiki/Test_Day:Current" \
 | grep -o "bugzilla\.redhat\.com.*[=\/][0-9]\{6\}" \
 | grep -o "[0-9]\{6\}" | tr '\n' ',' | sed 's|,$||' \
 | xargs bugzilla query --outputformat="%{bug_id} %{bug_status} %{resolution} - %{short_desc}" -b

You should do this after copying the results from the Test Day app back into the Wiki page, or else it will yield no results. The command will produce a list of bugs whose bug report URLs are in the Test Day page, with status information on each bug. (The second grep command is a safety valve because the first can give messy results when people mention multiple bug reports in a single line). You may have to weed out any 'false' references to pre-existing bugs which are mentioned on the page for some reason. This command requires the python-bugzilla package be installed. If you're not sure what a review email should look like, here's an example. You should send the report to the test-announce mailing list, and perhaps to other mailing lists related to the topic of the Test Day.

Who attended?

Test Days are promoted in the community and are community events. At some Test Days there is often a number of the developers/QA members attending, often Red Hat employees, so you might be interested in other members of the community who gave up some of their free time and contributed in testing.

For events with great number of participants, when it is difficult to weed out who is who (can't really say based on what testers write in the User: field unless they have a descriptive profile), you can use this program to generate the list of attendees and (optionally) check it up against the Red Hat roster (you will need python 2.7, check --help for more info)