QA/SOP Test Day management

This page explains how to go about arranging a Test Day event. If you have any questions or concerns about the Test Day process, please contact the QA group.

Get people on board

Make sure all involved parties are aware of the Test Day. Usually, you would want the maintainer(s) of the affected component(s) to be aware of the Test Day and, ideally, present when it happens. If you know of experienced users of the component(s) who will likely be able both to provide useful feedback at the Test Day and bring in other users, it will help to bring them on board in advance, too.

Set a date

Aim for a day when:

• Most involved people (organizer, maintainers, experienced users, triagers, QA team) can be available
• The feature will be close enough to complete for the testing to mean something, but outside of any deep related freeze periods
• No other event which would be likely to draw away users with an interest in the Test Day is happening - including other Test Days!

Try to be flexible when it comes to the time the Test Day actually happens: consider timezone issues, especially if many users of the feature are in a particular geographical area. Ideally, aim to have developers or experienced users present throughout the entire period when it is the date of the Test Day somewhere in the world.

Create the Wiki page

This will be the central source of information regarding your Test Day: make sure it's complete, accurate and up to date at all times. Use this Test Day template as a base: copy and paste the source of that page and fill it in for your event. Your page should be given a name of the form Test_Day:(DATE)_(TOPIC) - e.g. Test_Day:2014-09-25_Virtualization . It should be in the category for Test Days for the pending release - for instance, Category:Fedora 42 Test Days. You can add your event to the Test Day schedule for the appropriate release. Release Test Day schedules are linked to from the main Test Days page.

As a goal, your Wiki page should provide enough information that someone could perform all the testing and provide their results without any other reference or help.

Create test cases

A test case is simply a precise set of instructions which cause a reliably repeatable operation to occur which allows you to determine some kind of information from the results of the operation. There is a template to be used for creating test cases: if you visit that page, you will see a link which explains how to use this template to create a test case. You can see example test cases in the test case category page. When creating a test case, try to make sure the instructions are explicit enough that anyone who may want to participate in the test day can follow them, and entirely unambiguous so that the user cannot perform the test incorrectly. Don't cram multiple operations into a single test case: create separate test cases for each different operation you wish to test. If possible, try to create the Test Case in such a way that it will be re-usable in the future (ask yourself if someone running an edition of Fedora from five years after you write the test case would still be able to follow it), although this is not always possible.

It's always a good idea to also include the Exploratory Testing test case, to motivate people to try different things and not just lock themselves into the prepared test cases.

Link to each of the test cases from the How to Test section of the Wiki page.

Set up the Test Day App

Test Day results can be stored and managed in the TestDayApp. The app takes metadata from your test day's metadata page and creates an easy to use interface for submitting test results. There is an explanation of the required metadata. An example can be found at Test_Day:2013-10-23_Graphics_Radeon_TestdayApp_Metadata.

Step 1

Create the required metadata for the test day. Go to the create/update page, enter the full URL for the metadata of your test day and click 'Submit.' Your browser will reload and a new entry in the app will be created.

Should you update the metadata, once again visit the create/update page, enter the full URL for the metadata and click 'Submit'. As long as you keep the URLs for test cases the same, the results will be intact.

Step 2

Once the Test Day is completed you should export the results from the app. Go to the export page, enter the full URL for the metadata of your test day and click 'Submit.' Your browser will reload and you'll be presented with the results of your test day in wiki format.

Step 3

Copy the wiki-formatted results from step 2 and paste them into your Test Day wiki page. Use the "Show Preview" button to check your formatting.

Create a calendar event

For each test day we want to have a corresponding event in our QA calendar. Each event must have the string Test Day in its title, so that it shows up in our test day schedule. The description supports Markdown, but it's better to use plaintext, because while Fedocal shows Markdown formatting, most other calendar software (Google Calendar, Thunderbird, etc) don't. The description needs to contain the link to the test day page, and ideally also a paragraph or two explaining what this test day is about.

Build a live image

If some or all of the test cases can be successfully performed from a live boot, it is highly recommended to provide a live image. Instructions on doing so can be found here. Remember to customize the spin to include whatever application you want to test, and any utilities that are needed for testing it. Also make these available through a repository if they're not in the official repositories, for people who want to test from their installed system rather than the live image. Put the live image image up in a fedorapeople.org space. Contact the Infrastructure team to extend your quota if necessary.

Promote the Test Day

• Blog about it (and make sure your blog's on Fedora Planet). Multiple blog posts are good! Don't just have one person blog - have as many as possible, and you can blog more than once if you like (once two days ahead of the event and once on the day itself, for instance)
• Post to social media - Diaspora, Twitter, Facebook, Telegram, the more the merrier - using the #fedora hashtag, and the Fedora and/or Fedora QA groups for the platforms that have them
• Send an announcement mail to the test-announce mailing list. For general user-facing test days, it is also a good idea to mail the users list
• Contact the Fedora Magazine team and ask them to post about the Test Day - link to your blog and/or social media posts about the event. You can use their contact form, but it might be faster to contact the marketing mailing list or the #marketing:fedoraproject.org^{(other clients|?)} Matrix room.
• Submit an article to the Community Blog - see this page for instructions
• Post about the event on Fedora Discussion, at least if there seems to be a relevant space for it
• If it affects a specific country, contact the Ambassador(s) for that country. If it affects a specific community, contact the places where that community gathers - Matrix, IRC, forums, whatever
• If it's possibly of interest to a wider community than just Fedora users (make sure you have a live image, for this one!), submit it to some general interest news sites. Don't be scared - news sites love content. They live off it. They have a very low bar for it. Try OS News, Linux Today, LWN, and LXer for everything, and Phoronix for anything that a home enthusiast community is likely to be interested in. You should also get a post in the Fedora Forums news section, but you'll need someone with posting privileges to do that
• If you are a Red Hatter, send the announcement also to the internal QE mailing list.

It's probably best to announce the test day about a week in advance, and do a 'quick reminder' post the day before.

Pre-event reminder

Remind the involved parties - maintainers, experienced users, triagers, QA team - about the event shortly before it happens, to make sure they show up. And remember to show up yourself!

Update Test_Day:Current

Update the Test_Day:Current wiki redirect. The Test_Day:Current page is used to conveniently direct testers to the most current test day. To update the page, login to the wiki and click here.

Do a post-event review

Doing a review of the Test Day once it's finished helps make it clear what the Test Day achieved, acts as a useful reference for issues discovered, and helps to promote the Test Day process to other users.

The testdays app can help you with this, by generating some statistics about the Test Day and a list of the bugs reported. Follow the instructions on that page to enable the repository where it lives and install it. To get the information for a Test Day that took place on 2015-06-01, you would run: testdays stats --since 2015-05-31 --until 2015-06-02 --list-bugs

You should do this after copying the results from the Test Day app back into the Wiki page, or else it will yield no results. The command will produce a list of bugs referred in the page, and some statistics on the total number of testers, tests, and bugs. Note that the Fixed % it reports is most useful for a Test Day that took place some time ago, to see how many of the bugs reported were fixed - shortly after the Test Day takes place, that number is naturally likely to be very low. You might find it useful, for instance, if you run a regular Test Day on the same topic for multiple releases - perhaps when writing the report for the Fedora 42 Test Day, you could run the command for the Fedora 41 Test Day and report how many of the bugs from that event were fixed.

If you're not sure what a review email should look like, here's an example. You should send the report to the test-announce mailing list, and perhaps to other mailing lists related to the topic of the Test Day.

Who attended?

Test Days are promoted in the community and are community events. At some Test Days there is often a number of the developers/QA members attending, often Red Hat employees, so you might be interested in other members of the community who gave up some of their free time and contributed in testing.

For events with great number of participants, when it is difficult to weed out who is who (can't really say based on what testers write in the User: field unless they have a descriptive profile), you can use this program to generate the list of attendees and (optionally) check it up against the Red Hat roster (you will need python 2.7, check --help for more info)