QA:Recognition

There are a lot of people who spend their time on QA-related activities. Some of them are Red Hat employees who are paid for this, some of them are Red Hat employees who do it in their free time, and large portion of them is the community. We would like to recognize their work as valuable and publicly thank them for their effort regularly.

Recommendations

• Name people explicitly
  Don't post just a general un-targeted letter, but try to name the most helpful people explicitly. Seeing your name publicly is very gratifying and motivating. It's the least we can do for the people helping us out.

• Attach result values, but carefully
  If there are any numbers connected with the names (number of executed test cases, reported bugs, etc), you can add them too and create some form of ladder. This should be done carefully, though. It will certainly please people at the top and can motivate others to increase their effort next time. But, on the other hand, it might discourage some. The results should be taken lightly and with humor in the adjacent text, rather than considering it a serious metric. You should mention that these numbers should be taken with a pinch of salt, because they are inherently incomplete (e.g. different test cases have different complexity, and also it covers just one particular area of QA activities). See the footnotes in attached examples. Adam is putting it very nicely here.

• Don't filter nor brand the contributor list
  Filtering the list based on community/Red Hat membership is not recommended. You never know who participated in his/her free time and who was paid for it. Relevant discussion is in this thread. Branding Red Hat employees is not recommended as well, it might be seen as some 'elite' club. But it is recommended to acknowledge in your text that many of the contributors are Red Hat employees and specifically call out community contributors who did an outstanding job and thank them once again.

• Do not re-use old letters too much
  If you re-use your letter over and over again, just changing the list of contributors, it will appear as computer-generated. That does not convey the right message. It must include a human touch if it is to be sincere. Of course there's nothing wrong with re-using some old headers and footers of the letter, but the main text should not be copy-and-pasted, but written completely anew.

• Encourage people to join the testing effort
  Include information for newcomers how to join the effort and help us out in future activities.

• Cover multiple channels
  When sending out the statistics/thank-you letter, it's good to send it at least to Fedora Planet, test list and Fedora community on G+.

Our regular recognition letters

Below is a list of recognition letters we try to send out regularly and their short description.

The tools used for data gathering are:

• qa-stats

  git clone https://pagure.io/fedora-qa/qa-stats.git

• relval

  dnf install relval

• testdays

  git clone https://pagure.io/fedora-qa/testdays.git

Always try to gather the data as soon as possible after the relevant milestone. Due to some technical restrictions the numbers can change slightly in time; they are most accurate directly after the milestone is over.

Some tools require that you specify dates (the beginning and the end of the relevant period). For Alpha, use Branch date and Alpha Go decision date. For Beta, use Alpha Go +1 day and Beta Go date. For Final, use Beta Go +1 day and Final Go date.

Fedora Branched milestone statistics

Wiki matrices

• What: A list of all contributors who filled in some test case results in our release validation wiki matrices.
• How: Use relval:
  • relval user-stats --since YYYYMMDD --until YYYYMMDD > stats-wiki.html
  • DO NOT USE the --milestone parameter! It will omit relevant nightly events.
• When: Created and sent out for every Fedora Branched milestone (Alpha/Beta/Final) separately (i.e. specify the dates for the specific milestone).

Bugzilla

• What: A list of all reporters who created new bug reports in Bugzilla against a Fedora Branched release.
• How: Use stats-bugzilla.py, from qa-stats:
  • ./stats-bugzilla.py --threshold 5 18 Final 2012-11-29 2013-01-15 > stats-bugzilla.html
  • The dates denote the start and end date for the specific milestone (in this example F18 Beta release and F18 Final release dates).
• When: Created and sent out for every Fedora Branched milestone (Alpha/Beta/Final) separately (i.e. don't include previous milestones numbers in the current stats).

Bodhi

• What: A list of all contributors who provided feedback in Bodhi for Fedora Branched updates.
• How: Use stats-bodhi.py, from qa-stats:
  • ./stats-bodhi.py --release F19 --startdate 2013-03-12 --enddate 2013-04-23 --username yourname --threshold 3 > stats-bodhi.html
  • The dates denote the start and end date for the specific milestone (in this example F19 Branch time and F19 Alpha release dates).
• When: Created and sent out for every Fedora Branched milestone (Alpha/Beta/Final) separately (i.e. don't include previous milestones numbers in the current stats).

Test Days

• What: A list of the top testers at Test Day events during the release cycle. The list includes the top ten testers by number of tests run, plus any of the top ten testers by bugs referred who: a) referred at least two bugs, and b) were not already in the top ten by number of tests run.
• How: Use testdays:
  • ./testdays.py stats -r 22 to generate statistics for Fedora 22 Test Days.
• When: After all Test Days for the cycle have been run, or after the release as one of the Heroes of Fedora posts.

Quarterly statistics

Bodhi

• What: A list of all contributors who provided feedback in Bodhi for all Fedora releases in a given quarter.
• How: Use stats-bodhi.py, from qa-stats:
  • ./stats-bodhi.py --startdate 2013-01-01 --enddate 2013-03-31 --username yourname --threshold 5 > stats-bodhi.html
  • The dates denote the start and end date for the specific quarter.
• When: Created and sent out for every quarter separately (i.e. don't include previous quarters numbers in the current stats).

Examples

• kparal.wordpress.com
• roshi.fedorapeople.org