A proven tester, also known as a critical path wrangler, verifies and reports on the stability of test updates to critical path packages. They retrieve their updates from the Test Updates repository and report their findings as karma using Bodhi. Positive karma from a proven tester is required for each critical path update before it can be pushed to the stable repository.
Individuals must must be mentored and approved before becoming proven testers. A proven tester is a member of the proventesters group.
Responsibilities
Proven testers are responsible for installing, testing, and providing feedback to updates for critical path packages.
Feel free to vary the details as suits you, but please bear in mind the advice on what types of feedback to leave in what cases.
Find and install package updates for testing
Enable the updates-testing repository, if you haven't already, and do a full system update each day. It is of course fine to quickly install one specific critical path update and test it in isolation, if you are particularly interested in ensuring the update is tested promptly, or the package maintainer sends out a request for testing.
Check for major regressions
Perform a quick functionality test to ensure the updated package is working as intended. If the package is an application, run the application and check that basic operations work. If it is a library or other shared component, run an application which uses the component and ensure that that works.
- After doing the update, reboot the system and log in to the desktop (assuming you can). If you observe any problems or changed behavior, take a detailed note and try to identify the package responsible for the change.
- Ensure your network access is working as usual. If not, try to identify the cause.
- Check that the update application will run, although you now likely will have no updates remaining to test immediately whether it works. If not, try to identify the cause.
Investigate and provide feedback
If the component works as expected, post positive feedback. If you identify a major problem, post negative feedback. If you identify a problem which is minor in nature and does not impede the actual critical path functionality, please do not post negative feedback. Post neutral or positive feedback with a note of the issue encountered (and a link to a bug report if appropriate).
- Use
fedora-easy-karma - Pay particular attention to updates whose description notes that they are critical path updates. If you identified any serious problems in earlier testing and were able to identify the package responsible, post negative feedback for that package. If possible, please file a bug report on the problem and link to the bug report in your feedback message. A good feedback message quickly and clearly identifies the behavior change and the cause, if you were able to determine it.
- If you are not sure what the component does or how to test it, do not post positive or negative feedback. If the above general tests of booting, network functionality and update functionality identified no problems, it is fine to leave a neutral feedback message noting that you were able to boot the system and perform critical path tasks with the update installed.
- If your testing uncovers no problems but you see that another tester has identified a serious problem with the package, please try to replicate their problem, and post negative feedback if you are now able to confirm it. If you are not able to confirm the problem but you suspect this may be because you cannot recreate the necessary conditions, please post neutral feedback noting that you were unable to duplicate the problem. Only post positive feedback if you are sure your testing indicates the other reporter's negative feedback is a mistake.
Typical Scenarios
Since a proven tester's karma determines whether an update is allowed to be promoted, they follow special procedures based on the severity of regressions they encounter. A bug or regression is considered severe if it violates the Fedora Release Criteria or breaks a critical path action.
- Major bug - Report, vote down.
- Minor bug - Report, vote up/neutral
- Previously reported bug - Confirm, vote accordingly
Unusual Scenarios
- Unreproducible bug
- Unfamiliar package