A proven tester, or critical path wrangler, verifies and reports on the stability of updates to critical path packages. Acknowledgment of an update's stability is indicated through positive karma awarded in Bodhi. A positive vote from at least one proven tester is required before an update in updates-testing can be pushed to the stable repository. This requirement provides a measure of assurance that updates to critical packages are subject to basic verification before they are widely deployed.
The proven tester role, like others in Fedora, is not an exclusive position; individuals are free to go beyond what is required. Similarly, regular testers are free and encouraged to provide karma on both critical and non-critical packages.
Test criteria
A positive rating from a proven tester indicates that the rated update does not make the system unstable. Proven testers should vote positively on an update that keeps the package stable, even if it introduces minor bugs of its own. They should also restrict their vote to the update, not the package as a whole; major bugs or regressions for a package do not disqualify all updates that do not correct those problems. This focus on pragmatism, rather than purity, is designed to strike a balance between the stability of an update and the timeliness of that update's release.
Stable updates
A stable update can be roughly defined as an update that, when applied, does not cause, or introduce the possibility of, severe regressions to a user's system. The severity of a regression or bug is dependent on how, and how often, it impacts the user's experience. This impact can occur via a serious reduction of functionality, damage or data loss, or an increased level of vulnerability. Manual intervention should never be required to avoid a serious regression. Planned breakages, such as removing deprecated functionality or upgrading to an incompatible version, are not necessarily considered regressions. Benign messages and warnings, while they should be reported, are also not sufficient on their own to withhold a positive vote.
While some specific requirements exist for both the system as a whole and for different categories of packages, the diverse range of functionality provided by critical packages, combined with an even more diverse set of use cases, makes defining a baseline of stability difficult. Ultimately, a proven tester's judgment and objectiveness is necessary to ensure a sufficient and consistent level of stability across packages.
System-wide requirements
At a minimum, the update, when applied, should still allow for the successful execution of all critical path actions. These actions serve as the baseline of stability for any update:
- The system must boot. A user must be able to boot, log in to X, log off, and shut down without performing any unusual modifications or previously unnecessary steps. Some examples include having to modify grub, changing the way the system is booted, or having to avoid some option that was previously operational.
- The network should be accessible. A user should be able to perform common network-related tasks, such as viewing web pages, checking their email, and SSH'ing into other machines.
- Packages must update without error or intervention. A user should be able to graphically install new updates from Fedora without requiring manual intervention, such as using "--skip-broken" or manually resolving changed dependencies.
- Security must not be adversely affected. Security features, such as firewalls and permissions, must continue to operate at an expected level.
Package-specific requirements
Proven testers expand their requirements of a stable update to also include the major functionality of the updated package. What makes a feature major, or a regression severe, ultimately depends on the package, but proven testers should follow this common set of guidelines as they apply. For example, an update to a library or shared component cannot be tested easily in isolation, so instead, an application that depends on that library should be used to verify stability. This list, while not exhaustive, mentions a few areas of testability:
- The application must start and stop cleanly. A user must be able to start up and close the application without freezing or requiring the application to be forcefully closed.
- The application must not cause data loss. The user should be able to create, save, and load their work. Existing data should not be corrupted, unexpectedly modified, or lost.
Providing feedback
Proven testers fulfill their role by voting with Bodhi karma on updates they have tested. Generally speaking, positive karma from a proven tester is an indication of their trust in the stability of that update. Conversely, negative karma indicates that the update is considered unstable. Neutral karma should be left in cases of an untestable or insufficiently tested update.
Proven testers should leave comments with their karma, especially with negative karma, to indicate the reasons for their vote. If a severe regression is found, the comment should identify the behavior change and, if possible, the cause. Comments should also note problems that were found but did not warrant a negative vote. Bug reports are useful supplements in either case. These additional measures assist the maintainers in fixing the problem, and they guide other testers by giving them opportunities to confirm reported issues.
Insufficiently tested updates, such as packages that are very unfamiliar or incompatible, should not be given positive feedback. For example, a major update to CUPS can be installed and examined, but it should not ordinarily be given positive feedback if the proven tester does not have access to a printer. The proven tester should post neutral karma if the examined portions were stable and comment that only a limited amount of the update was tested.
Collaborating and verifying the results of other testers allows problems to be narrowed down and solved quickly. Comments, bug reports, and votes by other testers can be used to streamline a proven tester's job. Multiple votes in the same direction builds confidence in the stability, or instability, of that update. If a proven tester finds that karma has already been submitted for an update, she should attempt to confirm those results, posting her findings as a vote and comment of her own.
If a proven tester is unable to reproduce a problem reported by another tester, he should indicate this as a comment and a neutral vote. If in doubt, unreproducible bugs should be treated as problems with the update, rather than mistakes. Positive karma should only be given if the tester is confident that the negative result was in error.
Becoming a proven tester
Use 
fedora-easy-karma
- Sign up for the Fedora Account System. You must have a FAS account before you participate here.
- Pick a mentor by requesting one through the Fedora QA Trac.
- Apply for proven testers. Details of the procedure to apply for the Proven Testers QA group are below.
A mentor can be any current proven tester. An interested QA community member who would like to become sponsored may request a current Proven Tester as their Mentor as part of their trac ticket if they have a preference of who mentors them. Once the candidate Mentor accepts the interested QA Community Member's request this Mentor will oversee the interested member and help them work within the Critical Path QA efforts, policies, and procedures.
Once these steps are complete and your mentor confirms you have demonstrated an understanding of QA process and community, your application for approval to the Critical Path Wranglers (Proven Testers) will be brought up for review with the current proven testers. If accepted, your FAS account will be approved for the proventesters group.