User:Dafrito/Proven tester

A proven tester, or critical path wrangler, verifies and reports on the stability of updates to critical path packages. Acknowledgment of an update's stability is indicated through positive karma awarded in Bodhi. A positive vote from at least one proven tester is required before an update in updates-testing can be pushed to the stable repository. This requirement provides a measure of assurance that updates to critical packages are subject to basic verification before they are widely deployed.

Test criteria

A positive rating from a proven tester indicates that the rated update does not make the system unstable. Proven testers should vote positively on an update that keeps the package stable, even if it introduces minor bugs of its own. They should also restrict their vote to the update, not the package as a whole; major bugs or regressions for a package do not disqualify all updates that do not correct those problems. This focus on pragmatism, rather than purity, is designed to strike a balance between the stability of an update and the timeliness of that update's release.

Stable updates

A stable update can be roughly defined as an update that, when applied, does not cause, or introduce the possibility of, severe regressions to a user's system. The severity of a regression or bug is dependent on how, and how often, it impacts the user's experience. This impact can occur via a serious reduction of functionality, damage or data loss, or an increased level of vulnerability. Manual intervention should never be required to avoid a serious regression. Planned breakages, such as removing deprecated functionality or upgrading to an incompatible version, are not necessarily considered regressions. Benign messages and warnings, while they should be reported, are also not sufficient on their own to withhold a positive vote.

Some specific requirements do exist for both the system as a whole and for different categories of packages. However, the diverse range of functionality provided by critical packages, combined with an even more diverse set of use cases, makes defining a baseline of stability difficult. Ultimately, a proven tester's judgment and objectiveness is necessary to ensure a sufficient and consistent level of stability across packages.

System-wide requirements

At a minimum, the update, when applied, should still allow for the successful execution of all critical path actions. These actions serve as the baseline of stability for any update:

• The system must boot. A user must be able to boot, log in to X, log off, and shut down without performing any unusual modifications or previously unnecessary steps. Some examples include having to modify grub, changing the way the system is booted, or having to avoid some option that was previously operational.
• The network should be accessible. A user should be able to perform common network-related tasks, such as viewing web pages, checking their email, and SSH'ing into other machines.
• Packages must update without error or intervention. A user should be able to graphically install new updates from Fedora without requiring manual intervention, such as using "--skip-broken" or manually resolving changed dependencies.
• Security must not be adversely affected. Security features, such as firewalls and permissions, must continue to operate at an expected level.

Package-specific requirements

Proven testers expand their requirements of a stable update to also include the major functionality of the updated package. What makes a feature major, or a regression severe, ultimately depends on the package, but proven testers should follow this common set of guidelines as they apply. For example, an update to a library or shared component cannot be tested easily in isolation: instead, an application that depends on that library should be used to verify stability.

• The application must start and stop cleanly. A user must be able to start up and close the application without freezing or requiring the application to be forcefully closed.
• The application must not cause data loss. The user should be able to create, save, and load their work. Existing data should not be corrupted, unexpectedly modified, or lost.

Feedback Procedures

Since a proven tester's karma determines whether an update is allowed to be promoted, they follow special procedures based on the severity of regressions they encounter. Use fedora-easy-karma to list all installed packages from the updates-testing repository and allow to file feedback on each one at a time. Pay particular attention to updates whose description notes that they are critical path updates.

Major bugs

If you identified any serious problems in earlier testing and were able to identify the package responsible, post negative feedback for that package. If possible, please file a bug report on the problem and link to the bug report in your feedback message. A good feedback message quickly and clearly identifies the behavior change and the cause, if you were able to determine it.

Minor bugs

If you identify a problem which is minor in nature and does not impede the actual critical path functionality, please do not post negative feedback. Post neutral or positive feedback with a note of the issue encountered (and a link to a bug report if appropriate).

Previously reported bugs

If your testing uncovers no problems but you see that another tester has identified a serious problem with the package, please try to replicate their problem, and post negative feedback if you are now able to confirm it. If you are not able to confirm the problem but you suspect this may be because you cannot recreate the necessary conditions, please post neutral feedback noting that you were unable to duplicate the problem. Only post positive feedback if you are sure your testing indicates the other reporter's negative feedback is a mistake.

Unfamiliar packages

If you are not sure what the component does or how to test it, do not post positive or negative feedback. If the above general tests of booting, network functionality and update functionality identified no problems, it is fine to leave a neutral feedback message noting that you were able to boot the system and perform critical path tasks with the update installed.

Becoming a proven tester

1. Sign up for the Fedora Account System. You must have a FAS account before you participate here.
2. Pick a mentor by requesting one through the Fedora QA Trac.
3. Apply for proven testers. Details of the procedure to apply for the Proven Testers QA group are below.

A mentor can be any current proven tester. An interested QA community member who would like to become sponsored may request a current Proven Tester as their Mentor as part of their trac ticket if they have a preference of who mentors them. Once the candidate Mentor accepts the interested QA Community Member's request this Mentor will oversee the interested member and help them work within the Critical Path QA efforts, policies, and procedures.

Once these steps are complete and your mentor confirms you have demonstrated an understanding of QA process and community, your application for approval to the Critical Path Wranglers (Proven Testers) will be brought up for review with the current proven testers. If accepted, your FAS account will be approved for the proventesters group.

External Links

• Bodhi
• Bugzilla
• Current roster of proventesters