User:Dafrito/Proven tester

A proven tester, or critical path wrangler, verifies and reports on the stability of updates to critical path packages. Acknowledgment of an update's stability is indicated through positive karma awarded in Bodhi. A positive vote from at least one proven tester is required before an update in updates-testing can be pushed to the stable repository. This requirement provides a measure of assurance that updates to critical packages are subject to basic verification before they are widely deployed.

The proven tester role, like others in Fedora, is not an exclusive position; individuals are free to go beyond what is required. Similarly, regular testers are free and encouraged to provide karma on both critical and non-critical packages.

Test criteria

A positive rating from a proven tester indicates that the rated update does not make the system unstable. Proven testers should vote positively on an update that keeps the package stable, even if it introduces minor bugs of its own. They should also restrict their vote to the update, not the package as a whole; major bugs or regressions for a package do not disqualify all updates that do not correct those problems. This focus on pragmatism, rather than purity, is designed to strike a balance between the stability of an update and the timeliness of that update's release.

Stable updates

A stable update can be roughly defined as an update that, when applied, does not cause, or introduce the possibility of, severe regressions to a user's system. The severity of a regression or bug is dependent on how, and how often, it impacts the user's experience. This impact can occur via a serious reduction of functionality, damage or data loss, or an increased level of vulnerability. Manual intervention should never be required to avoid a serious regression. Planned breakages, such as removing deprecated functionality or upgrading to an incompatible version, are not necessarily considered regressions. Benign messages and warnings, while they should be reported, are also not sufficient on their own to withhold a positive vote.

While some specific requirements exist for both the system as a whole and for different categories of packages, the diverse range of functionality provided by critical packages, combined with an even more diverse set of use cases, makes defining a baseline of stability difficult. Ultimately, a proven tester's judgment and objectiveness is necessary to ensure a sufficient and consistent level of stability across packages.

System-wide requirements

At a minimum, the update, when applied, should still allow for the successful execution of all critical path actions. These actions serve as the baseline of stability for any update:

• The system must boot. A user must be able to boot, log in to X, log off, and shut down without performing any unusual modifications or previously unnecessary steps. Some examples include having to modify grub, changing the way the system is booted, or having to avoid some option that was previously operational.
• The network should be accessible. A user should be able to perform common network-related tasks, such as viewing web pages, checking their email, and SSH'ing into other machines.
• Packages must update without error or intervention. A user should be able to graphically install new updates from Fedora without requiring manual intervention, such as using "--skip-broken" or manually resolving changed dependencies.
• Security must not be adversely affected. Security features, such as firewalls and permissions, must continue to operate at an expected level.

Package-specific requirements

Proven testers expand their requirements of a stable update to also include the major functionality of the updated package. What makes a feature major, or a regression severe, ultimately depends on the package, but proven testers should follow this common set of guidelines as they apply. For example, an update to a library or shared component cannot be tested easily in isolation, so instead, an application that depends on that library should be used to verify stability. This list, while not exhaustive, mentions a few areas of testability:

• The application must start and stop cleanly. A user must be able to start up and close the application without freezing or requiring the application to be forcefully closed.
• The application must not cause data loss. The user should be able to create, save, and load their work. Existing data should not be corrupted, unexpectedly modified, or lost.

Providing feedback

Proven testers fulfill their role by voting with Bodhi karma on updates they have tested. Generally speaking, positive karma from a proven tester is an indication of their trust in the stability of that update. Conversely, negative karma indicates that the update is considered unstable. Neutral karma should be left in cases of an untestable or insufficiently tested update.

Proven testers should leave comments with their karma, especially with negative karma, to indicate the reasons for their vote. If a severe regression is found, the comment should identify the behavior change and, if possible, the cause. Comments should also note problems that were found but did not warrant a negative vote. Bug reports are useful supplements in either case. These additional measures assist the maintainers in fixing the problem, and they guide other testers by giving them opportunities to confirm reported issues.

Insufficiently tested updates, such as packages that are very unfamiliar or incompatible, should not be given positive feedback. For example, a major update to CUPS can be installed and examined, but it should not ordinarily be given positive feedback if the proven tester does not have access to a printer. The proven tester should post neutral karma if the examined portions were stable and comment that only a limited amount of the update was tested.

Collaborating and verifying the results of other testers allows problems to be narrowed down and solved quickly. Comments, bug reports, and votes by other testers can be used to streamline a proven tester's job. Multiple votes in the same direction builds confidence in the stability, or instability, of that update. If a proven tester finds that karma has already been submitted for an update, she should attempt to confirm those results, posting her findings as a vote and comment of her own.

If a proven tester is unable to reproduce a problem reported by another tester, he should indicate this as a comment and a neutral vote. If in doubt, unreproducible bugs should be treated as problems with the update, rather than mistakes. Positive karma should only be given if the tester is confident that the negative result was in error.

Becoming a proven tester

Individuals become proven testers by being sponsored by a current proven tester. This sponsorship may be requested by creating a ticket in the Fedora QA Trac. The sponsor is the chosen mentor for the candidate, providing answers, suggestions, and guidance for the procedures and inner-working of testing critical updates. When the applicant becomes proficient in this process, the candidate's application will be submitted to the group of proven testers. The formal mentorship ends when the applicant is approved, and the candidate becomes an independent proven tester.

In practice, a proven tester will know how to:

• Install updates from updates-testing. A proven tester may choose to enable this repository, or he may pick updates from testable packages. The tester should also be willing to install these updates on their system: since the proven tester will be working with untested updates, manual intervention and troubleshooting may be necessary to restore the system to a fully working state.
• Submit karma and comments. Karma may be submitted through fedora-easy-karma or through the Bodhi web interface. A proven tester should be familiar with submitting karma through at least one of these methods. An account with Fedora is required to submit karma.
• Rate critical updates according to proven tester guidelines. Since a proven tester has an important role in ensuring the stability and consistency of updates, she should follow the standards listed on this page when she submits karma.

Mentorship is designed to reduce the barriers of entry for proven testers, so those that aren't familiar with some of the concepts are still encouraged to apply. In other words, there is no need to be proficient before submitting an application. Also, since none of these requirements require proven tester access, individuals can try out the testing, karma, and bug reporting systems themselves before they request a sponsorship.

External Links

• Bodhi
• Bugzilla
• Current roster of proventesters