From Fedora Project Wiki
m (→‎Package-specific requirements: Added a transition to the list)
m (Changed to use Template:package)
 
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{for|the original version|Proven tester}}
A '''proven tester''', or '''critical path wrangler''', verifies and reports on the stability of updates to [[critical path package]]s. Acknowledgment of an update's stability is indicated through positive karma awarded in [[Bodhi]]. A positive vote from at least one proven tester is required before an update in [[updates-testing]] can be pushed to the stable repository. This requirement provides a measure of assurance that updates to critical packages are subject to basic verification before they are widely deployed.
A '''proven tester''', or '''critical path wrangler''', verifies and reports on the stability of updates to [[critical path package]]s. Acknowledgment of an update's stability is indicated through positive karma awarded in [[Bodhi]]. A positive vote from at least one proven tester is required before an update in [[updates-testing]] can be pushed to the stable repository. This requirement provides a measure of assurance that updates to critical packages are subject to basic verification before they are widely deployed.
The proven tester role, like others in Fedora, is not an exclusive position; individuals are free to go beyond what is required. Similarly, regular testers are free and encouraged to provide karma on both critical and non-critical packages.


== Test criteria ==
== Test criteria ==
Line 8: Line 12:
A stable update can be roughly defined as an update that, when applied, does not cause, or introduce the possibility of, severe regressions to a user's system. The severity of a regression or bug is dependent on how, and how often, it impacts the user's experience. This impact can occur via a serious reduction of functionality, damage or data loss, or an increased level of vulnerability. Manual intervention should never be required to avoid a serious regression. Planned breakages, such as removing deprecated functionality or upgrading to an incompatible version, are not necessarily considered regressions. Benign messages and warnings, while they should be reported, are also not sufficient on their own to withhold a positive vote.
A stable update can be roughly defined as an update that, when applied, does not cause, or introduce the possibility of, severe regressions to a user's system. The severity of a regression or bug is dependent on how, and how often, it impacts the user's experience. This impact can occur via a serious reduction of functionality, damage or data loss, or an increased level of vulnerability. Manual intervention should never be required to avoid a serious regression. Planned breakages, such as removing deprecated functionality or upgrading to an incompatible version, are not necessarily considered regressions. Benign messages and warnings, while they should be reported, are also not sufficient on their own to withhold a positive vote.


Some specific requirements do exist for both the system as a whole and for different categories of packages. However, the diverse range of functionality provided by critical packages, combined with an even more diverse set of use cases, makes defining a baseline of stability difficult. Ultimately, a proven tester's judgment and objectiveness is necessary to ensure a sufficient and consistent level of stability across packages.  
While some specific requirements exist for both the system as a whole and for different categories of packages, the diverse range of functionality provided by critical packages, combined with an even more diverse set of use cases, makes defining a baseline of stability difficult. Ultimately, a proven tester's judgment and objectiveness is necessary to ensure a sufficient and consistent level of stability across packages.  


=== System-wide requirements ===
=== System-wide requirements ===
Line 26: Line 30:
* ''The application must not cause data loss.'' The user should be able to create, save, and load their work. Existing data should not be corrupted, unexpectedly modified, or lost.
* ''The application must not cause data loss.'' The user should be able to create, save, and load their work. Existing data should not be corrupted, unexpectedly modified, or lost.


== Feedback Procedures ==
== Providing feedback ==
Since a proven tester's karma determines whether an update is allowed to be promoted, they follow special procedures based on the severity of regressions they encounter. Use {{package|fedora-easy-karma}} to list all installed packages from the updates-testing repository and allow to file feedback on each one at a time. Pay particular attention to updates whose description notes that they are critical path updates.  
Proven testers fulfill their role by voting with Bodhi karma on updates they have tested. Generally speaking, positive karma from a proven tester is an indication of their trust in the stability of that update. Conversely, negative karma indicates that the update is considered unstable. Neutral karma should be left in cases of an untestable or insufficiently tested update.


=== Major bugs ===
Proven testers should leave comments with their karma, especially with negative karma, to indicate the reasons for their vote. If a severe regression is found, the comment should identify the behavior change and, if possible, the cause. Comments should also note problems that were found but did not warrant a negative vote. Bug reports are useful supplements in either case. These additional measures assist the maintainers in fixing the problem, and they guide other testers by giving them opportunities to confirm reported issues.
If you identified any serious problems in earlier testing and were able to identify the package responsible, post negative feedback for that package. If possible, please file a bug report on the problem and link to the bug report in your feedback message. A good feedback message quickly and clearly identifies the behavior change and the cause, if you were able to determine it.


=== Minor bugs ===
Insufficiently tested updates, such as packages that are very unfamiliar or incompatible, should not be given positive feedback. For example, a major update to {{package|cups}} can be installed and examined, but it should not ordinarily be given positive feedback if the proven tester does not have access to a printer. The proven tester should post neutral karma if the examined portions were stable and comment that only a limited amount of the update was tested.
If you identify a problem which is minor in nature and does not impede the actual critical path functionality, please do not post negative feedback. Post neutral or positive feedback with a note of the issue encountered (and a link to a bug report if appropriate).


=== Previously reported bugs ===
Collaborating and verifying the results of other testers allows problems to be narrowed down and solved quickly. Comments, bug reports, and votes by other testers can be used to streamline a proven tester's job. Multiple votes in the same direction builds confidence in the stability, or instability, of that update. If a proven tester finds that karma has already been submitted for an update, she should attempt to confirm those results, posting her findings as a vote and comment of her own.
If your testing uncovers no problems but you see that another tester has identified a serious problem with the package, please try to replicate their problem, and post negative feedback if you are now able to confirm it. If you are not able to confirm the problem but you suspect this may be because you cannot recreate the necessary conditions, please post neutral feedback noting that you were unable to duplicate the problem. Only post positive feedback if you are sure your testing indicates the other reporter's negative feedback is a mistake.


=== Unfamiliar packages ===
If a proven tester is unable to reproduce a problem reported by another tester, he should indicate this as a comment and a neutral vote. If in doubt, unreproducible bugs should be treated as problems with the update, rather than mistakes. Positive karma should only be given if the tester is confident that the negative result was in error.
If you are not sure what the component does or how to test it, do not post positive or negative feedback. If the above general tests of booting, network functionality and update functionality identified no problems, it is fine to leave a neutral feedback message noting that you were able to boot the system and perform critical path tasks with the update installed.


== Becoming a proven tester ==
== Becoming a proven tester ==
# Sign up for the [https://admin.fedoraproject.org/accounts/login Fedora Account System]. You must have a FAS account before you participate here.
# Pick a mentor by requesting one through the [https://fedorahosted.org/fedora-qa/ Fedora QA Trac].
# Apply for proven testers. Details of the procedure to apply for the Proven Testers QA group are below.


A mentor can be any current proven tester. An interested QA community member who would like to become sponsored may request a current Proven Tester as their Mentor as part of their trac ticket if they have a preference of who mentors them. Once the candidate Mentor accepts the interested QA Community Member's request this Mentor will oversee the interested member and help them work within the Critical Path QA efforts, policies, and procedures.
Individuals become proven testers by being sponsored by a current proven tester. The sponsor is the chosen mentor for the candidate, providing answers, suggestions, and guidance for the procedures and inner-working of testing critical updates. When the applicant becomes proficient in this process, the candidate's application will be submitted to the group of proven testers. The formal mentorship ends when the applicant is approved, and the candidate becomes an independent proven tester.
 
In practice, a proven tester should know how to:
* ''Install updates from updates-testing.'' A proven tester may choose to enable this repository, or he may pick updates from testable packages. The tester should also be willing to install these updates on their system: since the proven tester will be working with untested updates, manual intervention and troubleshooting may be necessary to restore the system to a fully working state.
* ''Submit karma and comments.'' Karma may be submitted using {{command|[[Fedora_Easy_Karma|fedora-easy-karma]]}} or through the [http://bodhi.fedoraproject.org Bodhi web interface]. A proven tester should be familiar with submitting karma through at least one of these methods. [https://admin.fedoraproject.org/accounts/login An account with Fedora] is required to submit karma.
* ''Rate critical updates according to proven tester guidelines.'' Since a proven tester has an important role in ensuring the stability and consistency of updates, she should follow the standards listed on this page when she submits karma.
 
This sponsorship may be requested by creating a ticket in the [https://fedorahosted.org/fedora-qa/ Fedora QA Trac]. The ticket should contain information on the applicant's experience with Fedora, especially experience with the listed topics. The applicant should also mention a preferred mentor or a desired focus, such a specific desktop environment. This information will help ensure the applicant gets paired with an appropriate mentor. The ticket does not need to be thorough or lengthy - its primary purpose is to introduce the applicant to the proven tester group. Once the ticket is accepted, the applicant should request membership in the [https://admin.fedoraproject.org/accounts/group/view/proventesters proventester] group.


Once these steps are complete and your mentor confirms you have demonstrated an understanding of QA process and community, your application for approval to the Critical Path Wranglers (Proven Testers) will be brought up for review with the current proven testers. If accepted, your [https://admin.fedoraproject.org/accounts FAS] account will be approved for the [https://admin.fedoraproject.org/accounts/group/view/proventesters proventesters] group.
Mentorship is designed to reduce the barriers of entry for proven testers; those that aren't familiar with some of the concepts are still encouraged to apply. In other words, there is no need to be proficient before submitting an application. Also, since none of these requirements require proven tester access, individuals can try out the testing, karma, and bug reporting systems themselves before they request a sponsorship.


== External Links ==
== External Links ==

Latest revision as of 19:32, 6 July 2010

For the original version, see Proven tester.

A proven tester, or critical path wrangler, verifies and reports on the stability of updates to critical path packages. Acknowledgment of an update's stability is indicated through positive karma awarded in Bodhi. A positive vote from at least one proven tester is required before an update in updates-testing can be pushed to the stable repository. This requirement provides a measure of assurance that updates to critical packages are subject to basic verification before they are widely deployed.

The proven tester role, like others in Fedora, is not an exclusive position; individuals are free to go beyond what is required. Similarly, regular testers are free and encouraged to provide karma on both critical and non-critical packages.

Test criteria

A positive rating from a proven tester indicates that the rated update does not make the system unstable. Proven testers should vote positively on an update that keeps the package stable, even if it introduces minor bugs of its own. They should also restrict their vote to the update, not the package as a whole; major bugs or regressions for a package do not disqualify all updates that do not correct those problems. This focus on pragmatism, rather than purity, is designed to strike a balance between the stability of an update and the timeliness of that update's release.

Stable updates

A stable update can be roughly defined as an update that, when applied, does not cause, or introduce the possibility of, severe regressions to a user's system. The severity of a regression or bug is dependent on how, and how often, it impacts the user's experience. This impact can occur via a serious reduction of functionality, damage or data loss, or an increased level of vulnerability. Manual intervention should never be required to avoid a serious regression. Planned breakages, such as removing deprecated functionality or upgrading to an incompatible version, are not necessarily considered regressions. Benign messages and warnings, while they should be reported, are also not sufficient on their own to withhold a positive vote.

While some specific requirements exist for both the system as a whole and for different categories of packages, the diverse range of functionality provided by critical packages, combined with an even more diverse set of use cases, makes defining a baseline of stability difficult. Ultimately, a proven tester's judgment and objectiveness is necessary to ensure a sufficient and consistent level of stability across packages.

System-wide requirements

At a minimum, the update, when applied, should still allow for the successful execution of all critical path actions. These actions serve as the baseline of stability for any update:

  • The system must boot. A user must be able to boot, log in to X, log off, and shut down without performing any unusual modifications or previously unnecessary steps. Some examples include having to modify grub, changing the way the system is booted, or having to avoid some option that was previously operational.
  • The network should be accessible. A user should be able to perform common network-related tasks, such as viewing web pages, checking their email, and SSH'ing into other machines.
  • Packages must update without error or intervention. A user should be able to graphically install new updates from Fedora without requiring manual intervention, such as using "--skip-broken" or manually resolving changed dependencies.
  • Security must not be adversely affected. Security features, such as firewalls and permissions, must continue to operate at an expected level.

Package-specific requirements

Proven testers expand their requirements of a stable update to also include the major functionality of the updated package. What makes a feature major, or a regression severe, ultimately depends on the package, but proven testers should follow this common set of guidelines as they apply. For example, an update to a library or shared component cannot be tested easily in isolation, so instead, an application that depends on that library should be used to verify stability. This list, while not exhaustive, mentions a few areas of testability:

  • The application must start and stop cleanly. A user must be able to start up and close the application without freezing or requiring the application to be forcefully closed.
  • The application must not cause data loss. The user should be able to create, save, and load their work. Existing data should not be corrupted, unexpectedly modified, or lost.

Providing feedback

Proven testers fulfill their role by voting with Bodhi karma on updates they have tested. Generally speaking, positive karma from a proven tester is an indication of their trust in the stability of that update. Conversely, negative karma indicates that the update is considered unstable. Neutral karma should be left in cases of an untestable or insufficiently tested update.

Proven testers should leave comments with their karma, especially with negative karma, to indicate the reasons for their vote. If a severe regression is found, the comment should identify the behavior change and, if possible, the cause. Comments should also note problems that were found but did not warrant a negative vote. Bug reports are useful supplements in either case. These additional measures assist the maintainers in fixing the problem, and they guide other testers by giving them opportunities to confirm reported issues.

Insufficiently tested updates, such as packages that are very unfamiliar or incompatible, should not be given positive feedback. For example, a major update to cups can be installed and examined, but it should not ordinarily be given positive feedback if the proven tester does not have access to a printer. The proven tester should post neutral karma if the examined portions were stable and comment that only a limited amount of the update was tested.

Collaborating and verifying the results of other testers allows problems to be narrowed down and solved quickly. Comments, bug reports, and votes by other testers can be used to streamline a proven tester's job. Multiple votes in the same direction builds confidence in the stability, or instability, of that update. If a proven tester finds that karma has already been submitted for an update, she should attempt to confirm those results, posting her findings as a vote and comment of her own.

If a proven tester is unable to reproduce a problem reported by another tester, he should indicate this as a comment and a neutral vote. If in doubt, unreproducible bugs should be treated as problems with the update, rather than mistakes. Positive karma should only be given if the tester is confident that the negative result was in error.

Becoming a proven tester

Individuals become proven testers by being sponsored by a current proven tester. The sponsor is the chosen mentor for the candidate, providing answers, suggestions, and guidance for the procedures and inner-working of testing critical updates. When the applicant becomes proficient in this process, the candidate's application will be submitted to the group of proven testers. The formal mentorship ends when the applicant is approved, and the candidate becomes an independent proven tester.

In practice, a proven tester should know how to:

  • Install updates from updates-testing. A proven tester may choose to enable this repository, or he may pick updates from testable packages. The tester should also be willing to install these updates on their system: since the proven tester will be working with untested updates, manual intervention and troubleshooting may be necessary to restore the system to a fully working state.
  • Submit karma and comments. Karma may be submitted using fedora-easy-karma or through the Bodhi web interface. A proven tester should be familiar with submitting karma through at least one of these methods. An account with Fedora is required to submit karma.
  • Rate critical updates according to proven tester guidelines. Since a proven tester has an important role in ensuring the stability and consistency of updates, she should follow the standards listed on this page when she submits karma.

This sponsorship may be requested by creating a ticket in the Fedora QA Trac. The ticket should contain information on the applicant's experience with Fedora, especially experience with the listed topics. The applicant should also mention a preferred mentor or a desired focus, such a specific desktop environment. This information will help ensure the applicant gets paired with an appropriate mentor. The ticket does not need to be thorough or lengthy - its primary purpose is to introduce the applicant to the proven tester group. Once the ticket is accepted, the applicant should request membership in the proventester group.

Mentorship is designed to reduce the barriers of entry for proven testers; those that aren't familiar with some of the concepts are still encouraged to apply. In other words, there is no need to be proficient before submitting an application. Also, since none of these requirements require proven tester access, individuals can try out the testing, karma, and bug reporting systems themselves before they request a sponsorship.

External Links