QA:Update feedback guidelines

Introduction

This page presents general guidelines for providing feedback regarding candidate updates, using the Bodhi system. Bodhi explains how you should determine whether to provide positive, negative, neutral or no feedback for any given candidate update. For instructions on obtaining and installing candidate updates and posting feedback, see QA:Updates Testing. For a convenient tool which provides a command-line interface that allows you to quickly leave feedback regarding all or some candidate updates installed on your system, see the Fedora_Easy_Karma page.

Feedback entries

The current version of Bodhi provides several different items in the Feedback section for each update, including one for each bug the update claims to fix (if any), one for each test case known to be associated with the package(s) in the update (if any), and two standard ones: "Does the system's basic functionality continue to work after this update?" and "Is the update generally functional?"

If you use Fedora_Easy_Karma, it only allows you to provide feedback for a single item; your response is used to answer "Is the update generally functional?". This is because fedora-easy-karma was written for an older Bodhi implementation which did not have multiple feedback items, and has not yet been updated.

This page is mostly concerned with how you should answer the question "Is the update generally functional?", which is the most significant feedback item, because only the responses for this feedback item are used to calculate the update's "karma". Each positive response for this feedback item from a registered user counts as +1, and each negative response for this feedback item counts as -1. Later in this page, you will find some instructions for providing responses for the other feedback items.

'Is the update generally functional?'

Positive feedback

If you installed one or more candidate update(s), and found that the updated package seems to work as expected, and you do not encounter any of the situations below, you can leave positive feedback and note that you were able to use the package successfully and did not notice any significant problems.

Please be aware that not all packages are involved in minute-to-minute system use. Please do read the package description and the update text to try and understand the purpose of the package and the update, and make a genuine effort to ensure you have actually exercised the package's functions before leaving positive feedback.

Tip: test package removal to see package's purpose
One trick for deciding how to test an update if you are not sure what it does or why you have it installed is to test removing the package in order to see what depends on it. For example, run dnf remove (package), look at the list of other packages that would be removed along with it, then hit 'n' so the removal will not actually happen. This can give you an idea what applications you actually use directly may depend on the package being tested.

In particular, if a package is not already installed on your system, simply installing it and then leaving positive feedback if the system appears to continue to function correctly is not a good idea, as the package may well not be used at all in this case.

Please read all sections below, especially #Unfamiliar_packages, before posting positive feedback.

Neutral feedback or no feedback?

Neutral feedback - 0 karma, marked in the web interface as an empty box - is intended to be used in the case where you have a significant comment to make on the update, but you cannot with confidence post positive or negative feedback. For instance, you noticed what you think may be a bug, but you are not sure. It is not meant to be used in the case where you have nothing useful to contribute; active feedback is not required in this case. If you are using a tool like fedora-easy-karma and it presents an update you cannot confidently provide any feedback on, simply leave no feedback at all, do not leave neutral feedback with a note saying 'not tested' or similar. You can skip an update without leaving any comment in fedora-easy-karma by hitting 's'.

Major bugs

If you identified any serious problems in your testing and were able to identify the update responsible, check that the bug is new - that it did not exist in the stable version of the package - and then post negative feedback for that update. Do not post negative feedback against an update simply because it still contains a bug that already existed in the previous version of the package. Bear in mind that the function of indicating negative feedback is to prevent an update from being released; it makes no sense to file a negative feedback about an update which is no worse than the previous version of the package.

If possible, please file a bug report explaining the problem encountered and provide a link to the bug report in your feedback message. A good feedback message quickly and clearly identifies the behavior change and the cause, if you were able to determine it.

Minor bugs

If you identify a problem in an update which is minor in nature, use judgement to decide whether to post negative feedback. Consider whether overall, the update constitutes an improvement on the situation with the previous package - for instance, if an update fixes a significant security package but introduces a minor cosmetic bug, negative feedback is probably not appropriate. Instead, post neutral or positive feedback with a note explaining the issue encountered (and if appropriate, a link to a bug report).

Previously reported bugs

If your testing uncovers no problems but you see that another tester has identified a serious problem with the package, please try to replicate their test, and post negative feedback if you are now able to confirm their problem. If you are not able to confirm the problem but you suspect this may be because you cannot recreate the necessary conditions, please post neutral feedback noting that you were unable to duplicate the problem. Only post positive feedback if you are sure your testing indicates the other reporter's negative feedback is in error.

Update does not fix a bug it claims to

If you find an update does not fix a bug it claims to fix, this is not usually a case where you should file negative karma. Only file negative karma if that is the only change in the update. If an update claims to fix five bugs, but only fixes four of them, it is not helpful to post negative karma as this may result in the update being rejected, which does not help those suffering from the bug that wasn't fixed, and hurts those suffering from the bugs that are fixed.

You can report whether the update fixes each of the bugs it claims to fix by using the appropriate Feedback item in the web UI, separately from this 'Is the update generally functional?' feedback item. See below for more details on this. It is perfectly OK to leave a negative response for an individual bug but a positive response for 'Is the update generally functional?', if the update generally works and provides other benefits beyond fixing that bug.

Unfamiliar packages

If you are not sure what the component does or how to test it, do not post positive or negative feedback. For critical path updates, if general tests of booting, network functionality and update functionality identified no problems, it is fine to leave a neutral feedback message noting that you were able to boot the system and perform critical path tasks with the update installed. This is generally not useful for non-critical path updates, however: please only comment on them if you are familiar with the package and able to test it directly.

'Does the system's basic functionality continue to work after this update?'

This feedback item is quite self-explanatory: just answer the question! It is probably a good idea to reboot after installing updates and before providing any feedback of this type. It is also quite unlikely that you would want to give negative feedback for this, but positive feedback for Is the update generally functional? - usually an update which breaks 'basic functionality' should not be considered 'generally functional'.

Per-bug feedback items

When a packager submits an update, they may mark it as fixing specific bugs. If the update submitter does this, a feedback entry will be included for each specific bug listed, and as a tester, you may provide feedback on whether the update actually fixes the bug. This is visible only in the web interface, fedora-easy-karma does not offer this type of feedback entry.

To leave either positive or negative feedback, you must first follow the information included in the bug report and reproduce the bug with the older version(s) of the affected package(s). Then install the updated package(s), run through the steps to reproduce the bug, and see if it happens or not. If it does not, leave positive feedback for this entry. If the bug still happens, leave negative feedback. It is also a good idea to add a comment to the bug report in this case, explaining how you tried to verify the fix and what actually happened.

If you cannot reproduce the bug and verify the fix in this way, do not leave positive or negative feedback for the bug's feedback entry - just leave the blank box selected.

Note that both these responses to an update can be valid:

• Positive feedback for one or more bug items but negative feedback for Is the update generally functional? and/or Does the system's basic functionality continue to work after this update?

• Negative feedback for one or more bug items but positive feedback for Is the update generally functional? and/or Does the system's basic functionality continue to work after this update?

The first case is appropriate if an update fixes a bug it claims to fix, but also introduces a new bug which causes significant problems. The second case is appropriate if an update does not fix a bug it claims to fix, but does provide other improvements, and generally functions acceptably well.

Test case feedback items

Test cases can be associated with packages - see the package test plan creation SOP for details on how. When an update is submitted for a package with associated test cases, a feedback item will be included for each associated test case. This is visible only in the web interface, fedora-easy-karma does not offer this type of feedback item at present.

To leave either positive or negative feedback for these items, run through the test case with the updated package installed. For extra thoroughness, you can run through the test case before updating the package, so that if the test case fails, you will know whether it was already broken before or not.

If you successfully complete the test case, you can leave positive feedback for the relevant item. If the test case fails, you can leave negative feedback. In this case it is a good idea to also write a comment explaining what went wrong - and, ideally, file a bug report.

As with the per-bug feedback items, it is not always the case that a negative response to a test case means your response to the two standard feedback items should be negative, or a positive response to a test case item means your response to the two standard feedback items should be positive. An update may fail a trivial test case but provide a serious security fix, in which case you should probably give a positive response to 'Is the update generally functional?'; and an update may pass all its test cases but still have a serious bug which the test cases do not cover, in which case you may well want to give a negative response to 'Is the update generally functional?'.