Overview
The aim of the proven testers project is to ensure that updates to critical path packages do not cause major functional regressions. Two documents to keep in mind as you test are the critical path definition and the release criteria. If a proposed update to a critical path package causes problems with the critical path actions, or a new issue or regression which would constitute an infringement of the release criteria, then you provide negative feedback on the update.
Testing process
Here is a suggested workflow for testing and providing feedback on updates. Feel free to vary the details as suits you, but please bear in mind the advice on what types of feedback to leave in what cases. It is of course fine to quickly install one specific critical path update and test it in isolation, if you are particularly interested in ensuring the update is tested promptly, or the package maintainer sends out a request for testing.
- Enable the Fedora updates-testing repository, if you haven't already, and do a full system update each day.
- After doing the update, reboot the system and log in to the desktop (assuming you can). If you observe any problems or changed behaviour, take a detailed note and try to identify the package responsible for the change.
- Ensure your network access is working as usual. If not, try to identify the cause.
- Check that the update application will run, although you now likely will have no updates remaining to test immediately whether it works. If not, try to identify the cause.
- Use the Fedora_Easy_Karma tool to list all installed packages from the updates-testing repository and allow to file feedback on each one at a time.
- Pay particular attention to updates whose description notes that they are critical path updates. If you identified any serious problems in earlier testing and were able to identify the package responsible, post negative feedback for that package. If possible, please file a bug report on the problem and link to the bug report in your feedback message. A good feedback message quickly and clearly identifies the behaviour change and the cause, if you were able to determine it.
- For each update which is identified as being on the critical path, try to perform a quick functionality test to ensure the component is working as intended. If it is an application, run the application and check that basic operations work. If it is a library or other shared component, run an application which uses the component and ensure that that works. If the component works as expected, post positive feedback. If you identify a major problem, post negative feedback. If you identify a problem which is minor in nature and does not impede the actual critical path functionality, please do not post negative feedback. Post neutral or positive feedback with a note of the issue encountered (and a link to a bug report if appropriate).
- If you are not sure what the component does or how to test it, do not post positive or negative feedback. If the above general tests of booting, network functionality and update functionality identified no problems, it is fine to leave a neutral feedback message noting that you were able to boot the system and perform critical path tasks with the update installed.
- If your testing uncovers no problems but you see that another tester has identified a serious problem with the package, please try to replicate their problem, and post negative feedback if you are now able to confirm it. If you are not able to confirm the problem but you suspect this may be because you cannot recreate the necessary conditions, please post neutral feedback noting that you were unable to duplicate the problem. Only post positive feedback if you are sure your testing indicates the other reporter's negative feedback is a mistake.