Fedora Testing Meeting - November 9 2006, 1700 UTC, #fedora-testing on freenode
See FedoraTesting for more info about the Fedora Testing project.
Agenda
1. FC6 testing recap a. What was good, what was bad? a. Why did we delay, and how can we avoid that in the future? 1. Beaker status: a. Status of RHTS (found at testing.108.redhat.com) a. How this will affect Fedora 1. updates-testing a. Do we need an extras-testing? a. What's the policy for moving from updates-testing to updates? a. Who actually tests these packages? 1. bzbot a. Do we want to have a bzbot on freenode? a. Where can we host it? a. Who wants to maintain it?
Notes
Attendees: wwoods, craigt, dmalcolm, ixs, poelcat, rdieter, thl
FC6 testing
Major problems identified:
- Lack of ppc testers
Can we find ppc users and work with them to get help during Test releases?
Target a couple specific machines: powerbook, G5 desktop
- No documentation for testing new features (Xen, iSCSI)
Require how-to-test documentation from developers for all major features
iSCSI docs: http://linux-iscsi.sourceforge.net/, http://www.cuddletech.com/articles/iscsi/index.html
- post-freeze yum behavior change
Try not to change installer behavior after the freeze!
Manual Testing - Test Plans
12:33 < rdieter> maybe such "how to test" documents ought to come-from/contributed-to by maintainers of relavent packages (in Fedora). 12:34 <@wwoods> rdieter: yes! that would be an excellent idea. 12:34 < dmalcolm> i.e. for each feature proposed for Core, we need a "how do I get to play with it"? document? ... 12:41 < poelcat> I think it would be very easy to increase the amount of test coverage if we have clear instructions (tests cases) on what to test 12:42 * craigt agrees with poelcat
a. As above, we need "how-to-test" documentation for packages in Core / Extras, especially for new features a. ACTION: wwoods will create test-plan template for wiki "test plan" is (informally) defined as a how-to-test document in the wiki. a. Start writing test plans for common, important things (yum, firefox, etc.) a. Ask package maintainers to help write test plans a. Need a way to track execution of test plans, i.e. "wwoods tested ImageMagick.i386 on Nov. 9 2006"
Automated Testing - RHTS
12:46 < dmalcolm> the idea is to have a common packaging model for tests 12:46 < dmalcolm> so that for each test, you can make an RPM of it 12:46 < dmalcolm> I've used this to make yum repos of tests 12:46 < dmalcolm> for example: http://people.redhat.com/dmalcolm/tablecloth/tests/sandbox/ 12:47 < dmalcolm> and http://people.redhat.com/dmalcolm/tablecloth/tests/promoted/ 12:47 < dmalcolm> so you can install an RPM of a test and run it 12:47 < dmalcolm> or simply run the test from a checkout of the test source code ... 12:48 < dmalcolm> tests identify what packages they test you can do things like "yum install tests-of(ImageMagick)-\*"
a. http://testing.108.redhat.com/ hosts RHTS and other associated tools/projects a. RHTS yum repo: http://people.redhat.com/dmalcolm/tablecloth/tools/ a. We want package maintainers and testers to write new tests! a. Test writing guide: https://wiki.108.redhat.com/wiki/index.php/Testing/Rhts/Docs/TestWriting A simple shell script or other test program can be wrapped into a new test with "rhts-create-new-test", from the rhts-devel package a. Red Hat has purchased hardware for running tests on (see ["QA/Beaker"] )
Testing updates: the updates-testing repo
a. rdieter and wwoods advocate the creation of extras-testing, like updates-testing
a. automated tests should run on all packages pushed into -testing
a. rdieter: packages "cannot leave updates-testing if tests reported as failed"
dmalcolm: "I think if we have automated tests I'd want a way of waiving false positives."
a. updates-testing policy:
13:02 <@wwoods> I'd be much happier with updates-testing and updates if someone tested every package that went into updates ... 13:18 < craigt> as to requireing testing singoff, that just doesn't seem possible to me 13:18 <@wwoods> craigt: not enough testers? or too hard to get stuff tested quickly? 13:19 < craigt> not enough testers (which seems the == too hard to get stuff tested quickly)
i. If at all possible, someone should attempt to install and run the package, and report results to fedora-test-list. If there isn't a how-to-test document for that package, the tester and/or developer should write one. i. If there are any automated tests for a package in updates-testing, they should be run with results reported to fedora-test-list. If the automated tests fail, the package should not be allowed to move to updates. However, automated test results may be waived if there's a good reason - discuss on fedora-test-list. i. Given the current number of testers (rough guess: 6), we do not have the manpower to require the above items. Therefore, if there are no complaints on fedora-test-list after a week, a package may move to updates.
Other
a. BugZappers and Fedora Testers should merge! a. Make a tester-HOWTO, covering the following topics: 1. How to test packages in updates-testing 1. How to perform bug triage 1. How to write test plans / automated tests 1. How to test the installer a. bzbot i. If someone wants to set it up, that's fine, but demand is not high enough. i. RSS feed for FC6 bugs from the past 7 days: http://rdr.to/RI