It is time that Fedora starts doing automated QA testing.
Many minds met during FUDCon F11 Boston 2009 in order to brain storm on automated testing. (link to video here)
We identified a number of "triggers" to react to in order to test things, a number of tests to perform, and a set of places to notify of the results. Here is a quick dump of these three things.
Triggers
- cvs checkin
- koji build
- bodhi request
- rawhide compose
- manual (periodic)
- iso compose
- new test creation
- post test
- comps change
Tests
- repository Sanity (sublist here)
- package installation
- profile testing
- tree sanity
- rpmdiff
- build log sanity
- source sanity
- build root sanity
- fails to build from source
- make check
- comps grammar
Notifiers
- <package>-owner@fedoraproject.org
- project webpage
- fedora-test-list@redhat.com or fedora-devel-list@redhat.com
Also discussed were timeline targets. Next 3 weeks, next 3 months, next 3 quarters. We feel we can accomplish a number of the triggers and tests within the next 3 weeks, focusing more package and repo sanity. In the next 3 months we can likely add functional testing of installer via Lab in a Box (link here). Over the next 3 quarters we hope to be able to expand functional testing to individual packages.