it boots, ship it
understand your role, and tailor it to your project
- as much qa as makes sense for the goals of the project, but no more
- qa should not dictate development planning or schedules
- qa should not decide on features
- principle: don't get in the way
fedora: six month cycle. short freeze periods: F15 Alpha 8 days, F15 Beta 8 days, F15 Final 10 days: 26 days frozen per release (with no slips). can't hold up the release so QA can work frozen.
so: work outside freezes. fedora - test composes, rats. testing a moving target is bad, but better than not testing at all. limit scope of work done during freezes to what's most important and can actually be accomplished. fedora - install+desktop validation testing. importance of distinguishing between things that can be fixed with updates and things that can't. work fast: make sure appropriate groups (fedora: qa, development, releng) have direct links between the people who do the work.
http://rbergero.fedorapeople.org/schedules/f-15/f-15-all-tasks.html
use your people wisely
- paid staff
- volunteer contributors
yes, the difference is important: don't pretend it isn't. paid staff have an obligation to show up and do stuff; volunteers don't. this has to be taken into account in scheduling and staffing tasks. for things that absolutely must be done, make sure a paid staff person knows to do it if no-one else does.
contributors: be as flexible as you can. run events as long as possible, don't shut them off when the paid people are off work. give as much notice as possible of events and releases. remember differing schedules: contributors are often _more_ available/active on weekends, so don't leave things till monday. make things as easy as possible to get involved with, and self-starting if possible. never reject input, but direct it politely: use all the input you can find, even if you have to go look for it (forums, mailing lists...look for people testing your stuff). fedora - test days, tc and rc releases / validation testing. Join page. updates-testing in pre-releases.
dangers: never have important stuff done in a closed loop among paid staff: keep it in the proper public fora. don't value work and input of paid staff above that of contributors: neither is always right or more important than the other. leads into...
infrastructure - kiss, and write everything down
if it's dumb and it works, it's not dumb. use existing infrastructure where you can, don't over-design. get small things working, don't draw up Grand Plans. use as few tools and sites as possible: always ask if we can do this with the tools we already have. don't separate QA processes from the rest of the distro. fedora - mailing list, wiki, trac and irc cover almost everything. release management works through bugzilla, irc and infra trac.
document your processes. bus/lottery principle. if you feel like no-one else can possibly do what you do, that's bad: avoid superhero rock star syndrome. e.g. - warlyzilla. documented processes defuse tension: clear that decisions are being taken on a rational and pre-existing basis, not out of castle building or spite. documented processes make sure everyone can be involved in them and understand. documented processes make sure you always do things the same and don't forget anything. fedora - SOPs! rel blocker process, nth process, proven tester process, test day management, validation process.
dangers: don't get hidebound: remember the purposes processes are meant to serve and be ready to adjust them when necessary. if the release criteria turn out to be wrong, adjust them, don't make silly decisions because they're Written In Stone. but adjust the process, don't just ignore it.
make sure everything is public and archived. use public mailing lists, public irc and public sites for everything. you can remember what you did before, you can tell everyone else, and no-one thinks you're hiding anything. secrecy almost never has any benefit. never get caught in a 'secrecy by default' mindset.
QA:Testcase_nouveau_basic (a test case! in a wiki!) QA:SOP_blocker_bug_process http://lists.fedoraproject.org/pipermail/test/ (test ml archive)
engage
engage with other groups. make sure to understand what they want and also what they can contribute. links with processes, above. fedora - developer input into release criteria, SIG input into desktop validation testing qa/releng/devel loop on release blocker process, releng co-operation on package specific test cases in bodhi. don't work in a corner and pass down pronouncements from on high.
engage with the community and your volunteers: publicize! blogs, mailing lists, news sites, social media - whatever works for you. go and find the people relevant to your projects, don't wait for them to come to you. fedora - fwn, major test events on phoronix etc. forums, and comments on major releases: your users are in forums, even if you hate them.
http://www.phoronix.com/scan.php?page=news_item&px=OTM1MA (phoronix on gnome test day) http://www.happyassassin.net/2011/03/29/abrt-test-day-on-thursday-and-beta-validation-starting/ (announcing a test day) http://forums.fedoraforum.org/forumdisplay.php?f=80 (fedora NEXT forum) https://admin.fedoraproject.org/updates/xorg-x11-drv-nouveau-0.0.16-24.20110324git8378443.fc15 (update with test cases)