Introduction
There are a lot of moving bits and pieces to where we'd like to see qa automation go for Fedora. While it would be great to say that all of it could get done in a month or two, that's really not practical. We're aiming for a phased milestone approach to replacing and improving our existing systems in Fedora.
The list of projects that we're planning on is listed elsewhere. The aim of this document is to describe the order in which we're planning to tackle those tasks.
Phases
Phase 0: Investigation and Preparation
Phase 0 (if it can be called a phase) is mostly for investigation and preparation. There are still non-trivial details that need to be ironed out and decisions made on initial direction and preparation work that still needs to be done.
Preparation
We still need to set up and/or configure tools like a bug tracker, CI, code reviews and repositories. This doesn't mean that we're going to set up our own instances for everything - some or all of those roles will be filled by existing Fedora services.
Investigation
The large item for investigation is Task execution
- investigate the possible strategies
- make a decision on which one to implement
- determine whether a central database for tasks is needed
We also need to investigate the possibility of missing messages for scheduling and verify that the current strategy is likely to work.
Deliverables
- Decision on method and plan for task execution
- Results of fedmsg investigation and plan for any scheduling work that will be needed for phase 1
- Support tooling configured, setup and ready for use
Phase 1: AutoQA Replacement
In order to reduce the amount of resources (both computing and human) spent on automation, the focus of phase 1 will be to fill the roles currently filled by AutoQA
Checks Run for All Packages/Updates
- depcheck
- rpmlint
- upgradepath
- repoclosure
- rpmguard?
Depcheck
Depcheck does not currently work with yum in Fedora 19 and newer. As Fedora 18 will be going EOL approximately 30 days after the release of Fedora 20, this needs to be addressed soon.
This can be dealt with in one of two ways:
- replace depcheck with a similar test that is easier to maintain
- fix depcheck so that it works with yum in Fedora 19 and later, delaying the need to replace depcheck
Result Notification
Instead of implementing new methods for notifying maintainers of results, we will continue with using bodhi comments for check result notification.
Deliverables
- Deployment of functional system capable of the following:
- running appropriate checks on koji builds
- running appropriate checks on bodhi updates
- notification of results through bodhi comments
what to do about package checks that aren't appropriate for bodhi? what do we want to do about email notification?
Phase X: Beaker Integration
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases
Phase Y: Installation Checks
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases
Phase Z: More Sophisticated Checks
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases
Later Work
This is not a complete plan for qa automation but detailed plans farther out isn't going to be productive at this time.
Some discussion of eventual goals as a high level roadmap, however, would be very helpful.