User:Tflink/taskotron development plan

Introduction

There are a lot of moving bits and pieces to where we'd like to see qa automation go for Fedora. While it would be great to say that all of it could get done in a month or two, that's really not practical. We're aiming for a phased milestone approach to replacing and improving our existing systems in Fedora.

The list of projects that we're planning on is listed elsewhere. The aim of this document is to describe the order in which we're planning to tackle those tasks.

Phases

Phase 0: Investigation and Preparation

Phase 0 (if it can be called a phase) is mostly for investigation and preparation. There are still non-trivial details that need to be ironed out and decisions made on initial direction and preparation work that still needs to be done.

Preparation

We still need to set up and/or configure tools like a bug tracker, CI, code reviews and repositories. This doesn't mean that we're going to set up our own instances for everything - some or all of those roles will be filled by existing Fedora services.

Investigation

The large item for investigation is Task execution

• investigate the possible strategies
• make a decision on which one to implement
• determine whether a central database for tasks is needed

We also need to investigate the possibility of missing messages for scheduling and verify that the current strategy is likely to work.

Deliverables

• Decision on method and plan for task execution
• Results of fedmsg investigation and plan for any scheduling work that will be needed for phase 1
• Support tooling configured, setup and ready for use

Phase 1: AutoQA Replacement

In order to reduce the amount of resources (both computing and human) spent on automation, the focus of phase 1 will be to fill the roles currently filled by AutoQA

Checks Run for All Packages/Updates

• depcheck
• rpmlint
• upgradepath
• repoclosure
• rpmguard?

Depcheck

Depcheck does not currently work with yum in Fedora 19 and newer. As Fedora 18 will be going EOL approximately 30 days after the release of Fedora 20, this needs to be addressed soon.

This can be dealt with in one of two ways:

• replace depcheck with a similar test that is easier to maintain
• fix depcheck so that it works with yum in Fedora 19 and later, delaying the need to replace depcheck

Result Notification

Instead of implementing new methods for notifying maintainers of results, we will continue with using bodhi comments for check result notification.

Deliverables

• Deployment of functional system capable of the following:
  • running appropriate checks on koji builds
  • running appropriate checks on bodhi updates
  • notification of results through bodhi comments

Notification for builds
what to do about package checks that aren't appropriate for bodhi? what do we want to do about email notification?

Phase X: Beaker Integration

TBD
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases

Red Hat uses beaker for a lot of internal testing and we're working towards getting some of the checks from that ecosystem open sourced. Assuming that things go forward as planned, it will likely be worth enabling job submission to beaker from taskbot and integration of any relevant results.

Beaker Job Submission

• Figure out how to use beaker's api to submit jobs

Beaker Results Integration

• Figure out how to retrieve logs and job status from beaker
• Determine what data we need to be retrieving

Automate Beaker Job Updates

Once we have beaker jobs, it would be better if we had some sort of CI system which was capable of rebuilding tasks on change and pushing them to the beaker server.

Phase Y: Installation Checks

TBD
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases

Phase Z: More Sophisticated Checks

TBD
This is just a placeholder for now, we want to nail down phase 0 and phase 1 a bit better before getting into the order and detail of later phases

Later Work

This is not a complete plan for qa automation but detailed plans farther out isn't going to be productive at this time.

Some discussion of eventual goals as a high level roadmap, however, would be very helpful.