From Fedora Project Wiki

Fedora 21 Delay Proposal: taskotron Development

Summary

While progress on automated testing has been slow as of late, the changes proposed for Fedora.next will be made much more workable with better automation in the QA space. Instead of working on small improvements as time allows between Fedora releases, we propose that Fedora 21 be delayed so that we can focus on getting our next generation automation system in place to support the new direction for Fedora.

Name Change
Due to potential trademark/copyright conflicts, we have re-named the project as Taskotron

Owner

  • Name: Tim Flink
  • Email: tflink at fedoraproject org
  • Release notes owner:

Current status

  • Targeted release: Fedora 21
  • Last updated: October 7, 2013
  • Tracker bug: None

Assumptions

There are very few specifics on the timeframe for the proposals requested by fesco for projects which could be done during if f21 was delayed for infrastructure/tooling work in qa, releng and other areas. Therefore, I am making several assumptions to make this proposal clearer.

  • Any delay would start after January 31, 2014
  • Available people is still TBD - this will depend on any other proposals considered and accepted by FESCo for the possible delay period
    • I assume that there will be at least 3 full time engineers dedicated to this work. There may be additional folks from the community but that part is still TBD as I haven't asked for commitments from anyone else yet.
  • Conservative estimates that under promise and over deliver are preferred.
    • There are still unknowns in this plan so I am making conservative estimates and leaving a wishlist of things to work on if time permits

Background

While AutoQA has served us well over the years, it has some issues that we need to fix in order to move forward. With this in mind, we have proposed a new automation system which we are calling taskotron. We will be preserving as much of AutoQA as makes sense but moving forward is a higher priority than reusing old code and solutions.

The development plan for taskotron describes several phases for incremental development of automation features. The full list of taskotron sub-projects is also available. taskotron Phase 0 is currently in progress and unless there are significant disruptions from Fedora 20 testing, will be complete before the end of January 2014.

Detailed Description

We have been planning and starting the investigation process for a new automation system which will eventually replace AutoQA. Unfortunately, this is far from a small undertaking and most of our resources are occupied with testing during the normal Fedora release cycle. There is some time between releases but it has proven to be far shorter than we would need to get a significant project started and finished before the next release.

By delaying the release of Fedora 21, we would be able to commit more resources to automation development and the work would be done more quickly. Exactly how quickly is addressed with scope.

Benefit to Fedora

The main benefit to Fedora would be more test coverage stemming from more automated checking and freeing up human testers to do more complicated tests than currently done as part of release validation. This new automation is coming whether the delay is accepted or not but the delay would free up more developers to work on implementation, bringing these benefits to Fedora faster.

Benefit To End Users

  • More stability in Fedora

Benefit To Developers

  • More automated checks
  • Better communication of results
  • Opportunity to have checks added to automation

Benefit To Testers

  • Less time spent on the validation treadmill doing repetitive tasks
  • More time to participate in test days, test new features


Scope

The scope of this proposal depends on the length of the delay being considered.

No Delay

1 Month Delay

2-3 Month Delay

After the completion of taskotron Phase 1, there are multiple options for more functionality to implement (Later phases in the development plan, other subprojects) but with the exception of implementing an ABI break/change tool (less than 1 month), there are too many unknowns for us to make reasonable estimates on the time required to complete those phases.

We do think it is reasonably likely that we'd be able to complete either taskotron Phase X, taskotron Phase Y or a subset of taskotron Phase Z given an additional 2 months (making for a 3 month total delay) but we are not willing to commit to their completion in that time frame until more of taskotron Phase 1 has been implemented.

User Experience

Fedora users may notice the delayed release and will hopefully notice an increase in quality.

Dependencies

There are no dependencies that are not listed in this proposal.

Contingency Plan

We keep working on automation with more limited resources. The scope of our plans will remain the same but our timelines will stretch out without the resources we would otherwise gain from delaying Fedora 21.

Documentation

Documentation is still being written but so far, we have:

Retrieved from "https://fedoraproject.org/w/index.php?title=User:Tflink/f21_delay_taskotron_development_proposal&oldid=360403"