workflow more readable
Numbered list or some flow diagram could make the workflow section more readable. -- Kparal 13:31, 18 February 2010 (UTC)
- Flow diagram done. -- Kparal 16:52, 5 March 2010 (UTC)
mandatory or discretionary?
Let the policy be mandatory or discretionary? Can the package maintainer override the policy on will? -- Kparal 14:20, 18 February 2010 (UTC)
- Kparal: Current discussions suggest this should be a mandatory policy. But of course there must be possibilities to gain an exception (shortened waiting time for highly important updates, etc).
security updates QA check
Even if security updates don't follow Package update policy, we should make sure we check them at least after their release. -- Kparal
critical path packages
Should critical path packages be treated slightly differently from other packages? In what way? -- Kparal
important hot-fix exception
There may be cases when the update is not security fix, but it repairs severe regression/breakage and it is needed to land in updates as soon as possible. Should we offer a possibility to ask for exception and shorten the time required in updates-testing to e.g. 3 days (or even 0 days?). QA ACK should still be needed (to ensure the package is installable etc). The exception would be granted or refused by RelEng team, after considering if the update is really critical. -- Kparal
Example on exceptions process -- jlaska
time spent in updates-testing
Adamw noted that it could be very interesting to query Bodhi for past time experience. How long it usually takes to accumulate most of the positive/negative feedback? Is it in the first few days? Does substantial number of comments appear after the first week? If we can answer those question, we can then set the required time in 'updates-testing' much better. -- Kparal 12:24, 2 March 2010 (UTC)
policy for QA ACK decisions
Similarly to RelEng placeholder in the Workflow section, there might exist a document documenting rules for QA to accept/reject an update. This policy would then be implemented by Package update acceptance test plan. -- Kparal 11:48, 5 March 2010 (UTC)
update release schedule
This topic is related, but should probably be part of a separate policy and not directly included in this one. It has been moved from the draft into this discussion page.
- The 'updates-testing' repository is updated continuously, it is refreshed for every package update.
- The 'updates' repository is updated regularly once a week for regular updates and continuously for security updates.
- Will that help us improve quality by having updates as a weekly "pack" (with possible interactions and collisions, etc) instead of continuous stream of single updates? If QA can't leverage this, then it can be a daily push or a continuous push.
- By default the end-user is notified once in a fortnight for regular updates and daily for security updates.
- This may fall into other policy than Package update policy?
updates types
Which package update types should be allowed/disallowed? bugfixes, enhancements, new upstream versions, brand new packages, ... -- Kparal 17:09, 5 March 2010 (UTC)
Critique
Scope
- This policy is recommended also for security updates, but Security Response Team may choose not to follow it.
Information on the talk page makes it seem like this could be better stated as: "Security updates need not follow this as the Security Team can bypass the waiting period to get the package in ASAP. However, the Quality Assurance team should still test the packages, after they're pushed if necessary, and report any regressions that need to be fixed to the maintainer. --abadger1999 22:13, 5 March 2010 (UTC)
Workflow
All package updates are submitted to the 'updates-testing' repository. First they await Quality Assurance team approval. QA team approval is granted according to QA Acceptance Review guidelines. Then Release Engineering team approval is needed. RelEng team approval is granted according to RelEng Acceptance Review guidelines. When both approvals are granted, the update is pushed to the 'update-testing' repository.
- I don't like bottlenecking before things get into updates-testing.
- I don't think that releng should be involved at this stage. QA is doing the testing, not releng, correct?
- I think automated testing could enter the equation at this stage; although maybe it could be overridable by the package maintainer... Going from updates-testing to updates might require QA override of failing automatic tests, though. --abadger1999 22:13, 5 March 2010 (UTC)
The package updates must spend at least 14 days in the 'updates-testing' repository, or at least 7 days provided they have karma of at least 3 and no negative feedback. Only after that the package maintainer may submit them to the 'updates' repository. The Release Engineering team may approve or disapprove such an update according to RelEng Acceptance Review guidelines. If the approval is granted, the update is pushed to the 'updates' repository.
- This is much more restrictive than current practices. Probably unacceptably more restrictive.
- 14 days seems too long. I usually let a package sit for 1 week in updates-testing. On the list, Hans de Goede mentioned 4 days as desirable.
- One problem with current pushing practices is that it may take multiple days for a package to be moved into a repository. So we have pushing to updates-testing cutting into the time that a package can be tested and in the other direction, once an update is approved to move to stable, it may continue to sit in updates-testing where users can't get the critical fix.
- Current bodhi default is that +3 overall karma (settable by package maintainer) promotes to stable on the next available push. This piece changes both the amount of karma (+3 with 0 negative karma) and the time (next push that's seven days after submission)
- There's no indication here of any gains being made from this waiting period. Is QA committing to testing updates but needs to have time to test each package? Whatever the gain, it should be stated here.
- Also not sure that releng should be involved in approving every package. It makes more sense to have human (as opposed to automated) approval at this stage rather than updates-testing but it doesn't make much sense for releng to be signing off on this... you really want the people testing the package to sign off. Although currently releng does testing of the pre-release trees so perhaps releng members should join the QA group to give tested karma.
--abadger1999 22:13, 5 March 2010 (UTC)
QA Acceptance Review guidelines
- TODO
- The guidelines items which might be automated are implemented in Package update acceptance test plan. The result of this test plan must be ACCEPTED (it may be NEEDS_REVIEW initially and then changed to ACCEPTED by waiving warnings, that is allowed).
Need to get to the point where there's no TODO here otherwise people won't know what's really happening before their package is accepted. Make clear on this package that the Update Acceptance Test Plan is entirely for automated tests with human overrides for false positives. --abadger1999 22:34, 5 March 2010 (UTC)
RelEng Acceptance Review guidelines
- TODO
TODO needs to be emptied out or simply take releng as a separate entity from QA out of the picture as noted above. --abadger1999 22:34, 5 March 2010 (UTC)
Responsibilities
- When a serious problem is found in a not yet released updated package, package update builder is responsible for removing this update from 'updates-testing' repository. Fedora Update System must provide this option for package update builder, package maintainer, Quality Assurance team and Release Engineering team.
Unclear why responsibility is the package update builders to remove but the set of people who can remove is larger than that. This should be spelled out better. --abadger1999 22:43, 5 March 2010 (UTC)
- The Fedora Update System is responsible for sending notifications to package update builder and package maintainer when:
- the update receives a result from Package update acceptance test plan
- approval is granted or refused by Release Engineering team (both when submitting to 'updates-testing' or 'updates' repository)
- the update has fulfilled requirements for submitting into 'updates' repository
Why not automatically submit to updates when requirements are met (as it currently is)? --abadger1999 22:43, 5 March 2010 (UTC)
- the update becomes available in the 'updates' repository