User talk:Kparal/Proposal:Package update policy

workflow more readable

Numbered list or some flow diagram could make the workflow section more readable. -- Kparal 13:31, 18 February 2010 (UTC)

• Flow diagram done. -- Kparal 16:52, 5 March 2010 (UTC)

mandatory or discretionary?

Let the policy be mandatory or discretionary? Can the package maintainer override the policy on will? -- Kparal 14:20, 18 February 2010 (UTC)

• Kparal: Current discussions suggest this should be a mandatory policy. But of course there must be possibilities to gain an exception (shortened waiting time for highly important updates, etc).

security updates QA check

Even if security updates don't follow Package update policy, we should make sure we check them at least after their release. -- Kparal

critical path packages

Should critical path packages be treated slightly differently from other packages? In what way? -- Kparal

important hot-fix exception

There may be cases when the update is not security fix, but it repairs severe regression/breakage and it is needed to land in updates as soon as possible. Should we offer a possibility to ask for exception and shorten the time required in updates-testing to e.g. 3 days (or even 0 days?). QA ACK should still be needed (to ensure the package is installable etc). The exception would be granted or refused by RelEng team, after considering if the update is really critical. -- Kparal

Example on exceptions process -- jlaska

time spent in updates-testing

Adamw noted that it could be very interesting to query Bodhi for past time experience. How long it usually takes to accumulate most of the positive/negative feedback? Is it in the first few days? Does substantial number of comments appear after the first week? If we can answer those question, we can then set the required time in 'updates-testing' much better. -- Kparal 12:24, 2 March 2010 (UTC)

policy for QA ACK decisions

Similarly to RelEng placeholder in the Workflow section, there might exist a document documenting rules for QA to accept/reject an update. This policy would then be implemented by Package update acceptance test plan. -- Kparal 11:48, 5 March 2010 (UTC)

update release schedule

This topic is related, but should probably be part of a separate policy and not directly included in this one. It has been moved from the draft into this discussion page.

• The 'updates-testing' repository is updated continuously, it is refreshed for every package update.
• The 'updates' repository is updated regularly once a week for regular updates and continuously for security updates.
  • Will that help us improve quality by having updates as a weekly "pack" (with possible interactions and collisions, etc) instead of continuous stream of single updates? If QA can't leverage this, then it can be a daily push or a continuous push.
• By default the end-user is notified once in a fortnight for regular updates and daily for security updates.
  • This may fall into other policy than Package update policy?

updates types

Which package update types should be allowed/disallowed? bugfixes, enhancements, new upstream versions, brand new packages, ... -- Kparal 17:09, 5 March 2010 (UTC)

Critique

Scope

• This policy is recommended also for security updates, but Security Response Team may choose not to follow it.

Explain Better
Information on the talk page makes it seem like this could be better stated as: "Security updates need not follow this as the Security Team can bypass the waiting period to get the package in ASAP. However, the Quality Assurance team should still test the packages, after they're pushed if necessary, and report any regressions that need to be fixed to the maintainer. --abadger1999 22:13, 5 March 2010 (UTC)

Yes, that sounds better. Correction: Security Team may bypass not just the waiting period, but the whole policy (certainly QA approval, maybe even RelEng approval?). --Kparal 14:15, 8 March 2010 (UTC)

Workflow

All package updates are submitted to the 'updates-testing' repository. First they await Quality Assurance team approval. QA team approval is granted according to QA Acceptance Review guidelines. Then Release Engineering team approval is needed. RelEng team approval is granted according to RelEng Acceptance Review guidelines. When both approvals are granted, the update is pushed to the 'update-testing' repository.

• I don't like bottlenecking before things get into updates-testing.
  • I don't think that releng should be involved at this stage. QA is doing the testing, not releng, correct?
  • I think automated testing could enter the equation at this stage; although maybe it could be overridable by the package maintainer... Going from updates-testing to updates might require QA override of failing automatic tests, though. --abadger1999 22:13, 5 March 2010 (UTC)

Well this depends on how much we want to control updates-testing. Can anyone upload anything? Bump to any version, upload dozens updates a day? Is it a wild-west zone? Then we don't need RelEng approval. (When we have guidelines for RelEng then we can imagine more things that we would like to be protected from).
Do we want to allow to upload packages that break dependencies or that are not installable? Then we don't need QA approval (at least prior to pushing the update).
I don't say this is a bad approach, we just need to know what we want. For example we may require QA approval, but we can also allow it after the push (so broken packages may be pushed, but we find them eventually, and no time is lost in "bottlenecking").
As for QA approval overriding, it's an interesting idea, it may be included (for 'updates-testing', not for 'updates'). Just don't forget that a large set of our tests may already be waived, just to mandatory tests (package sanity etc) will force a rejection (and we will include ask-for-exception mechanism). --Kparal 14:15, 8 March 2010 (UTC)

The package updates must spend at least 14 days in the 'updates-testing' repository, or at least 7 days provided they have karma of at least 3 and no negative feedback. Only after that the package maintainer may submit them to the 'updates' repository. The Release Engineering team may approve or disapprove such an update according to RelEng Acceptance Review guidelines. If the approval is granted, the update is pushed to the 'updates' repository.

• This is much more restrictive than current practices. Probably unacceptably more restrictive.
  • 14 days seems too long. I usually let a package sit for 1 week in updates-testing. On the list, Hans de Goede mentioned 4 days as desirable.
  • One problem with current pushing practices is that it may take multiple days for a package to be moved into a repository. So we have pushing to updates-testing cutting into the time that a package can be tested and in the other direction, once an update is approved to move to stable, it may continue to sit in updates-testing where users can't get the critical fix.
  • Current bodhi default is that +3 overall karma (settable by package maintainer) promotes to stable on the next available push. This piece changes both the amount of karma (+3 with 0 negative karma) and the time (next push that's seven days after submission)
• There's no indication here of any gains being made from this waiting period. Is QA committing to testing updates but needs to have time to test each package? Whatever the gain, it should be stated here.
• Also not sure that releng should be involved in approving every package. It makes more sense to have human (as opposed to automated) approval at this stage rather than updates-testing but it doesn't make much sense for releng to be signing off on this... you really want the people testing the package to sign off. Although currently releng does testing of the pre-release trees so perhaps releng members should join the QA group to give tested karma.

--abadger1999 22:13, 5 March 2010 (UTC)

• The time interval (currently 14 days proposed) is certainly a value to discuss. From what I heard most people concluded it's too long. Some people even objected against fixed-time interval. I think the most important thing is that there is some interval present. There can be many mechanism to shorten it if necessary (important hot-fixes, karma, manual ask for exception at RelEng, ...). We don't want draconian terms.
• Time required for push should be as short as possible. We can include this in the RelEng guidelines. There can be also other improvements, like shortening time for critical fixes (see above), putting them into priority queue for push, etc.
• Current bodhi practices like direct submit to stable go against the requirement of a minimal time spent in updates-testing. Karma should certainly be taken into account, it can shorten the waiting period substantially (days value is subject to change). If there is any problem reported, I don't think it should be submitted automatically, the maintainer should review it.
• The rationale for waiting period should be included in the policy, correct.
• I will have to consult RelEng team to hear what are the possible dangers they look for when accepting packages to updates-testing. Then we will know if we can just omit that.

--Kparal 14:15, 8 March 2010 (UTC)

QA Acceptance Review guidelines

• TODO
• The guidelines items which might be automated are implemented in Package update acceptance test plan. The result of this test plan must be ACCEPTED (it may be NEEDS_REVIEW initially and then changed to ACCEPTED by waiving warnings, that is allowed).

Need to get to the point where there's no TODO here otherwise people won't know what's really happening before their package is accepted. Make clear on this package that the Update Acceptance Test Plan is entirely for automated tests with human overrides for false positives. --abadger1999 22:34, 5 March 2010 (UTC)

RelEng Acceptance Review guidelines

• TODO

TODO needs to be emptied out or simply take releng as a separate entity from QA out of the picture as noted above. --abadger1999 22:34, 5 March 2010 (UTC)

Responsibilities

• When a serious problem is found in a not yet released updated package, package update builder is responsible for removing this update from 'updates-testing' repository. Fedora Update System must provide this option for package update builder, package maintainer, Quality Assurance team and Release Engineering team.

Unclear why responsibility is the package update builders to remove but the set of people who can remove is larger than that. This should be spelled out better. --abadger1999 22:43, 5 March 2010 (UTC)

We could also define "serious problem". Again, I hope some RelEng member will help me understand current practices. --Kparal 14:15, 8 March 2010 (UTC)

• The Fedora Update System is responsible for sending notifications to package update builder and package maintainer when:
  • the update receives a result from Package update acceptance test plan
  • approval is granted or refused by Release Engineering team (both when submitting to 'updates-testing' or 'updates' repository)
  • the update has fulfilled requirements for submitting into 'updates' repository

Why not automatically submit to updates when requirements are met (as it currently is)? --abadger1999 22:43, 5 March 2010 (UTC)

For example I may want to keep my package in updates-testing for a longer time than a required minimum. This could be nicely solved by "Automatically submit to updates after requirements are met" checkbox in Bodhi when creating an update. --Kparal 14:15, 8 March 2010 (UTC)

• • the update becomes available in the 'updates' repository