From Fedora Project Wiki

Revision as of 16:55, 8 March 2010 by Kparal (talk | contribs) (answer to updates-testing questions)

mandatory or discretionary?

Let the policy be mandatory or discretionary? Can the package maintainer override the policy on will? -- Kparal 14:20, 18 February 2010 (UTC)

  • Kparal: Current discussions suggest this should be a mandatory policy. But of course there must be possibilities to gain an exception (shortened waiting time for highly important updates, etc).

security updates QA check

Even if security updates don't follow Package update policy, we should make sure we check them at least after their release. -- Kparal

Added to Responsibilities section. --Kparal 16:23, 8 March 2010 (UTC)

critical path packages

Should critical path packages be treated slightly differently from other packages? In what way? -- Kparal

important hot-fix exception

There may be cases when the update is not security fix, but it repairs severe regression/breakage and it is needed to land in updates as soon as possible. Should we offer a possibility to ask for exception and shorten the time required in updates-testing to e.g. 3 days (or even 0 days?). QA ACK should still be needed (to ensure the package is installable etc). The exception would be granted or refused by RelEng team, after considering if the update is really critical. -- Kparal

Example on exceptions process -- jlaska

time spent in updates-testing

Adamw noted that it could be very interesting to query Bodhi for past time experience. How long it usually takes to accumulate most of the positive/negative feedback? Is it in the first few days? Does substantial number of comments appear after the first week? If we can answer those question, we can then set the required time in 'updates-testing' much better. -- Kparal 12:24, 2 March 2010 (UTC)

policy for QA ACK decisions

Similarly to RelEng placeholder in the Workflow section, there might exist a document documenting rules for QA to accept/reject an update. This policy would then be implemented by Package update acceptance test plan. -- Kparal 11:48, 5 March 2010 (UTC)

Added a section "QA Acceptance Review guidelines". --Kparal

update release schedule

This topic is related, but should probably be part of a separate policy and not directly included in this one. It has been moved from the draft into this discussion page.

  • The 'updates-testing' repository is updated continuously, it is refreshed for every package update.
  • The 'updates' repository is updated regularly once a week for regular updates and continuously for security updates.
    • Will that help us improve quality by having updates as a weekly "pack" (with possible interactions and collisions, etc) instead of continuous stream of single updates? If QA can't leverage this, then it can be a daily push or a continuous push.
  • By default the end-user is notified once in a fortnight for regular updates and daily for security updates.
    • This may fall into other policy than Package update policy?

updates types

Which package update types should be allowed/disallowed? bugfixes, enhancements, new upstream versions, brand new packages, ... -- Kparal 17:09, 5 March 2010 (UTC)


Critique

Scope

Explain Better
Information on the talk page makes it seem like this could be better stated as: "Security updates need not follow this as the Security Team can bypass the waiting period to get the package in ASAP. However, the Quality Assurance team should still test the packages, after they're pushed if necessary, and report any regressions that need to be fixed to the maintainer. --abadger1999 22:13, 5 March 2010 (UTC)

Yes, that sounds better. Correction: Security Team may bypass not just the waiting period, but the whole policy (certainly QA approval, maybe even RelEng approval?). --Kparal 14:15, 8 March 2010 (UTC)

Yes, I think Security Team trumps everything. So maybe "Security updates need not follow this policy as the Security Team can bypass it to get the package in ASAP. However, the Quality Assurance team should still test the packages, after they're pushed if necessary, and report any regressions that need to be fixed to the maintainer." --abadger1999 15:48, 8 March 2010 (UTC)

Changed in the proposal, split between Scope and Responsibilities. Thanks. --Kparal 16:23, 8 March 2010 (UTC)

Workflow

All package updates are submitted to the 'updates-testing' repository. First they await Quality Assurance team approval. QA team approval is granted according to QA Acceptance Review guidelines. Then Release Engineering team approval is needed. RelEng team approval is granted according to RelEng Acceptance Review guidelines. When both approvals are granted, the update is pushed to the 'update-testing' repository.

  • I don't like bottlenecking before things get into updates-testing.
    • I don't think that releng should be involved at this stage. QA is doing the testing, not releng, correct?
    • I think automated testing could enter the equation at this stage; although maybe it could be overridable by the package maintainer... Going from updates-testing to updates might require QA override of failing automatic tests, though.

--abadger1999 22:13, 5 March 2010 (UTC)

Well this depends on how much we want to control updates-testing. Can anyone upload anything? Bump to any version, upload dozens updates a day? Is it a wild-west zone? Then we don't need RelEng approval. (When we have guidelines for RelEng then we can imagine more things that we would like to be protected from).
Do we want to allow to upload packages that break dependencies or that are not installable? Then we don't need QA approval (at least prior to pushing the update).
I don't say this is a bad approach, we just need to know what we want. For example we may require QA approval, but we can also allow it after the push (so broken packages may be pushed, but we find them eventually, and no time is lost in "bottlenecking").
As for QA approval overriding, it's an interesting idea, it may be included (for 'updates-testing', not for 'updates'). Just don't forget that a large set of our tests may already be waived, just the mandatory tests (package sanity etc) will force a rejection (and we will include ask-for-exception mechanism).
--Kparal 14:15, 8 March 2010 (UTC)

  • Currently anyone can upload anything. I think we probably still want anyone to upload anything to updates-testing. But there should definitely be autoqa tests before the package goes to updates.
  • Bumping versions: Our current packaging policies allow for enhancement as well as bugfix updates at the maintainers discretion. So there's really nothing to review at the updates-testing stage. The package maintainer could be using updates-testing to find out if the version bump has regressions that should keep it out of the stable repo.
  • dozens of updates per day -- This doesn't appear to end users at all. A push only happens about once a day. After the push, only a single build per package is in the repo. So this doesn't seem like a problem.
  • updates-testing probably should be something of a wild west zone due to the fact that it's the repo where testing is to occur.
    • Still don't understand what releng's role here, that's separate from QA's is. But we can move that discussion to a different section if they're no longer involved at this stage.
  • Breaking dependencies is an interesting question -- let's say that you need to update a library to a new ABI because upstream has released a critical bugfix. You are the maintainer of the library but not the maintainer of any of the dependent packages. You are not in provenpackager. Should we allow your update to go to updates-testing before the other packages are rebuilt? Getting it into updates-testing allows the other maintainers to find and get the package and do rebuilds quicker. OTOH, maybe being listed in bodhi is sufficient and avoids breaking updates-testing for people testing other packages via yum update. I lean towards allowing the dep breakage but it would be easy to convince me otherwise.
  • Overriding -- What kind of manual review are you anticipating, who is doing it, and how quickly? Without knowing what is being tested here and how quickly QA can respond to automated test results and pass things through that should be allowed anyway, it's very hard to make an informed decision here.

--abadger1999 16:13, 8 March 2010 (UTC)

  • I will ask a RelEng guy to help us answer RelEng related questions.
  • Errors in AutoQA introspection tests may be waived by the very package maintainer. We're still not absolutely sure about errors in mandatory tests. It should be very exceptional to ask for exception in mandatory tests.

--Kparal 16:55, 8 March 2010 (UTC)

The package updates must spend at least 14 days in the 'updates-testing' repository, or at least 7 days provided they have karma of at least 3 and no negative feedback. Only after that the package maintainer may submit them to the 'updates' repository. The Release Engineering team may approve or disapprove such an update according to RelEng Acceptance Review guidelines. If the approval is granted, the update is pushed to the 'updates' repository.

  • This is much more restrictive than current practices. Probably unacceptably more restrictive.
    • 14 days seems too long. I usually let a package sit for 1 week in updates-testing. On the list, Hans de Goede mentioned 4 days as desirable.
    • One problem with current pushing practices is that it may take multiple days for a package to be moved into a repository. So we have pushing to updates-testing cutting into the time that a package can be tested and in the other direction, once an update is approved to move to stable, it may continue to sit in updates-testing where users can't get the critical fix.
    • Current bodhi default is that +3 overall karma (settable by package maintainer) promotes to stable on the next available push. This piece changes both the amount of karma (+3 with 0 negative karma) and the time (next push that's seven days after submission)
  • There's no indication here of any gains being made from this waiting period. Is QA committing to testing updates but needs to have time to test each package? Whatever the gain, it should be stated here.
  • Also not sure that releng should be involved in approving every package. It makes more sense to have human (as opposed to automated) approval at this stage rather than updates-testing but it doesn't make much sense for releng to be signing off on this... you really want the people testing the package to sign off. Although currently releng does testing of the pre-release trees so perhaps releng members should join the QA group to give tested karma.

--abadger1999 22:13, 5 March 2010 (UTC)

  • The time interval (currently 14 days proposed) is certainly a value to discuss. From what I heard most people concluded it's too long. Some people even objected against fixed-time interval. I think the most important thing is that there is some interval present. There can be many mechanism to shorten it if necessary (important hot-fixes, karma, manual ask for exception at RelEng, ...). We don't want draconian terms. --Kparal 14:15, 8 March 2010 (UTC)
    • Need to have the rationale known before we can figure out what an appropriate time period is.--abadger1999 16:35, 8 March 2010 (UTC)
  • Time required for push should be as short as possible. We can include this in the RelEng guidelines. There can be also other improvements, like shortening time for critical fixes (see above), putting them into priority queue for push, etc. --Kparal 14:15, 8 March 2010 (UTC)
    • How are you going to actually shorten the time? My impression is that further shortening the time is hard because this is a manual process. Someone has to sign the packages and restart the process if any number of errors show up in the first push. Human time is limited therefore there's the distinct possibility of pushed taking variable amounts of time. The automated portions of the push (like mashing) take quite a bit of time themselves, so there's a hard limit on how short the pushes can be. --abadger1999 16:35, 8 March 2010 (UTC)
  • Current bodhi practices like direct submit to stable go against the requirement of a minimal time spent in updates-testing. Karma should certainly be taken into account, it can shorten the waiting period substantially (days value is subject to change). If there is any problem reported, I don't think it should be submitted automatically, the maintainer should review it. --Kparal 14:15, 8 March 2010 (UTC)
    • I can agree with negative karma requires the maintainer to make a decision. However, that would read like this: "If an update has reached +3 positive karma but negative karma is also present, the maintainer must manually decide whether to push the package to updates as the maintainer will need to evaluate whether the negative karma requires a different update be pushed out." The kernel and xorg drivers are prominent examples of why this is ultimately a maintainer decision: negative karma could mean that my wonky video card still doesn't work (therefore not a regression) while it has started working for another chipset. --abadger1999 16:35, 8 March 2010 (UTC)
    • On the list, Jesse stated that bypassing updates-testing was desirable if an update had gathered sufficient karma before being pushed to updates-testing. His rationale was that updates-testing is for testing -- +3 positive karma means that the update has already been tested. --abadger1999 16:35, 8 March 2010 (UTC)
    • Current practice is to remove the waiting period (not reduce it) once enough positive karma has been achieved. This goes along with Jesse's statement of bypassing updates-testing entirely --abadger1999 16:35, 8 March 2010 (UTC)
  • The rationale for waiting period should be included in the policy, correct. --Kparal 14:15, 8 March 2010 (UTC)
  • I will have to consult RelEng team to hear what are the possible dangers they look for when accepting packages to updates-testing. Then we will know if we can just omit that. --Kparal 14:15, 8 March 2010 (UTC)
  • Without having more information about the previous two bullet points, figuring out good values for time and selling maintainers on this idea are pretty hard. --abadger1999 16:35, 8 March 2010 (UTC)

QA Acceptance Review guidelines

  • TODO
  • The guidelines items which might be automated are implemented in Package update acceptance test plan. The result of this test plan must be ACCEPTED (it may be NEEDS_REVIEW initially and then changed to ACCEPTED by waiving warnings, that is allowed).

Need to get to the point where there's no TODO here otherwise people won't know what's really happening before their package is accepted. Make clear on this package that the Update Acceptance Test Plan is entirely for automated tests with human overrides for false positives. --abadger1999 22:34, 5 March 2010 (UTC)

RelEng Acceptance Review guidelines

  • TODO

TODO needs to be emptied out or simply take releng as a separate entity from QA out of the picture as noted above. --abadger1999 22:34, 5 March 2010 (UTC)

Responsibilities

  • When a serious problem is found in a not yet released updated package, package update builder is responsible for removing this update from 'updates-testing' repository. Fedora Update System must provide this option for package update builder, package maintainer, Quality Assurance team and Release Engineering team.

Unclear why responsibility is the package update builders to remove but the set of people who can remove is larger than that. This should be spelled out better. --abadger1999 22:43, 5 March 2010 (UTC)

We could also define "serious problem". Again, I hope some RelEng member will help me understand current practices. --Kparal 14:15, 8 March 2010 (UTC)


  • The Fedora Update System is responsible for sending notifications to package update builder and package maintainer when:
    • the update receives a result from Package update acceptance test plan
    • approval is granted or refused by Release Engineering team (both when submitting to 'updates-testing' or 'updates' repository)
    • the update has fulfilled requirements for submitting into 'updates' repository

Why not automatically submit to updates when requirements are met (as it currently is)? --abadger1999 22:43, 5 March 2010 (UTC)

For example I may want to keep my package in updates-testing for a longer time than a required minimum. This could be nicely solved by "Automatically submit to updates after requirements are met" checkbox in Bodhi when creating an update. --Kparal 14:15, 8 March 2010 (UTC)

Okay. Bodhi currently has an enable karma automatism checkbox. Perhaps this should be combined into a "push automatically if _requirements_ are satisfied" _requirements_ pops up help that shows the following:

Checking this box means the package will be automatically pushed to stable if no negative karma is given and either of the following two events occur:
  • Package has been in updates-testing for X days
  • Package has received +3 positive karma
If the package receives negative karma or you do not check this box, you will be notified via email when the package has satisfied either of the above. You can then promote the package to stable, remove the update and prepare a new package to replace it (for instance, if the negative karma shows there's a regression you need to fix) or continue to wait if you want the package to remain in updates-testing for longer. You will not receive another notification.

--abadger1999 16:48, 8 March 2010 (UTC)

    • the update becomes available in the 'updates' repository
Retrieved from "https://fedoraproject.org/w/index.php?title=User_talk:Kparal/Proposal:Package_update_policy&oldid=158093"