QA talk:Package Update Acceptance Test Plan

jlaska's test ideas

      * All updates must include a new changelog entry - someday I'd
        like to require a bug (or ticket) in the changelog entry, but
        perhaps that's too aggressive now.
      * What MUST sections can we automate from the package review
        guidelines [2]?
      * SPEC file sanity, including ...
              * Proper upstream Source URL included in SPEC?
              * When are changes to %config files are acceptable?
              * Is %defattr defined in the SPEC?
              * Any sanity tests we can do against the %scripts included
                in a spec file
              * How to handle Unapplied %patches?
      * License compat review?
      * Stripped vs unstripped binaries, is there a preference?
      * Validate man pages?
      * What existing *lint tools can we run, and what results are
        acceptable? (rpmlint, elflint, xmllint)
      * Any relationship to the new privilege escalation policy [3]?

[2] https://fedoraproject.org/wiki/Packaging:ReviewGuidelines
[3] https://fedoraproject.org/wiki/Privilege_escalation_policy

wwoods thoughts

• The introduction needs to be clear that this is an acceptance test plan - All these tests have to pass before we can even think about functional testing of the package.
  • Specifically: it needs to be clear that when an update has PASSED this test plan, that just means it's ready for real testing. The actual testing of the update is not complete; it has just barely started at this point.
  • Maybe the final result of the test plan should reflect this: If all the test cases pass, the package is ACCEPTED, otherwise it's REJECTED.
    • Each test case can still use PASS/FAIL, of course.
    • NEEDS_INSPECTION is fine as-is.

• If possible, we should have links for each of the listed test cases that outline exactly what's being tested (and/or link to the source code).

failed mandatory test can be brought to FESCO

Seth Vidal has provided me with an idea that if the package maintainers don't agree with a failed mandatory test (they claim it should pass), the issue can be brought to FESCO. FESCO could e.g. grant an exception for that package or deny the request.