From Fedora Project Wiki
(Draft)
 
m (Add questions section)
 
(40 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This is a copy of [[QA:Rawhide_Acceptance_Test_Plan]] but uses [[Template:QA/Test_Plan]] instead.
{{header|qa}}
{{draft}}


{{QA/Test_Plan
This page will outline criteria for success for a Fedora AMI release
|history=
* First draft: [[User:Wwoods|WillWoods]] 20:54, 17 June 2009 (UTC)


|introduction=
= Release Criteria =
This test plan documents the process used to check the basic requirements for a Rawhide tree to be acceptable for further testing. It aims to check whether Rawhide is installable, usable as a package repo for updating, and whether critical packages are present and functional.


In short, this is how we decide if Rawhide is broken or not.
In addition to the existing [[Fedora Release Criteria]], the following criteria apply to EC2 Fedora images.  A <span style="color:green; font-weight:bold;">good</span> AMI '''must'''...
|strategy=
# allow non-root user login
There are three main components here: Repo sanity, Installability, and Basic Functionality. These three categories can be tested mostly independent of one another.
# have SELinux enabled and ''enforcing'' by default
# not contain any package dependency problems
# not include any packages built outside of Fedora infrastructure


|priority=
= How to Test =
TBD. No clear priorities exist yet.


|scope=
To ''create'' an EC2 image, one needs...
This plan seeks to answer three basic questions:
# an amazon account
# the {{package|python-boto}} libraries installed
# a python script to tell Amazon to build an instance
# the ability to SSH into a machine to run tests, etc.


# Can current Rawhide users update their systems using this repo?
= Open Questions =
# Can this Rawhide tree be installed?
# Who is responsible for creating and building EC2 images?
# Does the basic system (a subset of the [[Critical_Path_Packages_Proposal|"critical path"]]) work as expected for simple testing?
#:


It is ''not'' intended to be an exhaustive test of any part of the system.
= Additional Reading =
 
* [[Publishing_image_to_EC2]]
|criteria=
* [[Cloud_SIG]]
Rawhide will be considered ''Good'' for each purpose if all of the underlying conditions are met:
* [[Cloud_SIG/EC2_Images]]
== Repo Sanity ==
* Contains valid yum metadata (repodata)
* Contains key packages (kernel, glibc, coreutils)
* No unresolved dependencies in critical packages
* comps.xml exists and is valid
 
== Installability ==
* installer images (kernel, initrd, install.img, and boot.iso) exist
* kernel boots on ''most'' machines of the primary architectures
* initrd is able to find stage2 (install.img) by at least one method (network, local CD)
* stage2 is able to detect the presence of disks attached to most common controllers
* installer can write packages to disk storage
* installer can setup up bootloader correctly for common boot systems
 
== Basic Functionality ==
* Kernel/X is able to set up common display configurations
** At least 2 out of 3 of the most common video drivers (intel, nouveau, radeon) must be functioning
* Kernel/X properly handle input from standard USB keyboard/mouse
* System is able to establish a wired network connection
** As with video, this should work for ''most'' common network drivers. Individual chipset failures may be waived.
* {{package|yum}} is able to install simple updates
 
|deliverables=
This test plan should produce:
* A summary report on whether Rawhide is broken or not
* Bug reports for broken dependencies / missing files / etc.
* A list of test cases used to verify the expected results. (see below)
 
|testcases=
''TODO'': Automate these test cases. See [https://fedorahosted.org/autoqa/query?status=new&status=assigned&status=reopened&component=tests&summary=~Automate&milestone=israwhidebroken.com&order=priority tickets for each test here].
 
== Repo Sanity ==
# [[QA:Repodata validity test case|Repodata validity]]
# [[QA:Comps Validity Test Case|comps.xml validity]]
# [[QA:Core package dependency closure test case|Core package dependency closure]]
# [[QA:Core package existence test case|Core package existence]]
 
== Installability ==
# [[QA:Installer image presence test case|Installer image existence]]
# [[QA:Kernel simple boot test case|Kernel boot]]
# [[QA:Anaconda stage2 fetch test case|Anaconda loader fetching stage2]]
# [[QA:Anaconda storage probe test case|Anaconda stage2 disk probe]]
# [[QA:Anaconda package install test case|Anaconda package install]]
# [[QA:Anaconda bootloader setup test case|Anaconda bootloader setup]]
 
== Basic Functionality ==
# [[QA:X basic display test case|X startup/basic display configuration]]
# [[QA:X basic input handling test case|X basic input handling]]
# [[QA:Network basic test case|Basic network connectivity]]
# [[QA:Yum simple update test case|yum update functionality]]
 
|environment=
* The tests will be run on a host which has the Rawhide tree accessible by HTTP or FTP.
** The tests will download a lot of data, so ideally the host should have a very fast network link to the HTTP/FTP mirror.
* The test host will be no older than RHEL5/CentOS5 or currently supported Fedora releases.
* The test host will have at least 1GB free disk space:
** critical path packages (~300MB)
** their associated metadata (~200MB)
** all of the boot images (~300MB)
** plus some extra scratch space for rebuilding images etc.
 
|resposibility=
Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.
 
|schedule=
Ideally this test plan should be run for ''every new Rawhide tree'' - i.e. daily.
 
|risks=
 
* Testing kernel boot and X startup is currently tricky to automate.
* [[Critical_Path_Packages_Proposal]] - while exact list of packages may change, it is possible that no agreement on the exact list can be made.  In this event, a temporary subset of critical packages will be used
 
|reviewedby=
<!-- Add a line like this:
* ~~~~
This will automatically be expanded to your username and timestamp. Preview to check it. -->
* [[User:Jlaska|jlaska]] 11:13, 19 June 2009 (UTC) <!-- jlaska -->
* [[User:Johannbg|viking-ice]] 16:14, 24 June 2009 (UTC)
 
|references=
* [[JohnPoelstra/ImproveRawhideF10#Defining_GOOD]] - A previous discussion of Rawhide requirements
* [[Critical Path Packages Proposal]] - Proposal to define the most critical packages - required for some repo sanity test cases
* [[Israwhidebroken.com Proposal]] - This test plan is a central part of this proposal.
<references/>
}}

Latest revision as of 17:19, 9 March 2011

This page is a draft only
It is still under construction and content may change. Do not rely on the information on this page.

This page will outline criteria for success for a Fedora AMI release

Release Criteria

In addition to the existing Fedora Release Criteria, the following criteria apply to EC2 Fedora images. A good AMI must...

  1. allow non-root user login
  2. have SELinux enabled and enforcing by default
  3. not contain any package dependency problems
  4. not include any packages built outside of Fedora infrastructure

How to Test

To create an EC2 image, one needs...

  1. an amazon account
  2. the python-boto libraries installed
  3. a python script to tell Amazon to build an instance
  4. the ability to SSH into a machine to run tests, etc.

Open Questions

  1. Who is responsible for creating and building EC2 images?

Additional Reading