From Fedora Project Wiki
m (Moved 'see also' content under 'references') |
(Add input handling (adamw), group tests/criteria into 3 categories (ldimaggi)) |
||
| Line 31: | Line 31: | ||
Rawhide will be considered Good if all of the following conditions are met: | Rawhide will be considered Good if all of the following conditions are met: | ||
=== Repo Sanity === | |||
* Contains valid yum metadata (repodata) | * Contains valid yum metadata (repodata) | ||
* Contains key packages (kernel, glibc, coreutils) | * Contains key packages (kernel, glibc, coreutils) | ||
* No unresolved dependencies in critical packages | * No unresolved dependencies in critical packages | ||
* comps.xml exists and is valid | * comps.xml exists and is valid | ||
=== Installability === | |||
* installer images (kernel, initrd, install.img, and boot.iso) exist | * installer images (kernel, initrd, install.img, and boot.iso) exist | ||
* kernel boots on ''most'' machines of the primary architectures | * kernel boots on ''most'' machines of the primary architectures | ||
* initrd is able to find stage2 (install.img) by at least one method (network, local CD) | * initrd is able to find stage2 (install.img) by at least one method (network, local CD) | ||
* stage2 is able to detect the presence of disks attached to most common controllers | * stage2 is able to detect the presence of disks attached to most common controllers | ||
=== Basic Functionality === | |||
* Kernel/X is able to set up common display configurations for at least 2 out of 3 of the most common video drivers (intel, nouveau, radeon) | * Kernel/X is able to set up common display configurations for at least 2 out of 3 of the most common video drivers (intel, nouveau, radeon) | ||
* Kernel/X properly handle input from standard USB keyboard/mouse | |||
= Test Deliverables = | = Test Deliverables = | ||
| Line 52: | Line 58: | ||
''TODO: create these test cases'' | ''TODO: create these test cases'' | ||
=== Repo Sanity === | |||
# Repodata validity | # Repodata validity | ||
# comps.xml validity | # comps.xml validity | ||
# Core package existence | # Core package existence | ||
# Core package dependency closure | # Core package dependency closure | ||
=== Installability === | |||
# Installer image existence | # Installer image existence | ||
# Kernel boot | # Kernel boot | ||
# Anaconda loader fetching stage2 | # Anaconda loader fetching stage2 | ||
# Anaconda stage2 disk probe | # Anaconda stage2 disk probe | ||
# X startup | |||
=== Basic Functionality === | |||
# X startup/basic display configuration | |||
# X basic input handling | |||
= Test Environment = | = Test Environment = | ||
Revision as of 15:13, 19 June 2009
Revision history
First draft: WillWoods 20:54, 17 June 2009 (UTC)
Introduction
This test plan documents the process used to check the basic requirements for a Rawhide tree to be acceptable for further testing. It aims to check whether Rawhide is installable, usable as a package repo for updating, and whether critical packages are present and functional.
In short, this is how we decide if Rawhide is broken or not.
Test Strategy
There are three main components here: Repo sanity, Installability, and Basic Functionality. These three categories can be tested mostly independent of one another.
Scope
This plan seeks to answer three basic questions:
- Can current Rawhide users update their systems using this repo?
- Can this Rawhide tree be installed?
- Does the basic system (the "critical path") work as expected for simple testing?
It is not intended to be an exhaustive test of any part of the system.
Test Pass/Fail Criteria
Rawhide will be considered Good if all of the following conditions are met:
Repo Sanity
- Contains valid yum metadata (repodata)
- Contains key packages (kernel, glibc, coreutils)
- No unresolved dependencies in critical packages
- comps.xml exists and is valid
Installability
- installer images (kernel, initrd, install.img, and boot.iso) exist
- kernel boots on most machines of the primary architectures
- initrd is able to find stage2 (install.img) by at least one method (network, local CD)
- stage2 is able to detect the presence of disks attached to most common controllers
Basic Functionality
- Kernel/X is able to set up common display configurations for at least 2 out of 3 of the most common video drivers (intel, nouveau, radeon)
- Kernel/X properly handle input from standard USB keyboard/mouse
Test Deliverables
This test plan should produce:
- A summary report on whether Rawhide is broken or not
- Bug reports for broken dependencies / missing files / etc.
- A list of test cases used to verify the expected results. (see below)
Test Cases
TODO: create these test cases
Repo Sanity
- Repodata validity
- comps.xml validity
- Core package existence
- Core package dependency closure
Installability
- Installer image existence
- Kernel boot
- Anaconda loader fetching stage2
- Anaconda stage2 disk probe
Basic Functionality
- X startup/basic display configuration
- X basic input handling
Test Environment
- Tests will run on a host which has the Rawhide tree accessible on the local filesystem. It may be network-mounted.
- The test host will be no older than RHEL5/CentOS5 or currently supported Fedora releases.
Responsibilities
Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.
Schedule
Ideally this test plan should be run for every new Rawhide tree - i.e. daily.
Risks
- Testing kernel boot and X startup is currently hard to automate.
- We don't have a good definition for "Core packages" - personal opinions may vary wildly.
Reviewers
- jlaska 11:13, 19 June 2009 (UTC)
References
- JohnPoelstra/ImproveRawhideF10#Defining_GOOD - A previous discussion of Rawhide requirements