From Fedora Project Wiki
(obsolete (old release))
 
(185 intermediate revisions by 5 users not shown)
Line 1: Line 1:
= Fedora 12 Installation Test Plan =
= Introduction =
This document describes the tests that will be created and used to verify the functions/components of {{FedoraVersion|long|12}}.


== Revision history ==
The goals of this plan are to:
{| border="1"
* Organize the test effort
|- style="color: white; background-color: #3074c2; font-weight: bold"
* Communicate the planned tests to all relevant stake-holders for their input and approval
| Date || Revision || Comment
* Serve as a base for the test planning for future Fedora 12 releases
|-
= Test Strategy =
| {{date|2009-06-04}} || 0.1 || Initial version
Instead of outlining all possible installation inputs and outputs, this test plan will focus on defining inputs and outputs at different ''stages'' in anaconda.  This will also allow different tests to be performed independently during a single installation.  For example, one may execute a kickstart delivery via HTTP, raid0 partitioning using 3 physical disks, and a minimal package installation on a para-virtualized xen guest all in single installation.  Scenarios where the ''stages'' are dependent will be indicated as such in the test case.
|}


== Introduction ==
= New features of {{FedoraVersion|long|12}} =
This document describes the tests that will be created and used to verify the installation of Fedora 12.


The goals of this plan are to:
As with Fedora 11, Fedora 12 will bring us some new features.  The following list outlines the larger changes that affect installation.  Test plans for these features will be designed/developed on each feature page.
* ''Organize'' the test effort
 
* ''Communicate'' the strategy, scope and priorities of the planned tests to all relevant stake-holders for their input and approval
* [[Anaconda/Features/MDRaid|Use MDRaid instead of dmraid]]
* Serve as a base for the test planning for future Fedora releases
* [[Anaconda/Features/StorageFiltering|Select devices better, plus storage cleanup]]
* [[Anaconda/Features/UICleanup|Storage UI Cleanup]]


== Test Strategy ==
Additional features outside the scope of testing can be found at:
* [[Anaconda/Features|List of anaconda features]]
* [[Releases/12/FeatureList|Fedora 12 Accepted Features]]


Instead of outlining all possible installation inputs and outputs, this test plan will focus on defining inputs and outputs at different ''stages'' in anaconda.  This will also allow different tests to be performed independently during a single installation.  For example, one may execute a kickstart delivery via HTTP, raid0 partitioning using 3 physical disks, and a minimal package installation on a para-virtualized xen guest all in single installation. Scenarios where the ''stages'' are dependent will be indicated as such in the test case.
= Schedule/Milestones =
* The {{FedoraVersion|long|12}} release schedule is available at [[Releases/12/Schedule]]
* Each major milestone(Beta,Preview,etc..) will demand a full regression run


Where possible, [https://fedorahosted.org/snake SNAKE] will be used to automate and aid in reproducibility.
= Test Priority =


== Test Priority ==
This test plan will use a 3 '''tier''' classification for test execution priority.
This test plan will use a 3 '''tier''' classification for test execution priority.


Tier1 is intended to verify that installation is possible on common hardware using common use cases.  Verification includes:
== [[QA/TestPlans/Fedora12InstallTier1|Tier#1 test cases]] ==
 
Tier1 is intended to verify that installation is possible on common hardware using common use cases.   
 
Verification includes:
* Common boot media
* Common boot media
* Common Installation source
* Common Installation source
* Installation using defaults installation options
* Installation using defaults installation options
* Default Partitioning
* Default Partitioning
== [[QA/TestPlans/Fedora12InstallTier2|Tier#2 test cases]] ==


Tier2 takes a step further to include more use cases.  Tier2 verification consists of:
Tier2 takes a step further to include more use cases.  Tier2 verification consists of:
Line 38: Line 47:
* Some architecture specific verification
* Some architecture specific verification


Lastly, Tier3 captures the remaining identified use cases:
== [[QA/TestPlans/Fedora12InstallTier3|Tier#3 test cases]] ==
 
Tier3 captures the remaining identified use cases:
* More exhaustive partitioning schemes
* More exhaustive partitioning schemes
* More complex networking scenarios
* More complex networking scenarios
Line 46: Line 57:
* Upgrade testing
* Upgrade testing


== Scope ==
= Test Pass/Fail Criteria =
Testing will include:
* Various methods of booting the installation program
* Manual and kickstart execution of the installation program
* System setup performed by the installation program (networking, modprobe.conf, bootloader, runlevel)
* Booting the installed system
 
Items outside the scope of this test plan include:
* Functional verification of software installed on the system
* Installation from media not generated by fedora release engineering
 
== Test Pass/Fail Criteria ==


{{admon/important|FIXME|Need to add a link to the FUDCon10 rawhide acceptance criteria}}
The milestone release of Fedora 12 should conform these criteria:


Entrance criteria
== Entrance criteria ==
* Trees must be generated using release engineering tools (not hand crafted)
* Trees must be generated using release engineering tools (not hand crafted)
* There must be no unresolved dependencies for packages included in the installation tree
* There must be no unresolved dependencies for packages included in the installation tree
Line 67: Line 67:
* Any changes in composition of the installation tree are explainable by way of bugzilla
* Any changes in composition of the installation tree are explainable by way of bugzilla


Alpha criteria
== Alpha criteria ==
* Entrance criteria have been met
* Entrance criteria have been met
* All tier#1 tests have been executed
* All test tiers have have been executed
* All tier#1 tests pass


Beta criteria
== Beta criteria ==
* Alpha criteria have been met
* Alpha criteria have been met
* All tier#1 tests pass
* All tier#2 tests pass
* All tier#2 tests have been executed


GA criteria
== GA criteria ==
* Beta criteria have been met
* Beta criteria have been met
* All test tiers must pass
* All test tiers must pass
* Any open defects have been documented as release notes
* Any open defects have been documented as release notes


== Test Deliverables ==
= Scope and Approach =
* This test plan
* A test summary document for each major milestone
* A list of defects filed
* Any test scripts used for automation or verification


== Test Cases (Functional) ==
Testing will include:


{{admon/important|FIXME|Need confirmation on the following list of features for F10 anaconda}}
* Manually execute test cases using a rawhide boot or live image media
* Where possible, the use of virtualization is encouraged to allow for greater test coverage and improve failure diagnosis
* Automated execution of tests is encouraged using kickstarts designed to execute specific test cases.  At this time, kickstarts are not provided for every test case.


The following list of features was obtained from [[Anaconda/Features]].  Test plans for these features will be designed/developed on each feature page.
Items outside the scope of this test plan include:
* Functional verification of software installed on the system
* Installation from media not generated by fedora release engineering


{| border="1"
= Test Deliverables =
|- style="color: white; background-color: #3074c2; font-weight: bold"
|Feature||Owner||Target completion
|-
| [[Anaconda/Features/Encrypted Boot IPA key Management|  Encrypted Boot IPA key Management]]  || Dave Lehman / Miloslav Trmac   || Fedora12
|}


== Test Cases (Non-Functional) ==
* [[QA:Fedora_12_Install_Test_Plan|This test plan]]
* [[:Category:Fedora_12_Test_Results|A test summary document for each major milestone]]
* A list of defects filed
* Any test scripts used for automation or verification


{| style="background-color: #def3fe; border: 1px solid #c5d7e0; color: black; padding: 5px; margin-bottom: 5px; min-height: 35px; padding-left: 45px;"
= Testing Tasks =
! colspan="3" align="left" | Image Sanity
|-
! colspan="3" align="left" style="font-weight: normal;" | Tests to check sums of Media
|-
| [[QA:Testcase_Mediakit_ISO_Size  ]] || [[QA:Testcase_Mediakit_ISO_Checksums  ]] || [[QA:Testcase_Mediakit_Repoclosure ]]
|-
| [[QA:Testcase_Mediakit_FileConflicts ]] || ||
|-


! colspan="3" align="left" | Boot Methods
Testing will execute test cases to verify installation of Fedora 12 on different hardware platforms and gather installation test feedback.
|-
* [[QA:Fedora_12_Install_Results_Template#Test_Areas|Installation Test Cases]]
! colspan="3" align="left" style="font-weight: normal;" | Tested designed to validate the bootable media
* [[QA:Create_Install_Test_Result_Page|Instructions for adding test result page]]
|-
| [[QA/TestCases/BootMethodsBootIso]] || [[QA/TestCases/BootMethodsCdrom]] || [[QA/TestCases/BootMethodsDvd]]
|-
| [[QQA:Testcase_efidisk.img]] || [[QA/TestCases/BootMethodsPxeboot]] || [[QA/TestCases/BootMethodsNetboot]]
|-
| [[QA/TestCases/BootMethodsXenParaVirt]] || [[QA/TestCases/BootMethodsKVM]] ||
|-


! colspan="3" align="left" | Installation Source
= Test Environment/Configs =
|-
! colspan="3" align="left" style="font-weight: normal;" | The media booted and the installation source used aren't always the same.  These tests verify installation using the described source.
|-
| [[QA/TestCases/InstallSourceHttp]] || [[QA/TestCases/InstallSourceNfs]] || [[QA/TestCases/InstallSourceFtpAnonymous]]
|-
| [[QA/TestCases/InstallSourceFtpNonAnonymous]] || [[QA/TestCases/InstallSourceCdrom]] || [[QA/TestCases/InstallSourceDvd]]
|-
| [[QA/TestCases/InstallSourceHardDrive]] || [[QA/TestCases/InstallSourceNfsIso]] || [[QA:TestCases/Install Source Live Image]]
|-


! colspan="3" align="left" | Kickstart Delivery
For {{FedoraVersion|long|12}},test cases will be executed on these hardware platforms:
|-
! colspan="3" align="left" style="font-weight: normal;" | Tests to validate acquiring a kickstart script through supported methods.
|-
| [[QA/TestCases/KickstartKsFilePathKsCfg ]] || [[QA/TestCases/KickstartKsHdDevicePathKsCfg ]] || [[QA/TestCases/KickstartKsHttpServerKsCfg ]]
|-
| [[QA/TestCases/KickstartKsNfsServerPathKsCfg ]] || ||
|-
 
! colspan="3" align="left" | Package Sets
|-
! colspan="3" align="left" style="font-weight: normal;" | Designed to exercise the most common package dependency and conflict pathways.
|-
| [[QA/TestCases/PackageSetsDefaultPackageInstall]] ||[[QA/TestCases/PackageSetsMinimalPackageInstall]]||
|-
 
! colspan="3" align="left" | Partitioning
|-
! colspan="3" align="left" style="font-weight: normal;" | The more common partitioning scenarios.  These cases ensure that anaconda (and friends) prepare the disk for post-install booting as directed.
|-
| [[QA:Testcase_Anaconda_autopart_install ]] || [[QA:Testcase_Anaconda_autopart_(encrypted)_install ]] || [[QA:Testcase_Anaconda_autopart_(shrink)_install ]]
|-
| [[QA:Testcase_Anaconda_autopart_(use_free_space)_install]] || [[QA/TestCases/PartitioningExt4OnNativeDevice ]] || [[QA/TestCases/PartitioningExt3OnNativeDevice ]]
|-
| [[QA/TestCases/PartitioningExt2OnNativeDevice]] || [[QA/TestCases/PartitioningRootfsOnLvmDevice ]] || [[QA/TestCases/PartitioningRootfsOnRaid1]]
|-
| [[QA/TestCases/PartitioningNoSwap]] || [[QA/TestCases/PartitioningRaid0OnLvmDevice]] || [[QA/TestCases/PartitioningSwapOnLvmDevice]]
|-
| [[QA/TestCases/PartitioningUninitializedDisks]] || [[QA/TestCases/PartitioningUsrOnRaid0 ]] || [[QA/TestCases/PartitioningUsrOnRaid5]]
|-
| [[QA/TestCases/PartitioningUsrOnRaid6]] || [[QA/TestCases/PartitioningRootfsOnDmraidDevice ]] || [[QA/TestCases/PartitioningPreExistingLvm2Lvm2]]
|-
| [[QA/TestCases/PartitioningPreExistingRaidRaid ]]  || ||
|-
 
! colspan="3" align="left" | Recovery
|-
! colspan="3" align="left" style="font-weight: normal;" | When ''stuff'' goes wrong ... we need to be able to handle it reasonably well.
|-
| [[QA:Testcase_Anaconda_rescue_mode ]] || [[QA:Testcase_Anaconda_updates.img_via_URL]] || [[QA:Testcase_Anaconda_updates.img_via_installation_source ]]
|-
| [[QA:Testcase_Anaconda_updates.img_via_local_media]] || [[QA:Testcases Anaconda save traceback to remote system ]] || [[QA:Testcases Anaconda save traceback to bugzilla ]]
|-
| [[QA:Testcases Anaconda save traceback to disk]] || [[QA:Testcases Anaconda traceback debug mode]] ||
|-
 
! colspan="3" align="left" | Storage Devices
|-
! colspan="3" align="left" style="font-weight: normal;" | Are we probing and booting post-install properly in the following scenarios?
|-
| [[QA/TestCases/StorageDeviceSata ]] || [[QA/TestCases/StorageDeviceScsi ]] || [[QA:Testcase_Install_to_Pata_Device ]]
|-
| [[QA:Testcase Anaconda iSCSI no authentication ]] || [[QA:Testcase_Anaconda_partitioning_dmraid_rootfs]] ||
|-
 
! colspan="3" align="left" | User Interface
|-
! colspan="3" align="left" style="font-weight: normal;" | Anaconda provides several user-interfaces for installation, the following cases are designed to ensure the desired interface operates as expected
|-
| [[QA/TestCases/UserInterfaceGraphical ]] || [[QA/TestCases/UserInterfaceText ]] || [[QA/TestCases/UserInterfaceVnc ]]
|-
| [[QA/TestCases/UserInterfaceCmdline ]] || [[QA/TestCases/UserInterfaceTelnet ]] ||
|-
 
! colspan="3" align="left" | Upgrade system
|-
! colspan="3" align="left" style="font-weight: normal;" | Tests to validate system upgrade
|-
| [[QA:Testcase_Anaconda_Upgrade_New_Bootloader ]] || [[QA:Testcase_Anaconda_Upgrade_Skip_Bootloader]] || [[QA:Testcase_Anaconda_Upgrade_Update_Bootloader ]]
|-
| [[QA:Testcase_Anaconda_Upgrade_Encrypted_Root ]] || [[QA:Testcase_Preupgrade  ]] ||[[QA:Testcase_Preupgrade_from_older_release ]]
|-
|}
 
== Test Environment/Configs ==
 
Hardware
* i386
* i386
* ppc
* ppc
* x86_64
* x86_64


== Responsibilities ==
= Responsibilities =
* who's doing what
 
== Schedule/Milestones ==
* The Fedora 10 schedule is available at [[Releases/10/Schedule]]
* Each major milestone will demand a full regression run (Alpha, Beta, PreviewRelease)


== Risks and Contingencies ==
Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.


* If a particular build is uninstallable, a ''updates.img'' may be used to work around any test blockers.  For information on building and using an ''updates.img'' please refer to [[Anaconda/Updates]].
= Risks and Contingencies =


{{admon/note|An ''updates.img'' can't always save the day|Please note, any issues affecting stage#1 loader (networking segfaults) or kernel boot issues may not be resolvable by using an ''updates.img''. For these issues, your defect will need to be prioritized as a test blocker so that it can be addressed in a subsequent build.}}
* If no physical media is available for testing, it is acceptable to use a <code>boot.iso</code> or PXE boot images from a current Rawhide release. For more information, see [[Releases/Rawhide#Direct_Rawhide_install]].


* If a build is uninstallable due to a bad kernel or driver, you may be able to work around the problem.  Anaconda provides several scripts for taking a newer kernel, and using that during installation.  For details, please refer to [[http://git.fedorahosted.org/git/?p=anaconda.git;a=blob;f=scripts/upd-kernel;h=30871135eff0b4ad87bad8e0c84498a46bbf05d0;hb=HEAD| scripts/upd-kernel]]
= Reporting Bugs and Debugging Problems =


== Approvals ==
If defects/problems are encountered, please go ahead and file the bugs following the guide below:
* [[Bugs_and_feature_requests|Bug Reporting Process]]
* [[How to debug installation problems]]


{| border="1"
= Reviewers =
|- style="color: white; background-color: #3074c2; font-weight: bold"
* [[User:Jlaska|jlaska]] 19:08, 3 August 2009 (UTC)
| Date || Approver || Comment
|-
| {{date|2009-06-04}} || [[user:jlaska]] || I approve this message
|}


== References ==
= References =
* Red Hat Enterprise Linux Installation Guide - http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/Installation_Guide/
* [http://docs.fedoraproject.org/install-guide/f11/ {{FedoraVersion|long|11}} Installation Guide]
* Previous test plans available at [[:Category:Test_Plans]]
* [https://fedoraproject.org/wiki/Category:Installer_Test_Cases Fedora Install test cases]
* Anaconda Documentation
* Anaconda Documentation
** Command-line options - http://fedoraproject.org/wiki/Anaconda/Options
** [[Anaconda/Options|Supported command-line options]]
** Kickstart options - http://fedoraproject.org/wiki/Anaconda/Kickstart
** [[Anaconda/Kickstart|Kickstart options]]
* Debugging Problems
** [[Anaconda/Updates|How to Create and Use an updates.img]]
** Anaconda Bug Reporting Guide - http://fedoraproject.org/wiki/Anaconda/BugReporting
** [[Anaconda/SourceOverview|Source Code Overview]]
** Source Code Overview - http://fedoraproject.org/wiki/Anaconda/SourceOverview
** Anaconda updates.img - http://fedoraproject.org/wiki/Anaconda/Updates
** Anaconda Stage1 Guide - http://fedoraproject.org/wiki/Anaconda/Stage1DevelopmentGuide
** Anaconda Stage2 Guide - http://fedoraproject.org/wiki/Anaconda/Stage2DevelopmentGuide


[[Category:Test Plans]]
[[Category:Obsolete_Test Plans]]

Latest revision as of 08:21, 6 August 2014

Introduction

This document describes the tests that will be created and used to verify the functions/components of Fedora 12.

The goals of this plan are to:

  • Organize the test effort
  • Communicate the planned tests to all relevant stake-holders for their input and approval
  • Serve as a base for the test planning for future Fedora 12 releases

Test Strategy

Instead of outlining all possible installation inputs and outputs, this test plan will focus on defining inputs and outputs at different stages in anaconda. This will also allow different tests to be performed independently during a single installation. For example, one may execute a kickstart delivery via HTTP, raid0 partitioning using 3 physical disks, and a minimal package installation on a para-virtualized xen guest all in single installation. Scenarios where the stages are dependent will be indicated as such in the test case.

New features of Fedora 12

As with Fedora 11, Fedora 12 will bring us some new features. The following list outlines the larger changes that affect installation. Test plans for these features will be designed/developed on each feature page.

Additional features outside the scope of testing can be found at:

Schedule/Milestones

  • The Fedora 12 release schedule is available at Releases/12/Schedule
  • Each major milestone(Beta,Preview,etc..) will demand a full regression run

Test Priority

This test plan will use a 3 tier classification for test execution priority.

Tier#1 test cases

Tier1 is intended to verify that installation is possible on common hardware using common use cases.

Verification includes:

  • Common boot media
  • Common Installation source
  • Installation using defaults installation options
  • Default Partitioning

Tier#2 test cases

Tier2 takes a step further to include more use cases. Tier2 verification consists of:

  • All boot media
  • All installation sources
  • All kickstart delivery methods
  • Some architecture specific verification

Tier#3 test cases

Tier3 captures the remaining identified use cases:

  • More exhaustive partitioning schemes
  • More complex networking scenarios
  • More architecture specific verification
  • Network device
  • Storage device
  • Upgrade testing

Test Pass/Fail Criteria

The milestone release of Fedora 12 should conform these criteria:

Entrance criteria

  • Trees must be generated using release engineering tools (not hand crafted)
  • There must be no unresolved dependencies for packages included in the installation tree
  • There must be no dependency conflicts for packages included in the installation tree
  • Any changes in composition of the installation tree are explainable by way of bugzilla

Alpha criteria

  • Entrance criteria have been met
  • All test tiers have have been executed
  • All tier#1 tests pass

Beta criteria

  • Alpha criteria have been met
  • All tier#2 tests pass

GA criteria

  • Beta criteria have been met
  • All test tiers must pass
  • Any open defects have been documented as release notes

Scope and Approach

Testing will include:

  • Manually execute test cases using a rawhide boot or live image media
  • Where possible, the use of virtualization is encouraged to allow for greater test coverage and improve failure diagnosis
  • Automated execution of tests is encouraged using kickstarts designed to execute specific test cases. At this time, kickstarts are not provided for every test case.

Items outside the scope of this test plan include:

  • Functional verification of software installed on the system
  • Installation from media not generated by fedora release engineering

Test Deliverables

Testing Tasks

Testing will execute test cases to verify installation of Fedora 12 on different hardware platforms and gather installation test feedback.

Test Environment/Configs

For Fedora 12,test cases will be executed on these hardware platforms:

  • i386
  • ppc
  • x86_64

Responsibilities

Fedora QA team members are responsible for executing this test plan. Contributions from Rawhide testers and other interested parties are encouraged.

Risks and Contingencies

  • If no physical media is available for testing, it is acceptable to use a boot.iso or PXE boot images from a current Rawhide release. For more information, see Releases/Rawhide#Direct_Rawhide_install.

Reporting Bugs and Debugging Problems

If defects/problems are encountered, please go ahead and file the bugs following the guide below:

Reviewers

  • jlaska 19:08, 3 August 2009 (UTC)

References

Retrieved from "https://fedoraproject.org/w/index.php?title=QA:Fedora_12_Install_Test_Plan&oldid=383536"