From Fedora Project Wiki
 
(13 intermediate revisions by the same user not shown)
Line 3: Line 3:
= History =  
= History =  


Created by [[User:Roshi | Roshi]] on 22 April 2014
* Created by [[User:Roshi | Roshi]] on 23 April 2014
* Edited by [[User:Roshi | Roshi]] on 02 October 2014


= Introduction =
= Introduction =
Line 16: Line 17:
Testing and Release Validation for the Cloud product will follow the same pattern Fedora 20 followed. There will be three milestones - each milestone will have a related set of Release Criteria and Validation Matrix for the release.
Testing and Release Validation for the Cloud product will follow the same pattern Fedora 20 followed. There will be three milestones - each milestone will have a related set of Release Criteria and Validation Matrix for the release.


=== Milestones ===
= Schedule/Milestones =


* Alpha
* Alpha
* Beta
* Beta
* Final
* Final
= Schedule/Milestones =


= Test Priority =
= Test Priority =
Line 36: Line 35:
| Final priority tests capture all remaining use cases and functionality checks.  These tests also attempt to validate [[User:Roshi/QA/Cloud_Docs/Cloud_Final_Release_Criteria|Final Release Requirements]].  
| Final priority tests capture all remaining use cases and functionality checks.  These tests also attempt to validate [[User:Roshi/QA/Cloud_Docs/Cloud_Final_Release_Criteria|Final Release Requirements]].  
|-
|-
| Verification consists of:
- Does the image boot on supported platforms?<br />
- Does the image properly utilize basic metadata (ssh-key, etc)?<br />
- Can you ssh into the booted image?
| Verification consists of:
- Does yum update the image properly?<br />
- Can the booted image be rebooted?
| Verification consists of:
| Verification consists of:
| Verification consists of:
- Can an in use cloud image be upgraded to a Fedora Server role?
| Verification consists of:
|}
|}


= Test Pass/Fail Criteria =
= Test Pass/Fail Criteria =


The milestone release of Fedora 17 should conform these criteria:
The milestone release of Fedora Cloud should conform these criteria:
 
== Entrance criteria ==
* Trees must be generated using release engineering tools (not hand crafted)
* There must be no unresolved dependencies for packages included in the installation tree
* There must be no dependency conflicts for packages included in the installation tree
* Any changes in composition of the installation tree are explainable by way of bugzilla


== Milestone specific criteria ==
* [[User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Release_Criteria | Cloud Alpha Release Criteria]]
* [[Fedora_17_Alpha_Release_Criteria]]
* [[User:Roshi/QA/Cloud_Docs/Cloud_Beta_Release_Criteria | Cloud Beta Release Criteria]]
* [[Fedora_17_Beta_Release_Criteria]]
* [[User:Roshi/QA/Cloud_Docs/Cloud_Final_Release_Criteria | Cloud Final Release Criteria]]
* [[Fedora_17_Final_Release_Criteria]]


= Scope and Approach =
= Scope and Approach =
Line 69: Line 67:
Testing will include:
Testing will include:


* Manually executed test cases using DVD, {{filename|boot.iso}}, PXE or live image media
* Manually executed test cases  
* Automatically executed test cases via the [[AutoQA]] system. For more information about automatic testing, please see [[Is_anaconda_broken_proposal|Is Anaconda Broken Proposal]] and the [https://fedorahosted.org/autoqa/milestone/Automate%20installation%20test%20plan install automation roadmap].
* Automatically executed test cases (exact testcases to come at a later date).
<!--Items outside the scope of this test plan include:
* Functional verification of software installed on the system
* Installation from media not generated by fedora release engineering
* [[Architectures#Secondary_Architectures|Secondary hardware platforms]]-->


= Test Deliverables =
= Test Deliverables =


* This test plan
* This test plan
* Test summary documents for each major milestone of F16: [[:Category:Fedora_17_Test_Results]]
* Test summary documents for each major milestone
* A list of defects filed
* A list of defects filed
* Any test scripts used for automation or verification
* Any test scripts used for automation or verification
Line 85: Line 79:
= Testing Tasks =
= Testing Tasks =


Testing will execute test cases to verify installation of Fedora 17 on different hardware platforms and gather installation test feedback.
Testing will include test cases to ensure the desired functionality of the cloud image works as intended.
* [[QA:Fedora_17_Install_Results_Template#Test_Matrix|Manual installation Test Cases]]
* Link to Test Matrix (to be written)
* [[Auto-installation Test Cases]]
* Automated testing
* [[QA:Create_Install_Test_Result_Page|Instructions for adding test result page]]
* Image works with major cloud providers


= Test Environment/Configs =
= Test Environment/Configs =


For {{FedoraVersion|long|17}}, test cases will be executed on the [[Architectures#Primary_Architectures|''primary'' supported hardware platforms]].  This includes:
Test cases will be executed on the [[Architectures#Primary_Architectures|''primary'' supported hardware platforms]].  This includes:
* i386
* i386
* x86_64
* x86_64
Line 98: Line 92:
= Responsibilities =
= Responsibilities =


Fedora QA team members are responsible for executing this test plan. Contributions from Branched testers and other interested parties are encouraged.
Cloud WG team members are responsible for executing this test plan. Contributions from QA testers and other interested parties are encouraged.


= Risks and Contingencies =
= Risks and Contingencies =


If ''new'' physical media are provided for an already ''inprogress'' test run, a new test run '''must''' be initiated.  Test results from the previous run may be carried forward to the new test run if they are not affected by the changes introduced by the new physical media.
If ''new'' images are provided for an already ''inprogress'' test run, a new test run '''must''' be initiated.  Test results from the previous run may be carried forward to the new test run if they are not affected by the changes introduced by the freshly generated image.


= Reporting Bugs and Debugging Problems =
= Reporting Bugs and Debugging Problems =
Line 108: Line 102:
If defects/problems are encountered, please go ahead and file the bugs following the guide below:
If defects/problems are encountered, please go ahead and file the bugs following the guide below:
* [[Bugs_and_feature_requests|Bug Reporting Process]]
* [[Bugs_and_feature_requests|Bug Reporting Process]]
* [[How to debug installation problems]]
* [http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/Cloud_Guide/index.html Cloud Guide]


= Reviewers =
= Reviewers =
Line 114: Line 108:


= References =
= References =
* [[QA:Installation_validation_testing|Installation Validation Testing Introduction Page]]
* [[Fedora Release Criteria]]
* [http://docs.fedoraproject.org/install-guide/f{{FedoraVersion||16}}/en-US/html/ {{FedoraVersion|long|16}} Installation Guide]
* [[User:Roshi/QA/Cloud_Docs/Cloud_Alpha_Release_Criteria | Cloud Alpha Release Criteria]]
* [https://fedoraproject.org/wiki/Category:Installer_Test_Cases Fedora Install test cases]
* [[User:Roshi/QA/Cloud_Docs/Cloud_Beta_Release_Criteria | Cloud Beta Release Criteria]]
* Previous test plans available at [[:Category:Test_Plans]]
* [[User:Roshi/QA/Cloud_Docs/Cloud_Final_Release_Criteria | Cloud Final Release Criteria]]
* Anaconda Documentation
* [http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/Cloud_Guide/index.html Cloud Guide]
** [[Anaconda/Options|Supported command-line options]]
* [[Getting_started_with_OpenStack_Nova|Getting Started with OpenStack Nova]]
** [[Anaconda/Kickstart|Kickstart options]]
* [[User:Gholms/EC2_Primer | EC2 Primer]]
** [[Anaconda/Updates|How to Create and Use an updates.img]]
** [[Anaconda/SourceOverview|Source Code Overview]]


[[Category:Test Plans]]
[[Category:Test Plans]]
[[Category:Install_Test_Plans]]
[[Category:Cloud_Test_Plans]]

Latest revision as of 01:00, 3 October 2014

History

  • Created by Roshi on 23 April 2014
  • Edited by Roshi on 02 October 2014

Introduction

With the advent of different Fedora Products for Fedora 21 there is a need for a specific test plan for each product. Historically, Release Validation and all Testing of Fedora releases was handled by QA - but now testing and release validation will be largely handled by the specific Working Group.

The goals of this plan are to:

  • Organize the test effort
  • Communicate the planned tests to all relevant stake-holders for their input and approval
  • Serve as a base for the test planning for future Fedora Cloud product releases

Test Strategy

Testing and Release Validation for the Cloud product will follow the same pattern Fedora 20 followed. There will be three milestones - each milestone will have a related set of Release Criteria and Validation Matrix for the release.

Schedule/Milestones

  • Alpha
  • Beta
  • Final

Test Priority

This test plan prioritizes tests according to the major release milestones for Fedora 21, including the Alpha, Beta and Final release milestones. All test cases are intended for execution at every milestone. However, priority should be given to tests specific to the milestone under test.

Alpha test cases Beta test cases Final test cases
Alpha priority tests are intended to verify that booting the image is possible on common cloud providers (EC2, Openshift). These tests also attempt to validate Alpha Release Requirements. Beta priority tests take a step further to include additional use cases. These tests also attempt to validate Beta Release Requirements. Final priority tests capture all remaining use cases and functionality checks. These tests also attempt to validate Final Release Requirements.
Verification consists of:

- Does the image boot on supported platforms?
- Does the image properly utilize basic metadata (ssh-key, etc)?
- Can you ssh into the booted image?

Verification consists of:

- Does yum update the image properly?
- Can the booted image be rebooted?

Verification consists of:

- Can an in use cloud image be upgraded to a Fedora Server role?

Test Pass/Fail Criteria

The milestone release of Fedora Cloud should conform these criteria:

Scope and Approach

For the cloud image, the following areas will be tested:

  • Boot process
  • Initialization
  • Post-Boot actions
  • Virtualization

In addition to those aspects of the image being tested, any bugs marked as a blocker for the current milestone must be addressed and tested.

Testing will include:

  • Manually executed test cases
  • Automatically executed test cases (exact testcases to come at a later date).

Test Deliverables

  • This test plan
  • Test summary documents for each major milestone
  • A list of defects filed
  • Any test scripts used for automation or verification

Testing Tasks

Testing will include test cases to ensure the desired functionality of the cloud image works as intended.

  • Link to Test Matrix (to be written)
  • Automated testing
  • Image works with major cloud providers

Test Environment/Configs

Test cases will be executed on the primary supported hardware platforms. This includes:

  • i386
  • x86_64

Responsibilities

Cloud WG team members are responsible for executing this test plan. Contributions from QA testers and other interested parties are encouraged.

Risks and Contingencies

If new images are provided for an already inprogress test run, a new test run must be initiated. Test results from the previous run may be carried forward to the new test run if they are not affected by the changes introduced by the freshly generated image.

Reporting Bugs and Debugging Problems

If defects/problems are encountered, please go ahead and file the bugs following the guide below:

Reviewers

References