Test Plan

1. Analyze the product.

► What to Analyze

o Users (who they are and what they do)

o Operations (what it’s used for)

o Product Structure (code, files, etc.)

o Product Functions (what it does)

o Product Data (input, output, states, etc.)

o Platforms (external hardware and software)

► Ways to Analyze

o Perform product/prototype walkthrough.

o Review product and project documentation.

o Interview designers and users.

o Compare w/similar products.

► Possible Work Products

o Product coverage outline

o Annotated specifications

o Product Issue list

► Status Check

o Do designers approve of the product coverage outline?

o Do designers think you understand the product?

o Can you visualize the product and predict behavior?

o Are you able to produce test data (input and results)?

o Can you configure and operate the product?

o Do you understand how the product will be used?

o Are you aware of gaps or inconsistencies in the design?

o Do you have remaining questions regarding the product?

Analyze product risk.

► What to Analyze

o Threats

o Product vulnerabilities

o Failure modes

o Victim impact

► Ways to Analyze

o Review requirements and specifications.

o Review problem occurrences.

o Interview designers and users.

o Review product against risk heuristics and quality criteria categories.

o Identify general fault/failure patterns.

► Possible Work Products

o Component risk matrices

o Failure mode outline

► Status Check

o Do the designers and users concur with the risk analysis?

o Will you be able to detect all significant kinds of problems, should they occur during testing?

o Do you know where to focus testing effort for maximum effectiveness?

o Can the designers do anything to make important problems easier to detect, or less likely to occur?

o How will you discover if your risk analysis is accurate?

Plan logistics.

► Logistical Areas

o Test effort estimation and scheduling

o Testability engineering

o Test team staffing (right skills)

o Tester training and supervision

o Tester task assignments

o Product information gathering and management

o Project meetings, communication, and coordination

o Relations with all other project functions, including development

o Test platform acquisition and configuration

► Possible Work Products

o Issues list

o Project risk analysis

o Responsibility matrix

o Test schedule

o Agreements and protocols

o Test tools and automation

o Stubbing and simulation needs

o Test suite management and maintenance

o Build and transmittal protocol

o Test cycle administration

o Problem reporting system and protocol

o Test status reporting protocol

o Code freeze and incremental testing

o Pressure management in end game

o Sign-off protocol

o Evaluation of test effectiveness

► Status Check

o Do the logistics of the project support the test strategy?

o Are there any problems that block testing?

o Are the logistics and strategy adaptable in the face of foreseeable problems?

o Can you start testing now and sort out the rest of the issues later?

A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The completed document will help people outside the test group understand the 'why' and 'how' of product validation. It should be thorough enough to be useful but not so thorough that no one outside the test group will read it. The following are some of the items that might be included in a test plan, depending on the particular project:

* Title

* Identification of software including version/release numbers

* Revision history of document including authors, dates, approvals

* Table of Contents

* Purpose of document, intended audience

* Objective of testing effort

* Software product overview

* Relevant related document list, such as requirements, design documents, other test plans, etc.

* Relevant standards or legal requirements

* Traceability requirements

* Relevant naming conventions and identifier conventions

* Overall software project organization and personnel/contact-info/responsibilties

* Test organization and personnel/contact-info/responsibilities

* Assumptions and dependencies

* Project risk analysis

* Testing priorities and focus

* Scope and limitations of testing

* Test outline - a decomposition of the test approach by test type, feature, functionality, process, system, module, etc. as applicable

* Outline of data input equivalence classes, boundary value analysis, error classes

* Test environment - hardware, operating systems, other required software, data configurations, interfaces to other systems

* Test environment validity analysis - differences between the test and production systems and their impact on test validity.

* Test environment setup and configuration issues

* Software migration processes

* Software CM processes

* Test data setup requirements

* Database setup requirements

* Outline of system-logging/error-logging/other capabilities, and tools such as screen capture software, that will be used to help describe and report bugs

* Discussion of any specialized software or hardware tools that will be used by testers to help track the cause or source of bugs

* Test automation - justification and overview

* Test tools to be used, including versions, patches, etc.

* Test script/test code maintenance processes and version control

* Problem tracking and resolution - tools and processes

* Project test metrics to be used

* Reporting requirements and testing deliverables

* Software entrance and exit criteria

* Initial sanity testing period and criteria

* Test suspension and restart criteria

* Personnel allocation

* Personnel pre-training needs

* Test site/location

* Outside test organizations to be utilized and their purpose, responsibilities, deliverables, contact persons, and coordination issues

* Relevant proprietary, classified, security, and licensing issues.

* Open issues

* Appendix - glossary, acronyms, etc.

Share the plan.

► Ways to Share

o Engage designers and stakeholders in the test planning process.

o Actively solicit opinions about the test plan.

o Do everything possible to help the developers succeed.

o Help the developers understand how what they do impacts testing.

o Talk to technical writers and technical support people about sharing quality information.

o Get designers and developers to review and approve all reference materials.

o Record and reinforce agreements.

o Get people to review the plan in pieces.

o Improve reviewability by minimizing unnecessary text in test plan documents.

► Goals

o Common understanding of the test process.

o Common commitment to the test process.

o Reasonable participation in the test process.

o Management has reasonable expectations about the test process.

► Status Check

o Is the project team paying attention to the test plan?

o Does the project team, especially first line management, understand the role of the test team?

o Does the project team feel that the test team has the best interests of the project at heart?

o Is there an adversarial or constructive relationship between the test team and the rest of the project?

o Does any member of the project team feel that the testers are “off on a tangent” rather than focused on important testing tasks?

Design test strategies.

► General Strategies

o Domain testing (including boundaries)

o User testing

o Stress testing

o Regression testing

o Sequence testing

o State testing

o Specification-based testing

o Structural testing (e.g. unit testing)

► Ways to Plan

o Match strategies to risks and product areas.

o Visualize specific and practical strategies.

o Look for automation opportunities.

o Prototype test probes and harnesses.

o Don’t overplan. Let testers use their brains.

► Possible Work Products

o Itemized statement of each test strategy chosen and how it will be applied.

o Risk/task matrix.

o List of issues or challenges inherent in the chosen strategies.

o Advisory of poorly covered parts of the product.

o Test cases (if required)

► Status Check

o Do designers concur with the test strategy?

o Has the strategy made use of every available resource and helper?

o Is the test strategy too generic could it just as easily apply to any product?

o Will the strategy reveal all important problems?

Effective and timely planning can have a huge impact on your testing success. To proven test planning methods and techniques, including the master test plan and specific test plans for acceptance, systems, integration, and unit testing. How to manage test activities, estimate test effort, analyze risks, achieve buy-in. test measurement and reporting tactics for monitoring and control.

Who should be writing the testing plans and when this should start. Should these plans be written by the developer or the publisher? Should a draft be written at the first playable build, or even eariler? Again, there is no one 'correct' answer; just factors that may help you decide what is right for your environment.

Having the developer write the test plan can help keep the integrity of the original product and can help ensure that a more thorough test plan is written. Having the publisher write the test plan can help free up developers to work in their respective disciplines. Writing the test plan early in the development stage can also help the developer discover any problems before they appear, while waiting until later in the development cycle can make for a more thorough test plan.

Having the development team write the test plan can overwhelm the developer and prolong the development time. Having the pulisher write the test plan may completely miss the mark on what to test for. Writing the test plan early in the development stage can lead to makeing too vague a test plan, while waiting until later in the developer cycle can make for a test plan that is overly complex.

It's not a bad idea to have the developer start a preliminary draft of a test plan and then pass it off to the pulisher to write the final stages of the test plan. Also, handing over the design document can help the publisher write a more thorough test plan. Ideally a test plan should be a living document. The test plan should be started as early as possible and be continually updated and revised as the development cycle moves forward.

Title

Submitted to: [Name] [Address]

Submitted by: [Name] [Address]

Document No

Contract No

Date

Approvals: [Person Name] [Person Title] [Business Name]

Table of Contents

1. Introduction

* Purpose

* Scope

2. Applicability

* Applicable Documents

* Documents

3. Program Management and Planning

* The SQA Plan

* Organization

* Tasks

4. Software Training

* SQA Personnel

* Software Developer Training Certification

5. SQA Program Requirements

* Program Resources Allocation Monitoring

* SQA Program Audits

1. Scheduled Audits

2. Unscheduled Audits

3. Audits of the SQA Organization

4. Audit Reports

* SQA Records

* SQA Status Reports

* Software Documentation

* Requirements Traceability

* Software Development Process

* Project reviews

1. Formal Reviews

2. Informal Reviews

* Tools and Techniques

* Software Configuration Management

* Release Procedures

* Change Control

* Problem Reporting

* Software Testing

1. Unit Test

2. Integration Test

3. System Testing

4. Validation Testing

Attachment 1 Coding Documentation Guidelines

Attachment 2 Testing Requirements

Test Plan Template

(Name of the Product)

TABLE OF CONTENTS

1.0 INTRODUCTION

2.0 OBJECTIVES AND TASKS

2.1 Objectives

2.2 Tasks

3.0 SCOPE

4.0 Testing Strategy

4.1 Alpha Testing (Unit Testing)

4.2 System and Integration Testing

4.3 Performance and Stress Testing

4.4 User Acceptance Testing

4.5 Batch Testing

4.6 Automated Regression Testing

4.7 Beta Testing

5.0 Hardware Requirements

6.0 Environment Requirements

6.1 Main Frame

6.2 Workstation

7.0 Test Schedule

8.0 Control Procedures

9.0 Features to Be Tested

10.0 Features Not to Be Tested

11.0 Resources/Roles & Responsibilities

12.0 Schedules

13.0 Significantly Impacted Departments (SIDs)

14.0 Dependencies

15.0 Risks/Assumptions

16.0 Tools

17.0 Approvals

18.0 References

Appendices

1. BACKGROUND

2. INTRODUCTION

3. ASSUMPTIONS

4. TEST ITEMS

List each of the items (programs) to be tested.

5. FEATURES TO BE TESTED

List each of the features (functions or requirements) which will be tested or demonstrated by the test.

6. FEATURES NOT TO BE TESTED

Explicitly lists each feature, function, or requirement which won't be tested and why not.

7. APPROACH

Describe the data flows and test philosophy.

Simulation or Live execution, Etc.

8. ITEM PASS/FAIL CRITERIA Blanket statement

Itemized list of expected output and tolerances

9. SUSPENSION/RESUMPTION CRITERIA

Must the test run from start to completion?

Under what circumstances may it be resumed in the middle?

Establish check-points in long tests.

10. TEST DELIVERABLES

What, besides software, will be delivered?

Test report

Test software

11. TESTING TASKS Functional tasks (e.g., equipment set up)

Administrative tasks

12. ENVIRONMENTAL NEEDS

Security clearance

Office space & equipment

Hardware/software requirements

13. RESPONSIBILITIES

Who does the tasks in Section 10?

What does the user do?

14. STAFFING & TRAINING

15. SCHEDULE

16. RESOURCES

17. RISKS & CONTINGENCIES

18. APPROVALS

Type of Computing Environments

Web-based, Mainframe Batch

Purpose of Testing

To validate business processes are supported by the customized application.

Type of Software

Vendor-developed, Web-based

Scope of Testing

A/P, A/R, Payroll, HR, Integration with existing systems

Critical Success Factors

Security, Correctness, Performance, Ease of Use, Interoperability

Phases of Testing

Unit, System, UAT

Audience

Internal – Accounting, Payroll, HR, new and existing employees, personnel in interfaced systems

Tradeoffs

Scope, Cost

Types of Testing

Functional, based on: Business processes, Security policies, Use cases

Development Tools and Test Tools (e.g., GUI builders, automated capture/playback, etc.)

Screen capture, test management, defect management

Business/operational concerns

Payroll processing must be correct – could be fined if in error

HR processing must be correct

Accounting processing must be correct

Performance times must be within specified limits

Risks

Application Risks - Security, Correctness, Data conversion

Project Risks – Lack of experience with application and technology, No defined requirements, No defined processes, High employee turnover

Constraints

Lack of time, lack of management support, lack of experience, lack of dedicated test environment

Assumptions

Critical need of new application, Ongoing vendor support, vendor will customize application, vendor will fix defects

Deliverables

Final test report, Defect log, Baseline of correct test results for future tests

Sample Test Strategy Worksheet

Project Phase Testing Phase Stakeholders Purpose/Why

The goal in this conversation is to be as concrete as possible, within time constraints.

The overall goal of having a hosted validator is to test conformance and interop by implementations, against the protocol's testable assertions.

Since the spec documents to date are incomplete, we can't get 100% of the way in the time allotted for the bounty program.

So given all this, the suggestion is to develop a draft (likely incomplete) test plan that highlights wherever more information is needed from the WG.

A good model is the SAML 2.0 test plan, but it had an advantage that UMA currently doesn't have: a conformance requirements document! This is something the group should put a priority on.