No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted by sections 107 or 108 of the United States Copyright Act of 1976. Operate without the publisher's prior written permission or authorization by paying the appropriate copy fee to Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA fax or online at www.copyright.com . Disclaimer/Disclaimer: Although the publisher and author have used their best efforts in the preparation of this book, they make no representations or warranties as to the accuracy or completeness of the contents of this book and expressly disclaim any implied warranties of merchantability or fitness for a particular purpose.
A Full Software Development Lifecycle Testing Project 203 1 Introduction 203
INTRODUCTION
The authors believe that cover story business disasters can best be prevented by such a comprehensive approach to software testing. Many excellent textbooks on the market deeply explore software testing for narrow segments of software development.
Chapter 1
3–5] One of the intermediate test textbooks that the authors recommend as a follow-up to this textbook is Dr.
Overview of Testing
OBJECTIVES AND LIMITS OF TESTING
- The Mind of a Tester
- Non-Software Testing at the Developer Level—
- The Four Primary Objectives of Testing
- Development Axiom—Quality Must Be Built In Because Quality Cannot Be Tested In
Some of these test objectives can be validated even before you get in the car and start the engine. It is the car design tester's job to validate the current condition of the new car against the car's requirements.
THE VALUE VERSUS COST OF TESTING
- Non-Software Testing at the Marketing Level—
- Estimating the Cost of Failure
- Basili and Boehm’s Rule of Exponentially Increasing Costs to Correct New Software
- The Pot of Gold at the End of the Internet Rainbow Software that provides businesses with a presence on the Internet can represent
- The Achilles Heel of e-Business
Present to the development manager an assessment of the risks expected due to premature termination of testing. One of the first measures a company uses to set limits around testing is the cost of testing.
RELATIONSHIP OF TESTING TO THE SOFTWARE DEVELOPMENT LIFE CYCLE
- The Evolution of Software Testing as a Technology Profession
- The Ten Principles of Good Software Testing
- The Game of “Gossip”
The question that is usually missing at this point is, "How much will it cost if the software doesn't work as advertised?" The unspoken assumption is that the software will run smoothly this time, even though no previous software development has been perfect. Because insufficient testing resources continue to be a concern, this approach provides the most "bang for the buck" testing. The other 80% of the business system typically represents exception transactions that are invoked only when the most active 20% cannot resolve a problem.
TESTER VERSUS DEVELOPER ROLES IN SOFTWARE TESTING
- A Brief History of Application Quality Expectations, or “Paradise Lost”
- The Role of Testing Professionals in Software Development
- The Role of Test Tool Experts in Software Development Mature automated test tools began to arise in the marketplace around 1995. The good
- Who Is on the Test Team?
Pressure from end users to return to the quality of the older systems prompted software development groups to seek and adopt development processes to deliver the same quality of software. Testing was considered 'monkey work'. The authors of this handbook argue that because testing was so devalued, developers with the best processes quickly ran into a wall of quality.
PUTTING SOFTWARE TESTING IN PERSPECTIVE
The mid-level testers can work within the test plan to create the test scenarios, cases and scripts that follow the plan. Then, with advice and guidance from the senior testers, a mix of mid-level and entry-level testers conduct the tests.
SUMMARY
- The Four Primary Objectives of Testing
- The Evolution of Software Testing as a Technology Profession
- The Ten Principles of Good Software Testing
By the mid-1980s, various groups of computer professionals recognized the need to develop a complete repertoire of software testing techniques. Thus, the Y2K testing frenzy shone a spotlight on the larger issues, processes, and strategies for full development lifecycle software testing.
INTRODUCTION
METHODOLOGIES AND TOOLS
Chapter 2
The Software Development Life Cycle
THE EVOLUTION OF SYSTEM DEVELOPMENT LIFE CYCLES
- The Classical SDLC
- Prototyping
- Rapid Application Development
While the order of the classic SDLC stages made intuitive sense, there were two major shortcomings. Once approved, it serves as a blueprint for developing the system according to a.
THE PHASED DEVELOPMENT METHODOLOGY
- Life Cycle Stages
- System Development Phases
- Software Testing in the Stages
This entire process represents the first cycle in the life of the system being developed. Once the new system is installed, some form of monitoring will likely be applied to ensure that the system continues to operate as designed and tested.
THE PRELIMINARY INVESTIGATION STAGE
- Perform Enterprise Analysis
- Define System Goals, Objectives, and Performance Criteria
- Evaluate System and Project Risk
- Evaluate System and Project Feasibility
- Conduct Joint Application Design (JAD) Sessions to Confirm Preliminary Findings
- Receive Approval to Proceed
With this understanding of the business and its environment, developers can focus their attention on the system and the project. If the system achieves the objectives of the performance measures, the achievement of the more general objectives is guaranteed.
THE ANALYSIS STAGE
- Analyze Functional Requirements
- Analyze Existing System Documentation
- Conduct Personal Interviews
- Conduct Surveys
- Conduct JAD Sessions
- Observe the Existing System
- Document Functional Requirements
When sources of information cannot be interviewed in person (perhaps there are too many or are spread over a wide geographic area), surveys can be designed to collect the information in the form of questionnaires by mail or telephone. Any number of documentation tools can be used to document the functional requirements, but an excellent one is the feature component matrix, illustrated in Table 2.2.
THE DESIGN STAGE
- Identify the Feasible Designs
- Evaluate the Feasible Designs
- Conduct JAD Sessions to Confirm Designs and Evaluations
- Select the Best Design
This design considers all functions from both a business perspective (sales, orders and payments) and a technology perspective (shared data, security and backup/restore). Attention is paid to the nature of the data that the system must store and how it will be stored (structure and devices).
THE PRELIMINARY CONSTRUCTION STAGE
- Construct the Software for Each Module
- Demonstrate the New System Modules to Users and Project Sponsors
The software for the integrated system is assembled and tested in the development integration environment. In the preliminary construction phase, data testing takes the form of building subsets of the database or repository and verifying that the data structures and content work as expected by the system design independently of the software that will maintain it.
THE FINAL CONSTRUCTION STAGE
- Construct and Test Production-Ready Software The tests that are conducted of the entire system are called system tests. One purpose
- Construct and Test a Production-Ready Database or Warehouse Data Repository
- Install Any Required New Hardware and Connectivity
- Make Available the Physical Facility
- Test the Hardware Configuration
- Complete the Documentation
- Gain User Acceptance to “Go Live”
- Train Participants and Users
User acceptance testing is the final test of the system to determine whether it meets the users' perceived needs. With this approval in the next phase, the system can be put into operational use.
THE INSTALLATION STAGE
- Install System Components
- Cutover to the New System
- Conduct the Postimplementation Evaluation Some time after cutover and after the system has had time to settle down, an evalu-
At this point, the developers are satisfied that the system meets its objectives. Over time as the system is used, additional assessments, both formal and informal, can be made.
PUTTING PHASED DEVELOPMENT IN PERSPECTIVE
The assessment can be carried out by users and impartial third parties such as internal auditors and consultants.
SUMMARY
Installation is where the system components are installed in the user area and the transition to the new system is performed. Sometime after cutover, a post-implementation evaluation is performed to verify the system's ability to meet the performance criteria.
INTRODUCTION
Chapter 3
Overview of Structured Testing
- CHECKLIST MENTALITY FOR SOFTWARE TESTERS
- SPRAE—A GENERIC STRUCTURED TESTING APPROACH
- Specification
- Premeditation
- Repeatability
- Accountability
- Economy
- PUTTING THE OVERVIEW OF STRUCTURED TESTING IN PERSPECTIVE
- INTRODUCTION
- THE CHESS PIECES FOR TESTING STRATEGIES Over the last 25 years of software testing, four different test approaches have emerged
- Chapter 4
This item stems from a software process dimension called maturity. The Software Engineering Institute at Carnegie-Mellon has created an industry-wide benchmark for measuring the relative success a company can expect to achieve when developing software. Likewise, you have a number of testing "pieces" on your side that you can use at various points in the game to "win" the testing game.
Testing Strategy
- Static Testing
- White Box Testing
- Black Box Testing
- Performance Testing
- THE TWO-DIMENSIONAL TESTING STRATEGY CHESS BOARD
- THE THREE-DIMENSIONAL TESTING STRATEGY CHESS BOARD
- Software Platforms—The Third Testing Strategy Dimension
- PUTTING THE TESTING STRATEGY INTO PERSPECTIVE
- INTRODUCTION
- THE TEST PLAN
- Chapter 5
Testing "inside" the code (white box) is no longer practical, because all the code components are "packaged" or integrated together by means of assembly, coupling or bindings. Reusing these tests is called "regression testing". The primary purpose of regression testing is to verify that all the changes, fixes, and additional features included in the next version do not inadvertently introduce bugs into the previously tested code.
Test Planning
- TEST CASES
- Test Case Details
- Step Action
- WRITING YOUR TEST PLAN AND TEST CASES IN THE REAL WORLD
- TEST DOCUMENT STANDARDS
- PUTTING TEST PLANNING IN PERSPECTIVE
- INTRODUCTION
- GOAL OF STATIC TESTING
- Chapter 6
One question that often comes up about a test plan is the nature of the physical document. The purpose of a test plan is to collectively document the "what" and "why" of testing.
Static Testing
- CANDIDATE DOCUMENTS FOR STATIC TESTING
- Software Development Manager’s Documents
- Software Developers’ Documents
- Testers’ Documentation
- Administrator’s Documentation
- End Users’ Documentation
- STATIC TESTING TECHNIQUES
- TRACKING DEFECTS DETECTED BY STATIC TESTING
- PUTTING STATIC TESTING IN PERSPECTIVE
- INTRODUCTION
- Chapter 7
It's time to take a look at a fairly representative list of software development document candidates for static testing. The second concern with static testing is that while writing the technical details, some of the details stray from the original business requirements or are not really part of the original business requirements (out of scope).
Functional Testing
FUNCTIONAL TEST CASES FROM USE CASES
Use case #001 provides a high-level description of the sequence of steps the actor (customer) must take to complete an online purchase. Exit Purchase Order Screen Screen Stop Successful Figure 7.3 A complete business path test case from use case.
AN APPROACH TO FUNCTIONAL TESTING
- User Navigation Testing
- Transaction Screen Testing
- Transaction Flow Testing
- Report Screen Testing
- Report Flow Testing
- Database Create/Retrieve/Update/Delete Testing Many applications use databases behind their transaction and report screens to
Similar to our suggestion with transaction flow testing, use successful report screen tests to drive the report flow test. This test is equivalent to looking at the "backside" of the application after a successful flow test has been completed.
AN APPROACH TO REGRESSION TESTING
This type of testing validates the viability of the database design for the intended application. The second step is to test the application software's use of the validated database design.
DETAILED WHITE BOX TESTING TECHNIQUES
- Statement Coverage Technique
- Branch (Simple Condition) Coverage Technique Branch coverage techniques focus on determining what percentage of the source
- Compound Condition Coverage Technique
- Path Coverage Technique
- Loop Coverage Technique
- Intuition and Experience
Loop coverage techniques focus on determining what percentage of source code loops in a program have been fully cycled. The underlying premise is that unexecuted source code is one of the most common and predictable sources of defects.
DETAILED BLACK BOX TESTING TECHNIQUES
- Equivalence Classes Technique
- Boundary Value Analysis Technique
- Expected Results Coverage Technique
- Intuition and Experience
Equivalent input classes with boundary values are run as one of the first steps in black box testing. Below is the beginning of the table of valid combinations for the hospital admission business rules.
SUMMARY
Adding new code or fixes to existing code always raises the possibility that the new or changed code may accidentally "break" already tested code. The objective of black box testing is to verify the correctness of the behavior of software that directly supports daily business activity.
PUTTING FUNCTIONAL TESTING IN PERSPECTIVE
INTRODUCTION
Chapter 8
Structural (Non-functional) Testing
- INTERFACE TESTING
- SECURITY TESTING
- INSTALLATION TESTING
- THE SMOKE TEST
- ADMINISTRATION TESTING
- BACKUP AND RECOVERY TESTING
- PUTTING STRUCTURAL TESTING IN PERSPECTIVE
- SUMMARY
- INTRODUCTION
- Chapter 9
This will allow testers to validate that the application is correctly accepting the data from the software platform. This will allow the tester to validate that the software platform is producing data according to its data specifications.
Performance Testing
WORKLOAD PLANNING TECHNIQUES
- Documenting the Performance Workload Requirements
- Documenting the Performance Workload Peaks The performance tester’s second workload challenge is to determine peak usage
In the background of all internet transaction rate discussions is the "Rule of 8". The Rule of 8 is a human behavior discovered in the 1970s by measuring the productivity of computer users as their system response time slows from subseconds to 20 seconds. Data center managers chose to "tune" these mainframes to a response time of 4-5 seconds (deliberately slowed down).
WORKLOAD EXECUTION TECHNIQUES
- Workload Ramp-up to Peak
- Workload Ramp-down From Peak
- Performance Measurement at the Peak
Workload churn is the process of gracefully terminating all the active user sessions that were driving the peak workload. Then we'll connect that measurement perspective to the workload ramp-up/down test we just completed.
COMPONENT PERFORMANCE TESTING
As with the ramp, a successful descent must first be attempted in increments of 5 or 10. After you have successfully scaled for the peak workload, you are ready to consider the timing aspects of performance measurement at peak workload.
ROUND TRIP PERFORMANCE
- A Typical Performance Response Curve
- Saturday Peak Workload in an Empty Test System We return to our performance workload Draft 4 plans to demonstrate what can hap-
- Saturday Peak Workload Mix
- Saturday Workload Showstoppers
- Saturday Workload Showstopper Corrections, We Think At this point in our discussion, we receive the corrected, functionally regressed software
- Weekday Workload Performance Plan Execution
Return performance numbers with a (red) notation indicate that the transaction group on that row is not ready for peak load testing. The next step is to run the round trip performance tests with the peak load mix.
PUTTING PERFORMANCE TESTING IN PERSPECTIVE
A weekday workload will require many more catalog browsing transaction groups than a Saturday workload test. Expect the first performance tests of any new software to reveal not "if" the software meets the requirements, but rather "how much" the software falls short of the performance requirements.
SUMMARY
The reason for this end-user attitude is because correct answers given too late lose the business opportunity just as surely as if correct answers had not been given at all. This chapter describes an approach to performance testing that will alert the development team to software performance deficiencies before the software is delivered to end users.
INTRODUCTION
Chapter 10
The Testing Environment
- SIMULATIONS
- BENCHMARKING
- TESTING ENVIRONMENTS
- THE GOAL OF A TESTING ENVIRONMENT
- GOOD TESTING ENVIRONMENTS AND WHY THEY SHOULD BE USED
- BAD TESTING ENVIRONMENTS AND WHY THEY SHOULD BE AVOIDED
- PUTTING THE TESTING ENVIRONMENT IN PERSPECTIVE
- SUMMARY
- INTRODUCTION
- Chapter 11
The closer the test environment resembles the production environment, the more valuable the test results become. The more different the test environment from the production environment, the less valid the test results become.
Automated Testing Tools
- BRIEF HISTORY OF AUTOMATED TESTING TOOLS FOR SOFTWARE
- TEST TOOL RECORD/PLAYBACK PARADIGM
- Test Script Command Language
- TEST TOOL TOUCHPOINT PARADIGMS
- Touchpoint Paradigm for Functional Test Tools The fi rst touchpoint paradigm is used by functional testing tools. The goal of this
- Touchpoint Paradigm for Performance Test Tools The second touchpoint paradigm is used by performance testing tools. The goal of
- TEST TOOL EXECUTION PARADIGM
- THE BENEFITS THAT TESTING TOOLS CAN PROVIDE Based on the previous sections in this chapter, you may conclude incorrectly that it is
- Repeatability—a Return on Investment Issue
- Repeatability Precision—a Technology Issue
- Hidden Action and Results Verification—a Technology Issue
- Timing Point Precision and Distribution—a Technology Issue
- Achievement of Impossible Manual Tests—a Return on Investment Issue
- Reusability (Regression Testing)—a Return on Investment Issue
- THE LIABILITIES THAT TESTING TOOLS CAN IMPOSE
- Testing Tool Costs—a Financial, Resources, and Skills Issue
- One-off Testing Setup Costs
- Boundaries of Testing Tool Capabilities
- PUTTING AUTOMATED TESTING TOOLS IN PERSPECTIVE
- SUMMARY
- INTRODUCTION
- TEST CASES ATTEMPTED VERSUS SUCCESSFUL
- Chapter 12
This unique way is referred to as "record/playback." The term is shorthand for the following repeatable three-step process. The test tool "looks" at the end user operations and records/saves the actions in a file of reusable commands called a script file.
Analyzing and Interpreting Test Results
DEFECT DISCOVERY FOCUSING ON INDIVIDUAL DEFECTS
The implication is that a single successful test case does not guarantee defect-free code in the area of the software being tested. The first type of severity code indicates the severity related to the test, that is, "Is this a test indicator?" The second type of severity code indicates severity related to development, that is, "Is this an indicator of development?" The third type of severity code indicates the severity related to the completion of the development, namely, "Is this a transport/deployment merchant?" The trend is towards capturing all three severity codes for each defect and using the one that makes the most sense depending on how close the development project is to completion.
DEFECT DISCOVERY FOCUSING ON THE DEFECT BACKLOG
