Please note that I cannot assist you with technical issues related to the subject matter of this book. The methods presented in this book are general and can be applied to testing any type of computer software.
The Big Picture
The examples used in this book of various applications, software bugs, and software testing tools are in no way intended as an endorsement or disparagement of the software. This book is designed to guide you through the essential knowledge and skills needed to become a good software tester.
Testing Fundamentals
Applying Your Testing Skills
Chapter 14, "Site Testing," takes everything you've learned so far and applies it to a contemporary situation. You'll see how something as simple as testing a website can encompass almost every aspect of software testing.
Supplementing Your Testing
Working with Test Documentation
The Future
You will learn about various software industry goals, such as ISO 9000 and the Capability Maturity Model, and what it takes to achieve them. You will learn what types of jobs are available and where to look for them.
IN THIS PART
It's easy to take software for granted and not really appreciate how much it has infiltrated our daily lives. Most of us now can't go a day without going online and checking our email.
Disney’s Lion King, 1994–1995
Intel Pentium Floating-Point Division Bug, 1994
Their software test engineers had found the problem while running their own tests before the chip was released. In the end, Intel apologized for how it handled the defect and received a fee of more than $400 million to cover the costs of replacing the bad chips.
NASA Mars Polar Lander, 1999
Intel management decided that the problem was not severe enough or likely enough to warrant fixing it or even publishing it. After the bug was discovered, Intel tried to downplay its perceived severity through press releases and public statements.
Patriot Missile Defense System, 1991
Unfortunately, during their testing, the Failure Review Board found that in most cases when the legs opened for landing, a mechanical vibration also activated the landing switch, activating the fatal bit. The first team never looked to see if the touch-down play had been set; it was not their territory; the second team always reset the computer and cleared the bit before starting testing.
The Y2K (Year 2000) Bug, circa 1974
It is very likely that, thinking it had landed, the computer turned off the thrusters and broke the Mars Polar Lander into pieces after falling 1,800 meters to the surface. One team tested the leg drop procedure and another the landing process from that point.
Dangerous Viewing Ahead, 2004
Terms for Software Failures
As a software tester, it's important to understand the personality behind the product development team you're working with. It doesn't matter if it's big, small, planned, unintentional, if someone's feelings are going to be hurt because they created it.
Software Bug: A Formal Definition
True or False: It’s important what term your company or team calls a problem in its software
True or False: A good tester relentlessly strives for perfection
The specifics of what these people do, how they interact, and how they make decisions are all part of the software development process. The goal is to give you an overview of all the parts that go into a software product and a look at some of the common approaches used today.
What Effort Goes Into a Software Product?
All this information is then studied, condensed and interpreted to decide exactly what features the software product should have. The test plan describes the general methodology to be used to verify that the software meets the product specification and customer needs.
What Parts Make Up a Software Product?
Now that you know what's part of a software product and what comes with it, it's time to get to know all the people who make the software. The process used to create a software product from its initial design to public release is known as the software development life cycle model.
Big-Bang Model
The longer it takes you to do your job and the more bugs you encounter, the more controversial the situation will become.
Code-and-Fix Model
You will most likely encounter the code-and-reg model during your work as a software tester. It is a good introduction to software development and will help you appreciate the more formal methods.
Waterfall Model
The downside is that in today's fast-paced culture, where products are being developed in the Internet age, by the time a software product is so carefully thought out and defined, the original reason for its creation may have changed. Once the software is delivered to the test team, every detail has been determined, written down, and changed into the software.
Spiral Model
You will get the opportunity to influence the product early by participating in the early design stages. In Chapter 1, "Background of Software Testing" and Chapter 2, "The Software Development Process," you learned about the basics of software testing and the software development process.
It’s Impossible to Test a Program Completely
Remember, you're not limited to just clicking numbers on the screen—you can also press buttons on your computer keyboard. If you decide to eliminate any of the trial conditions because you feel they are redundant or unnecessary, or simply to save time, you have decided not to fully test the program.
Software Testing Is a Risk-Based Exercise
Everything you've tested so far should be retested by pressing the Backspace key for each entry, for every two entries, and so on. If you cut testing short or make poor decisions about what to test, the costs are low, but you'll miss a lot of defects.
Testing Can’t Show That Bugs Don’t Exist
If you try to test everything, the costs increase dramatically and the number of missed bugs decreases to the point that it is no longer profitable to continue. The goal is to achieve the optimal amount of testing so that you don't test too much or too little.
The More Bugs You Find, the More Bugs There Are
You can run your own tests, find and report bugs, but at no time can you guarantee that there are no more bugs to find.
The Pesticide Paradox
To overcome the pesticide paradox, software testers must continuously write new and different tests to train different parts of the program and find more bugs.
Not All the Bugs You Find Will Be Fixed
Bugs that occur infrequently or bugs that appear in little-used features can be ignored. Defects that have workarounds, ways a user can prevent or avoid the error, are often not fixed.
When a Bug’s a Bug Is Difficult to Say
Following these rules helps clarify the dilemma by making a bug a bug only if it is observed. One person might say the program is incredibly buggy, while another might say it's perfect.
Product Specifications Are Never Final
The software is difficult to understand, difficult to use, slow, or - in the eyes of the software tester - will be perceived by the end user as simply not correct. Claiming that the software does or does not do "something" implies that the software has been run and that "something" or the lack of "something" has been seen.
Software Testers Aren’t the Most Popular Members of a Project Team
Features you had already tested and reported bugs about will be changed or even removed.
Software Testing Is a Disciplined Technical Profession
This chapter concludes the first section of this book with a list of software testing terms and their definitions. Because they are often confused or used inappropriately, they are defined here as pairs to help you understand their true meanings and the differences between them.
Precision and Accuracy
Be aware that there is little agreement in the software industry on the definition of many seemingly common terms. Whether the software you test needs to be precise or accurate depends a lot on what the product is and ultimately what the development team is aiming for (pardon the pun).
Verification and Validation
Unfortunately, soon after putting it to use, it turned out that the images it returned were out of focus. Testing had confirmed that the mirror met the specification - verification - but it did not confirm that it met the original requirement - validation.
Quality and Reliability
For this reason, the only means of testing was to carefully measure all of its properties and compare the measurements to what was specified. By checking the specification and validating the final product, you will avoid problems like the one that afflicted the Hubble telescope.
Testing and Quality Assurance (QA)
If that's the case, don't worry—you can still use the techniques presented here to test the final specs. Think back to the four development models presented in Chapter 2, "The Software Development Process": big-bang,.
Black-Box and White-Box Testing
Static and Dynamic Testing
Static Black-Box Testing: Testing the Specification
Tell your project team, "This is what I'm going to test and submit bugs." You'll be surprised how much detail they'll fill in right away. You can test a specification using static black-box techniques, regardless of the format of the specification.
Pretend to Be the Customer
If you better understand the whys and hows behind the specification, you'll be much better at examining it in detail. Your users will assume that the software is secure, but you cannot assume that the programmers will handle security issues properly.
Research Existing Standards and Guidelines
Safety standards. Your software, its interfaces and protocols may need to meet certain security standards or levels. It may also need to be independently certified that it indeed meets the necessary criteria.
Review and Test Similar Software
Graphical User Interface (GUI). If your software runs under Microsoft Windows or Apple Macintosh operating systems, there are published standards and guidelines for how the software should look and feel to a user. Armed with this information, you can proceed to test the specification at a lower level.
Specification Attributes Checklist
Once you've reviewed the high-level product specification, you'll gain a better understanding of what your product is and what external influences impact its design. When you test a product specification, read its text, or examine its numbers, carefully consider each of these features.
Specification Terminology Checklist
Another name often used for dynamic black box testing is behavioral testing because you are testing how the software actually behaves when it is used. However, you are also forcing an error, so it could be considered a test to failure.
Boundary Conditions
The second partition should contain data that you would expect to cause an error - the one or two invalid points out of bounds. It's the most efficient way to reduce the amount of testing you need to do.
Sub-Boundary Conditions
For the 16th through 256th commands, the software can then switch to sending longer byte-encoded commands. For example, if you are testing a text box that accepts only the characters A–Z and a–z, you should include in your null break the values just below and above those in the ASCII table—@, [ ‘, and.
Default, Empty, Blank, Null, Zero, and None
If you are testing software that performs text entry or text conversion, you would be very wise to refer to a copy of the ASCII table and consider its boundary conditions when defining which values to include in your data partitions. Although ASCII is still very popular as the common means for software to represent character data, it is being replaced by a new standard called Unicode.
Invalid, Wrong, Incorrect, and Garbage Data
There are no real rules to this test other than trying to break the software. So far, what you have tested is the data - the numbers, words, input and output from the software.
Testing the Software’s Logic Flow
If you had infinite time, you would want to test every path through the software. State and state transition testing involves checking all state variables: the static conditions, information, values, functionality, and so on, associated with being in that state or moving to and from that state.
Testing States to Fail
Shutting down or starting up two or more instances of the software at the same time. If the software works on peripherals such as printers or communication ports, connect as many as you can.
Behave Like a Dumb User
It's probably not possible to open and close your program a million times if you do it by hand. Finding them might seem a bit subjective and not necessarily based on common sense, but if you want to eliminate every single bug, you'll have to get a little creative.
Look for Bugs Where You’ve Already Found Them
Adding these test cases to your designed test case library will create a very complete set.
Think like a Hacker
Follow Experience, Intuition, and Hunches
True or False: You can perform dynamic black-box testing without a product specification or requirements document
Suppose you have a text box with a zip code that is ten characters wide, as shown in Figure 5.13. True or False: Visiting all states of a program will ensure that you have completed all transitions between them.
True or False: Visiting all the states that a program has assures that you’ve also traversed all the transitions among them
If you are testing a program's ability to print to a printer, what generic test-fail test cases might be appropriate. What boundary conditions exist for the print range function shown in the lower left corner.
True or False: It’s an unfair test to perform stress testing at the same time you perform load testing
True or False: It is an unfair test to perform stress testing and load testing at the same time. It's not rocket science (unless you're designing rockets), but to get started you'll need to know a few basic techniques.
Peer Reviews
The solutions. Solutions to difficult problems can be found, although whether they are discussed depends on the rules for review. Formal reviews are a great way to get them in the same room, discussing all the same project issues.
Walkthroughs
Inspections
These are best found through careful code analysis – senior programmers and testers are great at this. Some teams implement standards and guidelines for stylistic aspects (like indentation) so that the look and feel of the code doesn't become too random.
Obtaining Standards
Do not use setjmp and longjmp if there are objects with destructors that can be created]. These checklists1 supplement the comparison of the code against a standard or guideline and ensure that the code meets the design requirements of the project.
Data Reference Errors
There are also documents demonstrating programming guidelines and best practices available from professional organizations such as Is a variable used where a constant would actually work better - for example when checking the bounds of an array.
Data Declaration Errors
Are integer values of arrays and arrays and are always within the bounds of the array or string dimension. Are there any possible "off by one" errors in indexing operations or string subscript references?
Computation Errors
Comparison Errors
Control Flow Errors
Subroutine Parameter Errors
Input/Output Errors
Does the software strictly adhere to the specified format for the data read or written by the external device. Does the software handle the situation of the external device being disconnected, unavailable or full during a read or write.
Other Checks
True or False: Static white-box testing can find missing items as well as problems
In this chapter, you will learn the fourth fundamental technique—dynamic white-box testing. You will look inside the software "box" with your X-ray glasses while testing the software.
Unit and Integration Testing
The display module would read the data and display the temperature, just as if it were reading directly from a real thermometer. With this test stub configuration, you can quickly loop through several test values and validate the functionality of the display module.
An Example of Module Testing
Finally, you would look at the code to see how the function was implemented and use your white-box knowledge of the module to add or remove test cases. Adding and removing test cases based on your white-box knowledge is really just a further refinement of the equivalence partitions done with internal information.
Data Flow
The logical approach is to split the code just as you did in black box testing - into its data and its conditions (or program flow). By looking at the software from the same perspective, you can more easily map the white-box information you find to the black-box cases you've already written.
Sub-Boundaries
You need to ask yourself if there is any way that n can become zero and figure out what inputs you need to feed the program to do so. Scan your code for formulas and equations, see the variables they use, and create test cases and equivalent breaks for them in addition to normal program inputs and outputs.
Error Forcing
Code coverage analyzers connect to the software you test and run transparently in the background while you run your test cases. If your test cases cover 90 percent of the software and find no bugs, the software is in reasonably good shape.
Program Statement and Line Coverage
If your test cases cover 90 percent of the software and you don't find any bugs, your software may still not be in good shape - it could be that it has become immune. You can run your own tests and add test cases until you touch every statement in the program.
Branch Coverage
END IF
In the case of the short program shown in Listing 7.1, 100 percent statement coverage would be the execution of lines 1 through 4. You might think that this would be the perfect way to make sure you have fully tested your program.
Condition Coverage
True or False: If your product development is in a hurry, you can skip module testing and proceed directly to integration testing
True or False: Always design your black-box test cases first
If you are just starting to test software, one of your first tasks may be to test the configuration. Interfaces. Components and peripherals connect to your computer through various types of interface connectors (see Figure 8.3).
Isolating Configuration Bugs
Say the fault is in a printer, and that printer is the most popular in the world, with tens of millions in use. In the end, the sound card manufacturer admitted that there was a problem and promised to fix the problem in updated versions of its device driver.
Sizing Up the Job
This problem only happened with the most popular sound card on the market—obviously. After extensive test setup and debugging, the fault was isolated to the sound card hardware.
Decide the Types of Hardware You’ll Need
If you don't have experience with the hardware your software runs on, you should learn as much as you can and get other experienced testers or programmers to help you. The following sections outline the general process you should use when planning to test your configuration.
Decide What Hardware Brands, Models, and Device Drivers Are Available
Decide Which Hardware Features, Modes, and Options Are Possible
Pare Down the Identified Hardware Configurations to a Manageable Set
When you create your equivalence partitions, you can decide to test only the most popular printers or those that are less than five years old. With the type information—in this case laser or inkjet—you can decide to test 75 percent lasers and 25 percent inkjets.
Identify Your Software’s Unique Features That Work with the Hardware Configurations
Note that Figure 8.7 also has columns with information about the popularity, type, and age of the device. For example, if you are testing a word processor such as WordPad (see Figure 8.8), you do not need to test the file save and load feature in every configuration.
Design the Test Cases to Run on Each Configuration
A good test would be to create a document containing different (chosen with equal partitioning, of course) fonts, point sizes, colors, embedded images, etc. In reality, the steps would be much more complicated, including more details and specifics about what exactly to do and what to look for.
Execute the Tests on Each Configuration
Rerun the Tests Until the Results Satisfy Your Team
If you've been tasked with doing configuration testing for your project, take a deep breath, reread this chapter, plan your work carefully, and take your time. If the software you are testing is a platform, what applications are designed to run under it.
Backward and Forward Compatibility
The Impact of Testing Multiple Versions
In short, you can't test all the thousands of software programs on your operating system, so you have to decide which ones are the most important to test. You need to decide which platform versions you should test your software on and which other software applications you should test your software with.
High-Level Standards and Guidelines
Low-Level Standards and Guidelines
True or False: All software must undergo some level of compatibility testing
True or False: Compatibility is a product feature and can have different levels of compliance
It is important that you or someone on your testing team is at least somewhat familiar with the language you are testing. It is not a requirement that everyone on the testing team speak the language in which the software is being localized; maybe you only need one person.
Text Expansion
Of course it would be helpful to know a little about the language, but you will find that you may be able to do quite a bit of the test without being completely fluent.
ASCII, DBCS, and Unicode
If your software is running in Quebec on a French PC, it may load and use a code page that supports French characters. Because Unicode is a global standard supported by the major software companies, hardware manufacturers, and other standards groups, it is becoming increasingly common.
Hot Keys and Shortcuts
A common approach in the days of MS-DOS, but still in use today, is to use a technique called code pages. If it is at all possible that your software will ever be localized, you and the programmers in your project should shorten the links with "ol".
Extended Characters
In localized versions of your software, you'll need to test that all the main keys and shortcuts work properly and aren't too difficult to use - for example, if a third keystroke is required.
Computations on Characters
Reading Left to Right and Right to Left
Text in Graphics
Keep the Text out of the Code
Do not allow strings to be nested in code, and do not allow them to be built into larger strings by code. Remember those terms from Chapter 3, “The Realities of Software Testing”: accuracy, precision, and reliability, and quality.
Content
Content is all the other “stuff” besides the code that goes into the product (see Chapter 2, “The Software Development Process”). Don't consider it an exhaustive list; depending on the product there could be many more examples.
Data Formats
If you are testing localized software, you will need to become very familiar with the units of measurement used by the target location. To properly test the Software, you will need to create different equivalence partitions of test data from those you create to test the original version of the Software.
Data Compatibility
Recall from Chapter 1, "Software Testing Background," the fifth rule of what constitutes a bug: The software is difficult to understand, difficult to use, slow, or—in the eyes of the software tester—will be perceived by the end user as just plainly not right. As you learned in Chapter 10, "Foreign Language Testing," the reason may be that the software is not localized properly.
Follows Standards and Guidelines
It is also possible that your platform does not have a standard, or perhaps your software is the platform. In these situations, your design team will be the ones creating the usability standards for your software.
Intuitive
Consistent
In WordPad, a very similar program, you access it through the Edit menu or by pressing Ctrl+F. Shortcut keys and menu selections. In a voicemail system, pressing 0, not other numbers, is almost always the exit button that connects you to a real person.
Flexible
Comfortable
Correct
You can repeatedly click the Abort button throughout the scan, which can take a minute or more, and nothing happens. If clicking the Abort button with the hourglass cursor stopped the scan, would that be an error?
Useful
Such a person may not be able to hear the sounds or voices that accompany an on-screen video, voice assistance, or system alerts. For example, they may not be able to press more than one key at a time, or they may find it impossible to press a key only once.
Legal Requirements
Hearing damage. Someone may be partially or totally deaf, have trouble hearing certain frequencies, or pick out a certain sound out of background noise. Movement impairments. Disease or injury can cause a person to lose fine, gross or total motor control of their hands or arms.
Accessibility Features in Software
