The Testing Environment
10.5 THE GOAL OF A TESTING ENVIRONMENT
The goal of a testing environment is to cause the application under test to exhibit true production behavior while being observed and measured outside of its produc- tion environment. Achieving this goal can be just as challenging as designing and executing the tests themselves.
As previously stated, the test team becomes aware of the application’s target pro- duction environment very early in the software development lifecycle. Because it takes time to design and code software to be tested, the test team has a built-in window of opportunity to plan the test environment, acquire the needed hardware and software, and set up the test environment well in advance of the fi rst scheduled test executions.
This test environment setup should include activities that validate the viability and correctness of the test environment for running tests. If the decision is made to use au- tomated test tools, then the test environment setup must include installing the test tool in the testing environment, validating that the test tool is operational and, if necessary, calibrating the tool. We will examine automated test tools in the next chapter.
You may need to enlist the help of several different kinds of systems experts to acquire and set up the test environment components. Out-of-the-box default operat- ing system installations rarely refl ect the way they are used in business. If the test environment operating system is very complex, you will need to fi nd an operating systems expert to set it up correctly in the testing environment. Here is a more com- plete list of test environment components for which you may need to seek expert systems assistance for testing setup.
Operating systems Security
•
•
File systems Databases
Connectivity (LANs, WANs, Satellite, Wireless, and Internet)
Finally, you will need to seek test data experts for three reasons. The fi rst reason is that you will need to fi nd the best possible test data sources to represent expected production data. The second reason is that you will need to determine how best to load that test data into the test environment as a testing starting point. The third reason is that you will need to determine how best to reload that test data for reruns once a series of tests has been executed, regardless of whether the tests were success- ful. Examples of groups that you might contact for test data help include experienced end-users, senior developers, database administrators, and operations management.
The way you load and reload test data in the test environment seldom refl ects the production backup and recovery strategy for the application under test. Produc- tion backup and recovery serve an entirely different purpose for the application in production, namely business continuity. Recall from Chapter 8 that there are testing techniques specifi cally for backup and recovery.
It is common to fi nd multiple testing environments in many development shops.
The fi rst and most common testing environment you will fi nd is the developer environ- ment itself that may not be considered a true testing environment. Recall the white box testing discussion in Chapter 7. This kind of testing must be performed by the devel- oper because the tester rarely has access to the source code under construction. Quite naturally, the developer completes white box testing in the same environment in which he or she does development. The observation is made to differentiate code debugging from testing but does not add any further value to the discussion of test environments.
The next kind of testing environment you may fi nd is called a “staging” environ- ment or a “migration” environment or a “deployment” environment. The developers see this environment as the next place their programs go prior to “going live.” It repre- sents the collection point over time of all fi nished programming. It also represents an excellent testing environment for the fi nished programming both as the fi nished com- ponents become available and as the application itself becomes complete. Figure 10.1 illustrates what a simple computing environment strategy such as this might look like.
•
•
•
Figure 10.1 Simple computing environment strategy Development
environment Version 1
Version 2
Version 3
Staging (t est) environment
Next version Untested
and corrected
software
Production environment
Next version Tested software
only
10.5 The Goal of a Testing Environment 153
Depending on the size and complexity of the application being developed, there may be multiple smaller staging environments for major subcomponents of the ap- plication. For example, an enterprise-wide application may have separate staging environments for the accounting subcomponent, the marketing subcomponent, the manufacturing component, and the warehousing component. All of these subcom- ponent staging environments represent excellent testing environments prior to the complete enterprise-wide application availability for testing. Figure 10.2 shows what a complex computing environment strategy like this might look like.
It is important for the test team to control the test environment from a test sched- ule perspective and from a data reload and rerun perspective. Control of the testing environment needs to encompass the additional dimension of controlled staging or
“builds.” A “build” is the collection of specifi c versions of application components for executing specifi c test cases. When software is ready to be tested, that software must be moved somehow from the development environment to the testing environ- ment. Staging of software from development to test is typically accomplished either by manual procedures or by automated staging tools.
The lifetime of a test environment is determined by several factors. If the soft- ware being developed is expected to have multiple versions over a number of months or years, then it makes good sense to establish and maintain a reusable test environ- ment for the duration of the software development schedule. The economy of scale is derived from the increasing number of subsequent versions tested in the same test environment because the cost of personnel to set up and tear down a testing environ- ment will exceed the cost of the test hardware over time.
When it is clear that the testing environment is needed for just a small number of versions to be tested or the time in between version tests is months or years, then a short-term testing environment might make more economic sense. The largest cost of setting up a short-term testing environment is usually the hardware cost. It is very diffi cult to justify the outright purchase of hardware for one or two tests. Often, it Figure 10.2 Complex computing environment strategy
Production environment
Enterprise- wide application
Tested software
only Development
environment Accounting subsystem Marketing subsystem Manufacturing subsystem Warehousing subsystem
Staging (t est) environments
Accounting Untested and corrected
Marketing Untested and corrected
Manufacturing Untested and corrected
Warehousing Untested and corrected
Final staging (t est) environment
Enterprise- wide application Untested and
corrected
is signifi cantly less expensive to rent or lease computer equipment for a short time (1–3 months) than it is to purchase the equipment. The cost of setting up and tearing down the test environment becomes a standard line item of the testing project budget.
What about testing environments whose lifetime falls somewhere in between the two extremes? The recommendation is to start with pricing research for the needed testing equipment both as a short-term rental and as a long-term purchase.
Then, determine from previous testing projects the estimated effort in setting up and tearing down the testing environment. Finally, estimate the number of times the test environment will be used over a 1-year period starting as soon as the software development project begins. Do the math and determine which approach is most cost-effective for your software development projects.
10.6 GOOD TESTING ENVIRONMENTS AND WHY