contract.
NoteWhen should the test plan be written?
NoteWho should write the test plan?
The acceptance test plan would ideally be written by the end-user. This is frequently not possible for a variety of reasons. If, for example, your product is commercial shrink-wrap software or a Web application, you may have thousands or even millions of very different and unknown users. In that situation, it's important to find a group of people that represent the final end-users. In many organizations, the acceptance testers are called business associates or some similar name. These people are testers that typically come from the user community. Other organizations employ the users from a particular site or company as testers. Whoever is chosen, they need to have the most complete understanding of the business application as possible.
Figure 4-5 illustrates how the realism of the test environment increases at higher levels of test.
Figure 4-5: Typical Test Levels
As we pointed out earlier, the acceptance test is not intended to be comprehensive and, therefore, doesn't last as long as the system test. Sometimes, the acceptance test on extremely large systems may take only a few days or weeks. If your acceptance testing today is going on for weeks and weeks, you may be doing "good" work, but you also may be doing more than what we call acceptance testing. In fact, you may be doing some of what we call system testing. That's fine, though, because you've just defined your levels differently than we have. The point is that a comprehensive test is not necessary in order to demonstrate that the requirements work.
NoteHow long should acceptance testing last and how comprehensive should it be?
Of course, anything is possible. Rick used to have a colleague (okay, not a colleague, but his boss) who felt that most acceptance tests were a waste of time (probably because they were often ad hoc) and should be done away with. We believe, though, that the acceptance test is a very valuable way to prove the validity of the requirements and to support buy-in from the users.
NoteCan acceptance testing be combined with system testing?
Still, with tight time crunches and shortage of personnel, does it make sense to combine system and acceptance testing? Remember that the levels are what define the environment, and each environment can be associated with a level of test. The environment includes the hardware, software, documentation, data, and people. One of the most common
differences between the system test environment and the acceptance test environment is who is doing the testing. Therefore, there are two situations where it might make sense to combine the two levels if necessitated by resource or time constraints:
1. When the users have a very active role in systems test. It's always desirable to have the users involved in the system test (and throughout the lifecycle). If they're an integral part of the systems test effort and the rest of the test environment is as realistic as possible, then it might make sense to combine acceptance testing with system testing.
2. When the users have no involvement in testing whatsoever. If the most frequent difference in the system and acceptance test environments is who is conducting the test and if the test group is conducting both tests in similar environments, then why not combine the two? We've seen situations where organizations had
historically conducted both system and acceptance testing and continued to do so even though both tests were virtually identical. In one case, the acceptance test set was a subset of the system test set and was rerun in exactly the same environment by the same people.
Should the execution of one level begin before the execution of the previous level is complete? In order to cut the elapsed time (schedule) from inception to delivery, many
organizations overlap various testing levels. The penalty for overlapping the levels, however, is extra effort. Even though the lifecycle is shortened, it will typically take more effort
because the higher level of test may find bugs that may have already been found (and sometimes fixed) by the lower level of test. Sometimes, both levels of test may be finding some of the same bugs at the same time and reporting them in slightly different words. This can add overhead to defect and configuration management.
NoteCan acceptance testing begin before the system testing is done?
Another issue may be one of first impressions. The first time the users see a nearly
complete product is often during the acceptance test. If users receive a product that has not yet completed system testing, it will potentially contain more (possibly many more) bugs, which may cause them to get the feeling that the system is unsound. Hopefully, this attitude doesn't exist in your company, because, ideally, we would like to have the users involved throughout the lifecycle.
In some instances, system testing can be combined with integration testing. Our clients who successfully use this strategy have excellent configuration management, a relatively high level of test automation, and a good working relationship between the developers and testers. The basic strategy is that the integration tests are developed around each build -usually by the test group. That is, each progressively larger build is tested as part of the integration effort. The final build contains the entire system and becomes, in fact, the
system test. This technique is not for everyone and depends on a lot of the factors that we explained above. One of the downsides of combining system and integration testing is that it removes part of the developer's responsibility for developing and delivering a finished
product and passes that responsibility to the test team.
NoteCan the system testing be combined with integration testing?