Software configuration management is critical to the testing effort. It is so important that if the software configuration management is done poorly, the testing effort and indeed the entire project may fail. Normally, we think of software configuration management as a process that has two distinct but related functions, as illustrated in Figure 4-10.
Figure 4-10: Functions of Configuration Management
The first of these is an administrative function that is itself sometimes called configuration management, although library management is probably a more accurate term. This is the function of creating the builds, managing changes to the software, and ensuring that the correct versions of the software and documentation are maintained and shipped. It's frequently done with the aid of commercially available or homegrown tools. The
configuration manager may be organizationally independent, or the function may fall under the development organization, the QA team, or the testing team. Configuration management is a difficult and thankless job where you're hardly ever noticed until something goes wrong -kind of like testing.
Key Point
Change Control Board (CCB) members may include:
testers developers
configuration managers database admins
users/customers customer support marketing
The other part of software configuration management is more of a management function.
This is usually done with a group called the Change Control Board (CCB), Change
Management Board (CMB), bug committee, the every-other-Tuesday meeting, or whatever you call it - really, the name is not that important. The purpose of the CCB is to determine how incidents, defects, and enhancements should be handled, as illustrated in Figure 4-11.
Basically, the CCB determines what changes should be made and in what order (i.e., the priority).
Figure 4-11: Defect Analysis Decision Tree
Key Point
After digesting dozens of books, we found that about half refer to the CCB as the Configuration Control Board, while the other half referred to CCB as the Change Control Board. We like Change Control Board.
Ideally, the CCB would comprise a group of users, developers, testers, and anyone else with a vested interest in the final product. Normally, priorities are based upon the severity of the bug (users are useful), the effort to fix the bug (developers are useful), the impact on the schedule (managers are useful), and the effort to re-test (testers are useful).
Key Point
An incident is something that doesn't work as desired or expected. It may be caused by a defect (software or testing), it may be an enhancement, or it may be a one-time anomaly (i.e., incident).
Most companies have some type of formal or informal CCB that looks at production bugs and enhancement requests. Many of these same companies do not have any special
process to handle and prioritize bugs and enhancements that occur during the course of the software development lifecycle. This is not necessarily a big problem at the lower levels of test (e.g., unit), where the required degree of formality of configuration management is less, but becomes critical at system and acceptance test times. We're discussing configuration management in greater detail in the section on system testing because it seems that is where some of the biggest configuration management problems occur. Case Study 4-4 describes a configuration management problem that occurs all too often.
Case Study 4-4: If a developer fixes a bug really fast and creates a new build, did the bug really exist?
The Bug That "Never Existed"
See if this scenario sounds familiar… The systems test is being conducted and Heather, the tester, discovers a defect. She immediately documents the bug and sends it back to the developer for resolution. Bob, the developer, can't stand the
thought that someone found a bug in his work, so he fixes it immediately and updates the version of the software under test. Out of sight, out of mind… Many developers feel that if they fix the bug fast enough, it's as if it never happened. But now, Heather and maybe some of her colleagues have to test the bug fix, possibly retest other parts of the system, or even re-run a significant part of the regression suite. If this is
repeated over and over with different developers and testers, the testing gets out of control and it's unlikely that the testers will ever finish testing the entire system. At the very least, they will have spent way too much time and money.
On the other hand, testers are also a problem. You may have heard or even said yourself, "How do you expect me to ever get this tested if you keep changing it?
Testing a spec is like walking on water, it helps if it is frozen." Well, obviously if the software is frozen prematurely, at some point the testing will become unrealistic. A bug found by tester 'A,' for example, would certainly change what tester 'B' was doing if the first bug had been fixed.
The solution to Case Study 4-4 is not to fix every bug immediately or to freeze the code prematurely, but to manage the changes to the system under test. Ideally, changes should be gathered together and re-implemented into the test environment in such a way as to reduce the required regression testing without causing a halt or slowdown to the test due to a blocking bug. The changes to the software under test should begin to slow down as the system test progresses and moves into the acceptance test execution phase. Case Study 4-5 describes an example of how one company's system test plan identifies a strategy for implementing changes to their software.
Case Study 4-5: How does your organization manage changes to the software under test?
An Example Strategy for Implementing Changes
For the first two weeks of testing, all change requests and bug fixes will be implemented in a daily build to occur at 5:30 p.m.
For the next two weeks, the test manager, development manager, user
representative, and configuration manager will meet at 10:00 every Tuesday and Thursday to prioritize all outstanding or open change requests or bug fixes. They will also decide which completed fixes will be promoted into the System Test Environment.
For the final two weeks of scheduled system test, only" show-stopper" bug fixes will be implemented into the test environment.
Many of our students and some of our clients ask about daily builds. Daily builds are not for everyone and not for the entire software development lifecycle. Early in the development phase, unit and integration testing, and even in the early stages of the system test execution phase, daily builds can be a useful tool. However, using daily builds until the very end
assures most companies that they have spent too much time doing regression testing and/or shipping a product that isn't fully tested. If the regression test suite is automated, then it is
possible to extend the daily builds well into the system test execution phase, but there still needs to be a code freeze (show-stoppers aside) during the later stages of system test and during acceptance testing.
There are very few things that a test manager can do to improve the efficiency of the testing effort that will pay a dividend as much as managing how changes are re-implemented into the test environment. Some managers of testing groups have little or no control over how changes are promoted into the test environment. Other managers may not even have a clear view of what changes are introduced and when (i.e., the developer has total control of the software under test). If you fall into one of these two categories, getting control of how changes are promoted into your test environment should be one of your top priorities.
Otherwise, you're not really managing the testing effort - you're only reacting to the changes to your environment.
Key Point
There are very few things that a test manager can do to improve the efficiency of the testing effort that will pay a dividend as much as managing how changes are re-implemented into the test environment.