sources, making it important to find agreement for the different ways to measure the level of market uncertainty. The voice and email case studies are convincing because there is such agreement.
It is important to measure market uncertainty in current markets in order to make better decisions. In Chapter 10, I discuss how to do this by example with the Voice-over IP market, illustrating how industry pundits are in disagreement about what will happen. It looks at the great variety in features with IP PBXs, showing vendors’ experimentation. Other current technologies are examined within this framework, such as wireless ser- vices in Chapter 11 and Web applications and services in Chapter 12. These examples show that it is harder to estimate market uncertainty in current environments than historical ones because there is less data to analyze.
Effect of Market Uncertainty on the Value
experiment has a value around 2.5 away from the mean. When market uncertainty is high (variance = 10), similar data yields a value of 25 away from the mean, an order of magnitude better than the low-variance case.
This shows how, in an absolute sense, as market uncertainty increases, so does the possibility of performing an experiment that is a superior match to the market, as indicated by a value far above the mean of the distribution. It illustrates that when market uncertainty is low, even the best experiment is not far from the mean, but high market uncertainty disperses the outcomes over a greater distance from the mean.
The results of this simulation match what is expected for a normal dis- tribution, as illustrated in Figure 6.2. It shows the probability of experi- ments being a particular distance from the mean. This matches the results from Figure 6.1 of a simulation with 100 experiments with different vari- ances. Looking at the percentages in Figure 6.2, it is expected that 34 per- cent of these experiments will fall between the mean and +1 standard deviation from it, but only 2 percent of these experiments will range between +2 and +3 standard deviations from the average, which is con- firmed by the simulated data in Figure 6.1. You expect to need more than 769 experiments to find a single service instance that has a value greater than +3 standard deviations from the mean.
Figure 6.1 Link of market uncertainty to value of experimentation.
20 15 25
Best Experiment
Experiment Value
Experiment Number 5
0 10
-5 0 20 40 60 80 100
-10
-20 -25 -15
Var=1 Var=5 Var=10 92 Chapter 6
Figure 6.2 Normal distribution and experimentation.
In Figure 6.2, the expected values of the best experiment of 2, 10, 100, and 1,000 are indicated as: U2, U10, U100, and U1000(as calculated by Baldwin and Clark [7]). Again, this matches the results in Figure 6.1. As expected, roughly 2 percent of the results are between +2 and +3 standard deviations from the mean, and the best of these 100 experiments is around 2.5 standard deviations above the average. As expected, for each additional experiment the margin gain is less, which shows the diminishing advantage for each extra experiment. The intuition behind this is that, as you increase the num- ber of experiments, the best of these experiments has a value that grows fur- ther from the mean, but at a decreasing rate. For example, with two experiments you expect one of the outcomes to be greater than the mean and within the first standard deviation of it. It takes roughly 10 experiments to expect the best result to be greater than 1 standard deviation from the mean, 50 experiments to be over 2 standard deviations from the average, and around 1,000 experiments to exceed the mean by 3 standard deviations.
The difference between the mean of the distribution and the best experi- mental result grows as the standard deviation increases. This implies that guessing right in markets with high market uncertainty produces greater value than guessing correctly where market uncertainty is low. In these examples, the distribution is normal, and the experiments are uncorrelated
-3 -2
Mean
U2 U10 U100 U1000
Normal Density Function
Probability
Experiment Value s.d.
-1 0 1 2 3
0 0.05
2% 2%
13% 13%
0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45
34% 34%
S.D=1
Market Uncertainty 93
(the Random Walk model). My argument holds no matter what the distri- bution or correlation between experiments. As you get more experiments, or as the variance increases, then the difference between the average of the distribution and the expected value of the best of all the experiments increases. I model the Random Walk approach as the worst case because it allows an equal probability of either a very bad result or a very good one.
When experimenters have more direction, you just expect better results.
There is an argument about whether experiments with new technologies are directionless, as indicated in Figures 6.1 and 6.2, or have a focus. This was a point of contention in my thesis defense, with Bradner and I defend- ing the Random Walk theory against HT, who believed in focused experi- ments. This idea is explored in Kaufman [8], who also believes in the Random Walk theory. As pointed out previously, the answer to this does not hurt my argument. The likely answer is that experiments are not com- pletely without focus, but the focus is less than one would like to believe.
Conclusion
Managers who don’t understand uncertainty are at a disadvantage in the uncertain world of today. Investors who understand the uncertainty of user preferences in network-based services are better able to profit from this uncertainty by thinking with an options point of view, as explained in Chapter 5. High market uncertainty is the environment in which experi- mentation thrives and promises the most money to those that predict this market. The high uncertainty means that some winners can win big, which keeps things exciting. When experimentation has huge potential payouts it is likely that more players will be in the game; this is a good thing because having more players improves the odds of finding a superior market match.
The next chapter puts all the pieces together by explaining the theory of this book in detail: how market uncertainty increases the value of distributed management. It first states some assumptions about what group of network- based services this theory is useful for. These services must have business and technical reasons that imply central management is the right choice, but because of the difficulty of meeting uncertain markets, distributed manage- ment provides services with the most value. Next, the statement of the cen- tral theory in this book links the effectiveness of management structure to market uncertainty. It explains how high market uncertainty implies greater value of distributed management structure because it allows experimenta- tion and how low market uncertainty means that central management is the best choice. The next chapter finishes up Part One and leads into Part Two with the detailed case studies of email and voice services.
94 Chapter 6
95 Managers and investors who understand at an intuitive level how market uncertainty links to the value of a particular management structure are bet- ter able to understand how users’ needs are best met in uncertain condi- tions. The theory proposed in this book explains that distributed management is a better choice in uncertain markets because it enables more experimentation, even allowing users to create new services. It explains that when users’ needs are well understood, the centralized struc- ture is likely to work well. This theory represents a framework allowing a new type of analysis that focuses on the effects of market uncertainty on the choice of management structure for network-based services. Under- standing this theory is a critical success factor for the modern manager and investors who must make decisions with incomplete information in uncer- tain conditions.
Previous chapters outlined the basic elements of this theory, explaining how market uncertainty is one important factor in the determination of how to best manage network-based services. In Chapter 2, the attributes of management structure are explained. Chapter 2 illustrated how central- ized management has many business and technical advantages but tends to limit innovations because of the difficulty of experimentation — it may even be impossible for users to experiment. It explained how distributed management allows easy experimentation, even by users, but how it may