A Diversity of attention for beginners
Throughout the book, we analyzed the diversity of attention to works, as well as of the distribution of revenues, which those who produce them receive from var-ious channels. Sometimes we have exhaustive data to study the diversity of atten-tion directly, but in many cases we have only some partial or indirect evidence or hints about what its future may look like in various conditions. The whole debate on the Long Tail theory formulated by Chris Anderson (Anderson 2004, Ander-son 2006) revolves around the way in which the diversity of attention is likely to evolve in various situations (see below section 13.4). We need then to model the diversity of attention, and for this purpose we used a mathematical model known as Zipf’s law. This appendix is intended to give the interested reader a brief intro-duction to the modeling of diversity of attention in general, and to explain why Zipf’s law is used, how it works and its limitations. Less mathematically inclined readers should not be deterred by the presence of equations: one can ignore them and still understand the text. However, the equations will be helpful for those who wish to verify the mathematical reasoning for themselves, or to carry out further investigations.
This appendix uses ideas from the ranking tutorial of Lada M. Adamic (Ada-mic 2002b) and from the remarkable Wikipedia page on Zipf’s law.1
Fig. A.1. Share of total wealth held by the 20% richest individuals, depending on the value of Pareto’s law parameter. If the wealth distribution in a particular group is described by a Pareto’s law with parameter k, this graph shows the share of the total wealth held by the 20%
richest individuals, depending on the value of k.
wealth (or own more land, or have greater income) than a given number x is inversely proportional to x raised to a characteristic constant power k. The corre-sponding probability distribution can be formulated as:
P½X > x / m x h ik
;
where x is the wealth of the person under consideration, and m is the wealth of the poorest person. Throughout this appendix, we use the symbol P followed by square brackets to mean“the probability that [the expression in the brackets] is true”, and the symbol / means “directly proportional to”. When k is just below 1, about 20% of the people own 80% of the wealth, which has become known as the Pareto principle. As often happens, this rule of thumb has become well-known, but the law is less known, and people tend to forget that it has a parameter.
Figure A.1 shows that when k is varied from 0:5 to 1:5, the share of wealth held by the 20% richest individuals can range from 99% to 55%. The full Pareto prin-ciple actually states two things: that the distribution of wealth follows Pareto’s law, and that, in some observed situations, its parameter is close to 1.
Pareto’s law describes a family of functions which are examples of the wider class of power law distributions. These are so called because their probability density functions2follow power laws, meaning that it is proportional to a nega-tive exponential power of the variable. In the case of Pareto’s law, the probability density function is:
fðxÞ ¼kmk xkþ1
Pareto’s law is thus a power law distribution with index k þ 1. Power law distribu-tions are extremely widespread in nature, where they arise from processes which have no preferred scale. They also crop up often, and most relevantly for this book, in the study of ranked distributions, namely distributions of values that have been ordered by decreasing value.
The star of the show
For convenience, we would like to characterize the degree of diversity of attention to cultural works with a single number. The best way to do this is to model the ranked distribution of the number of times each work has been accessed using Zipf’s law, which has been shown often to approximate closely the real-life ranked distribution of popularity of cultural works.
George Kingsley Zipf was a Harvard University linguist who studied the fre-quency of occurrence of words in different languages. He formulated the law that bears his name in 1935, though the same pattern was already noticed much ear-lier, probably as early as 1916, by a French stenographer named Jean-Baptiste Estoup (Petruszewycz 1973). Estoup wrote a stenography manual which was re-edited many times, and he used his analysis of the frequency of words to design his method, but he did not model it as systematically as Zipf.
What Zipf stated is that, if one ranks the number of occurrences of words in a large text, starting with the most frequent word and continuing in decreasing order of frequency, then the number of occurrences O of each word is inversely proportional to its rank k elevated to a constant power :
OðkÞ / 1 k
Zipf also remarked that the parameter was close to 1, so that the 100th most frequent word is approximately 100 times rarer than the most frequent word. As with Pareto’s law, this rule of thumb has stuck, and commentators often forget that the parameter can take other values.
diversity of attention for beginners 183
Fig. A.2. The number of occurrences of words in a French text of 30,000 words, as reported by Estoup, reproduced in (Petruszewycz 1973).
The use of‘‘Zipf’s law with parameter close to one” has become part of the folk-lore in several fields: linguistics and bibliometrics, of course, but also more re-cently Internet studies (Shirky 2003b) and cultural studies. In this last field, Zipf’s law was used to study the distribution of access to various works in libraries, in commercial sales, and on the Internet. As we will see below, the distribution of popularity of works does not follow Zipf’s law exactly (in the statistical sense).
However, one can still tune the parameter so that the function reproduces the observed distribution as closely as possible. The resulting match is then often so good that, for practical purposes, the value of that gives the best fit can be used as a single number characterizing the degree of diversity of attention to works. In other cases, we will have to be cautious because the Zipf’s law approximation becomes too distant from what is observed in real-life.
When this diversity is modeled by Zipf’s laws, changes in the parameter lead to great differences in the degree to which attention is concentrated on a limited number of works, or sales concentrated on a limited number of products. For
example, let us consider a sample, or“universe”, of one thousand works. If the access to these works is described by Zipf’s law with parameter 0:5, the 5% most popular works receive 20% of the attention. If the parameter is 1:5, the same 5%
accounts for 92% of all access. Perhaps surprisingly, differences as strong as these are in fact observed between the various forms of access to cultural works.
Why do some distributions follow Zipf’s law?
There have been many attempts at explaining why some distributions observed in real life are very similar to Zipf’s law. George Zipf himself suggested that it results from the tendency of human beings to select the least-effort route to any particu-lar result. Here we propose an explanation that builds on the same ideas, though we formulate it slightly differently. However, it is beyond the scope of this book to model the conditions which give rise to Zipf’s law in detail: the remarks which follow are merely intended to shed some light on the processes at work.
There are common traits of the kind of variables which follow Zipf’s law, such as the level of attention, the usage of or access to works, the income or wealth of individuals, the size of cities, or the occurrences of words in a language. Gener-ally, a variable follows a power law distribution when two conditions are fulfilled:
– its values are constrained by a resource which, while abundant globally, is limited at the level of individual units;
– the larger the variable, the easier it is for it to grow.
Both parts of the first condition are equally important. The constraining resource can be on the offer or on the demand side. There is a lot of wealth overall, but it is not so easy for most individuals to get more of it for themselves. There are many words in a language, but using the rarer ones requires extra effort, for most peo-ple at least (in fact, the effort is approximately inversely proportional to the scar-city of each word).3There are many inhabitants in a country, but a given individ-ual can only live in one place. And finally, the overall attention span of Internet users on the planet is certainly not scarce, but each individual has only a limited amount of time. The second condition may be somewhat more self-evident: it is a well-known fact that being rich makes it considerably easier to become richer.
Frequently used words are in everybody’s mind and get used – to some degree – more frequently. This dynamic is most visible when a new buzzword appears. It has limits, however, since one cannot use the same word all the time. Similarly, a cultural work that has already received significant attention is likely to receive more, through word of mouth or through investment in its promotion… up to a certain point where it becomes more difficult to attract the attention of additional persons.
We will now look at finding out if distributions follow Zipf’s law, and at meth-ods for estimating the value of their parameter.
diversity of attention for beginners 185