Part 3: The Epistemology of Augmented Reality
4. Augmented perception technology and Augmented Reality
Wittgen-stein (1966b, 185) wrote in 6.43 of the Tractatus that the “ethical will” cannot change anything in the world, “it can only change the limits of the world […]
the world must thereby become quite another, it must so to speak wax or wane as a whole. The world of the happy is quite another than that of the unhap-py.”
This is another context where one might naturally speak of ‘augmenting re-ality’. It arguably involves a somewhat subjective use of the terms world or real-ity; the person whose world enlarges will likely feel that they have discovered this larger world that was there the whole time, so to say that ‘their world’ or
‘their reality’ is larger really means that their sense of reality has been augment-ed, while of course reality itself remains unchanged. People naturally tend to-wards this subjective characterization, I think, precisely because they are de-scribing an experience where nothing has changed except everything. They have not discovered any new facts; all facts now appear to them in a changed light. Thus it is not anything in the world that has changed for them; and they will naturally want to say that only the world itself has changed. Though this in-volves a subjective use of the terms world or reality, nonetheless we can see this experience as an achievement of greater objectivity, since this enlarged sense of reality is only possible for people who put in the effort to overcome their own self-involvement and transcend their own narrow concerns. And it should be possible to combine this notion of augmenting reality with the notion of aug-menting reality I sketched in the preceding sections, according to which the de-velopment of culture is the creation of new anthropocentric realities. We might want to say that each person already is an absolute value, that this is revealed to anyone who can put in the work of imagination and empathy that is needed to overcome their own self-involvement; and we might at the same time want to say this reality of absolute value only exists as a creation of culture, and that to sustain this reality a sufficient portion of the people must always be exerting themselves sufficiently to acknowledge each other’s humanity. But to argue for this would take be far beyond the scope of this paper.
4. Augmented perception technology and
about our surroundings into our perception of those surroundings. In what sense is this an augmentation of reality?
We should first bear in mind that the word “reality” in the phrase “Augment-ed Reality” as appli“Augment-ed to informational technologies can be read in contrast to virtuality. In a paper from 1994 the engineering professor Paul Milgram proposed understanding “Augmented Reality” and “virtual reality” as counterparts on a spectrum of “mixed realities”. Milgram (1994, 283) defines one end of the spec-trum as the unmixed or unaugmented reality of the “real environment”, which is
“any environment consisting solely of real objects.” On the other end is the “vir-tual environment” unmixed with any reality, which Milgram (1994, 283) defines as an environment “consisting solely of virtual objects, examples of which would include conventional computer graphic simulations, either monitor-based or im-mersive”. ‘Augmented Reality’ is situated just right of “the real environment”
and just left of “augmented virtuality” – it is an environment comprised mostly of “real” objects, augmented by some “virtual objects”.
This distinction is intuitively clear in a sense, but it tends to drive certain quite important philosophical distinctions out of sight. Consider a Heads-Up Dis-play (HUD) built into a car windshield that disDis-plays the speed I’m driving at. If the product is worth anything, the number it shows me will be the speed that I really am driving at. In Milgram’s conceptual scheme, the “real” would presum-ably include the other cars on the road, the road itself, road signs, etc. and the visual information about my speed would be not real but virtual. As Milgram’s intuitive distinction shakes out in practice, essentially only physical objects count as part of reality. But this blurs a very important distinction: the Augment-ed Reality display should display my real speAugment-ed, and not a falsehood or a fiction;
it should only augment ‘my reality’ with more information about reality. A dys-functional Heads-Up Display would give me a falsehood, and a virtual reality play would give me a fiction; but a properly functioning Augmented Reality dis-play is entirely constrained by reality and nothing else. To put it quite simply:
nothing about the speedometer or the speed that the speedometer measures be-comes any less real when we move it from the dashboard to the windshield.
The essentials of Milgram’s understanding of “Augmented Reality” have been repeated by others; for example, Marcus Tönnis (2010, V, my translation) has written that “in contrast to VR, in Augmented Reality the actual reality sur-rounding the user is augmented with three-dimensional visual elements.” Bim-ber and Raskar (2005, 2) write that “in contrast to traditional VR, in Augmented Reality the real environment is not completely suppressed; instead it plays a dominant role. Rather than immersing a person into a completely synthetic world, Augmented Reality attempts to embed synthetic supplements into the real environment.”
There are some examples of technological works labelled ‘Augmented Real-ity’ that are in fact fictional, such as Helen Papagiannis’ “AR pop-up book”,
“Who’s Afraid of Bugs.”⁶ But the vast majority of technologies that are labelled
‘Augmented Reality’ are meant to be informational rather than fictional, which is what makes “tracking” one of the fundamental challenges of this technology. (cf.
Bimber and Raskar 2005, 4) Augmented Reality devices typically explore reality, we might say, while VR devices typically explore fantasy; at any rate Augmented Reality devices do not aim to augment reality with irreality. The philosopher Or-tega y Gasset (1942, 26; my translation) characterized reality as the “counter-will”, as “that which we must reckon with, whether we want to or not”, and this comes out clearly in the HUD case: I want the display to show the speed I’m actually driving, whether I want to or not. Of course I might also want to be driving a certain speed and then adjust my speed until the desired number appears in the display; nonetheless the display is only serving its purpose if it not under the direct control of my will but rather controlled only by the reality of my actual speed.
It is important to keep this in mind, since the enthusiasts and evangelists of Augmented Reality often describe the potentialities of Augmented Reality tech-nology in terms of greater control and freedom, for example: “We’re only at the beginning of the journey right now to help give people full control of their world.” (Choi 2016) There is some truth to this, and I will return to the point in a moment; to put it succinctly: Augmented Reality technology promises to give us more control over the sorts of information about reality that our per-ceived environment displays and over the format of that display. But there is an absolute distinction to be made between technologies that allow us to control the content of those perceptions, that make the content a function of the will of the user or the designer and hence ultimately a product of whim or fantasy – vir-tual reality technologies − and technologies which are only successful if the con-tent is not in the control of anyone’s will but rather tracks an independent real-ity. (We might say that what Augmented Reality technologies really offer is augmented perception; what VR technologies offer is augmented fantasy.)
In section one of this paper I outlined one notion of what ‘augmenting real-ity’ would look like, and section two described a related notion of ‘augmenting reality’ that can be derived from the “sensibility theory” of McDowell and Wig-gins. In that sense of ‘augmenting reality’, Augmented Reality technologies do not augment reality at all. My car is really moving at a certain speed anyways
Papagiannis’ projects can be seen at http://augmentedstories.com/projects/, accessed 17 Feb-ruary 2016.
Augmented Reality and Augmented Perception 185
– the HUD simply delivers the information about that speed in a new way. No new domain of reality has emerged. Of course if Augmented Reality devices be-come far more advanced and their use far more common and widespread, it will inevitably shape our culture in unforeseeable ways, and it is possible that rich new cultural forms will emerge that truly augment reality in the way that the de-velopment of our musical sensibility has Augmented Reality. For reasons that should emerge in what follows, I think this is somewhat overoptimistic.
Augmented Reality technology might be usefully compared to the other sense of ‘Augmented Reality’ I outlined in the third section, in which we can wake up out of our own self-involvement and see our whole ‘world’ expand sud-denly and every aspect of reality become, in a certain sense, more ‘real’. It can be argued that the phrase ‘Augmented Reality’ as a description of Augmented Real-ity technologies involves a subjective use of the term “realReal-ity”, just as we are speaking subjectively when we say that the ‘reality’ of the happy person is differ-ent than the ‘reality’ of the sad person. I had said that the distinction between
“real objects” and “virtual objects” is an intuitive one. When I am driving, the other cars, the road, etc. are the real objects, and the display of my speed is the virtual object. Of course the virtual object is also ultimately a real object – the display of pixels actually does exist in the windshield and is not an optical illusion. The distinction between real and virtual is perhaps clearer in the case of simulations, for example when Augmented Reality technology allows a surgeon to see the position of the organs underneath the skin of a patient, through the visual projection of the organs onto the patient’s skin. Here again the virtual ob-ject is in one sense a real obob-ject – the proob-jection of light onto the skin is a real part of the real environment. And in a sense the surgeon is really seeing the po-sition of the organs – if the technology works well and the projection is faithful and accurate. But the real object that is the projection of light, and which allows the surgeon to see where the organs really are, is not really the organs. The pro-jection of light by itself is not “virtual” in contrast to something that is more
“real”, like the organs; the projection of light and the organ are both real parts of the real environment. But the projection of light simulates something that it is not, and that simulated content could usefully be called a “virtual ob-ject” in contrast with the “real obob-ject” that it simulates.
All this is a long way of getting to the point that the term ‘reality’ is being used subjectively here: it is the reality available from the perspective of the user that is being augmented, not reality itself. When I put on Augmented Reality glasses, ‘my reality’ is augmented, in a twofold sense. Firstly, I simply have eas-ier access to more information; if I know more facts about my environment when wearing the glasses, then in a sense ‘my reality’ is larger. In this sense however I also “augment reality” whenever I look something up in Wikipedia, or even in a
regular printed encyclopedia. Secondly, the information presented by Augment-ed Reality devices is seamlessly integratAugment-ed into my perception, such that it seems like the objects themselves are informing me about themselves, as if the whole world were more gregarious all of a sudden. If we leave aside the subjective uses of ‘world’, ‘reality’ etc., what is really happening is this: information that is al-ready known or in principle knowable is integrated in a particularly convenient way into my perception. Subjectively, however, we might say: we have given the world speech; now reality readily answers every question I ask it.
This subjective use of terms is clear in Milgram’s use of “environment” – in his terms, when I wear Augmented Reality glasses, I have added something vir-tual to the real environment.With the glasses I might be able to see what temper-ature my coffee is before I drink it, but this does not add anything to the real en-vironment, objectively speaking – the temperature of the coffee was already a part of the real environment. At most we have added the glasses to the real en-vironment, but in this sense I am adding something to the environment of my kitchen whenever I go grocery shopping. The glasses do however add informa-tion to ‘my environment’, to my percepinforma-tion of my surroundings. As Augmented Reality designer Helen Papagiannis (2011) says, Augmented Reality technology allows us “be able to see different information layered on top of our reality.”
The same could be said of Wittgenstein’s use of ‘world’ in the closing sections of the Tractatus. If I wake up out of my egocentric rut and notice the outside world, objectively speaking nothing has been added to the environment – it was all already there – but it is still intelligible if I speak as if everything has been added to my environment, to my world, my reality, where before there was very little. It is perhaps odd and verging on paradoxical that though the ex-perience is one of coming out of oneself and one’s narrow and self-centered con-cerns and discovering the outside world, and is in that sense an experience of achieving greater objectivity, it is nonetheless natural to express the experience in entirely subjective terms as an enlargening of one’s world.
The typical Augmented Reality technologies are objectively speaking no aug-mentations of reality at all; they deliver an informationally augmented percep-tion of reality rather than any true augmentapercep-tion of reality. I don’t think this ob-jection is entirely pedantic. We have seen that the intuitive distinction between
‘real objects’ and ‘virtual objects’ involves a tendency to include only physical and solid objects under the heading of the ‘real’, carrying with it the suggestion that everything else is unreal, including “the movable, the moment, / The com-ing on of feasts and the habits of saints”, etc. This is already a tendency of our materialistic and scientistic age, and it seems to me that the tendency is bad for us spiritually and morally. The term “Augmented Reality” as applied to Augment-ed Reality technologies does not neAugment-ed to necessarily confuse us, but I believe it Augmented Reality and Augmented Perception 187
encourages a certain confusion or at least reflects it. Moreover this sense of ‘the real’ as solid physical objects tends to obscure the working practices of Augment-ed Reality engineers and designers, who are of course perfectly aware that tem-perature, for example, and speed, are quite real and must be accurately meas-ured and displayed by the end products. The designation of Augmented Reality technologies as “Augmented Reality” also encourages a subjective under-standing of “reality”, which again strikes me as a potentially quite harmful con-ceptual innovation, and moreover one that obscures rather than illuminating the working practices of Augmented Reality engineers and designers, who want their devices to really work, which means tracking real phenomena. (VR designers of course also want their devices to really work and are to that extent interested in objective reality.)
Let us accept this subjective use of ‘reality’ and ask: do Augmented Reality technologies “augment reality” in the sense outlined in section three of this paper – do they enrich the user’s sense of reality? This also strikes me as unlikely and misguided. Augmented Reality technologies deliver more information to the user’s perception – information about the facts of the user’s surroundings. But Gaita and Wittgenstein both emphasize that greater factual knowledge is irrele-vant here. If I fail to acknowledge the humanity of others, this means that their independent lives are not real to me. But that other people exist and lead lives independent of me, and have cares and concerns and projects independent of mine, is not a fact I am unaware of. I am aware of the fact, but I discount it for the most part. We might say that even though I know that other people exist, I don’t see it for the most part; the fact that they lead lives of their own remains a bit of background information that is largely irrelevant to my concerns, and I see them only in terms of my own concerns. Now there is no technology that will fix this. You could not invent Augmented Reality glasses that show me people as real, such that I suddenly see beyond my own egocentric rut when I wear them. No matter how many facts we assemble, the quality of our attention to the facts and our involvement in those facts remains an ongoing problem for us, a problem that cannot be solved by the addition of new facts.
And when Wittgenstein suggests that the happy person inhabits a larger world than the unhappy person, he does not mean that the happy person knows more. The happy person accepts the world; the unhappy person places demands on the world, she accepts it only insofar as it meets the conditions she lays down. Now Augmented Reality technologies do in a sense increase our control over the world we perceive, as I suggested above – they increase our control over the form and scope of our perception. They aim to increase this control and they encourage a desire for greater control. To this extent, they seem to en-courage a personal development diametrically opposed to that recommended by
Gaita and Wittgenstein; in the terms I’ve developed here, these technologies would tend to diminish our reality. There is such a thing as an ‘information junk-ie’ or a ‘news junkjunk-ie’, someone who obsessively seeks out more and more facts;
in one sense this person would have a very broad knowledge of their world, but in another sense this is obviously a kind of escapism, and they are failing to get out of their own head and attend to the world around them. In fact, Stevens, McDowell, Wiggins, Gaita and Wittgenstein all emphasize in their various ways the need to get beyond our own will, our own need to possess and control, in order to augment reality; as does Hume in his own way, and Coleridge when he contrasts imagination with fancy and defines fancy as a function of choice.⁷ I also do not think that this is an entirely pedantic objection – I am not say-ing that Augmented Reality technologies fail to achieve somethsay-ing they are not trying to achieve, measuring them against a completely foreign standard. The term ‘Augmented Reality’ is clearly meant to be inspirational. I suspect that the term itself, and the enthusiasm for the technologies grouped under that term, reflect a spiritual longing for a transformation of our experience. I have been arguing that technology will not deliver this, precisely because and to the extent that it seeks to enhance our control over our experience⁸. The possi-bilities of augmented perception that Augmented Reality technology promises could occupy our attention endlessly, while distracting us from the task of im-proving our attention to the world as a whole. Augmented Reality technologies are perfectly positioned to seduce us – since we do long for a transformation of our experience, but we do not want to do the constant work on ourselves this would actually require, and would much rather just buy a machine that could do it for us. This is the harmful but seductive fantasy that Kierkegaard continu-ally warns us against:
From the external and visible world there comes an old adage: “Only one who works gets bread.” Oddly enough, the adage does not fit the world in which it is most at home, for im-perfection is the fundamental law of the external world, and here it happens again and again that he who does not work does get bread, and he who sleeps gets it even more abun-dantly than he who works. […] It is different in the world of the spirit.” (Kierkegaard 1983, 25)
See footnote 2.
This is perhaps not entirely true; technology in a loose sense might be able to help us. There are techniques for overcoming self-involvement in the Christian and Buddhist traditions at least;
and arguably psychoactive inventions such as LSD or ayahuasca can help someone as well.
However, these techniques and technologies are helpful only to the extent that they override our need for control; they do not place more of our experience under our control, as Augmented Reality technology promises to do, but rather actively subvert our control.
Augmented Reality and Augmented Perception 189