Part 1: Augmented Reality and Historical Issues
1. From AR Systems to Embodied Mind
1.3 Embodied Mind and Brain Dynamics
The coordination of the complex cellular and organic interactions in an organ-ism needs a new kind of self-organizing controlling. Their development was made possible by the evolution of nervous systems that also enabled organisms to adapt to changing living conditions and to learn bodily from experiences with its environment. We call it the emergence of the embodied mind (Mainzer 2009).
The hierarchy of anatomical organizations varies over different scales of magni-tude, from molecular dimensions to that of the entire central nervous system (CNS). The research perspectives on these hierarchical levels may concern ques-tions, for example, of how signals are integrated in dendrites, how neurons in-teract in a network, how networks inin-teract in a system like vision, how systems interact in the CNS, or how the CNS interact with its environment.
In the complex systems approach, the microscopic level of interacting neu-rons can be modeled by coupled differential equations modelling the transmis-sion of nerve impulses by each neuron. The Hodgekin-Huxley equation is an ex-ample of a nonlinear reaction diffusion equation of a travelling wave of action potentials which give a precise prediction of the speed and shape of the nerve impulse of electric voltage. In general, nerve impulses emerge as new dynamical entities like the concentric waves in chemical reactions or fluid patterns in non-equilibrium dynamics.
But, local activity of a single nerve impulse is not sufficient to understand the complex brain dynamics and the emergence of cognitive and mental abili-ties. The neocortex with its more than 1011 neurons can be considered a huge nonlinear lattice, where any two points (neurons) can interact with neural im-pulses. How can we bridge the gap between the neurophysiology of local neural activities and the psychology of mental states? A single neuron can neither think nor feel, but only fire or not fire. They are the “atoms” of the complex neural dy-namics.
In his famous book The organization of Behavior, Donald Hebb (1949) sug-gested that learning must be understood as a kind of self-organization in a com-plex brain model. As in the evolution of living organisms, the belief in organizing
“demons” could be dropped and replaced by the self-organizing procedures of the self-organizing procedures of the complex systems approach. Historically, it was the first explicit statement of the physiological learning rule for synaptic modification. Hebb used the word “connectionism” in the context of a complex brain model. He introduced the concept of the Hebbian synapse where the con-nection between two neurons should be strengthened if both neurons fired at the same time (Hebb 1949, 50).
Hebb’s statement is not a mathematically precise model. But, later on, it was used to introduce Hebb-like rules tending to sharpen up a neuron’s predis-position “without a teacher” from outside. For example, a simple mathematical version of Hebb’s rule demands that the change !wBAof a weight wBAbetween a neuron A projecting to neuron B is proportional to the average firing rate "A of A and "Bof B, i.e., !wBA= ! "B"Awith constant !. In 1949, the “Hebbian synapse”
could only be a hypothetical entity. Nowadays, its neurophysiological existence is empirically confirmed.
On the macroscopic level, Hebb-like interacting neurons generate a cell as-sembly with a certain macrodynamics (Haken 1996). Mental activities are corre-lated with cell assemblies of synchronously firing cells. For example, a synchro-nously firing cell-assembly represents a plant perceptually which is not only the sum of its perceived pixels, but characterized by some typical macroscopic fea-tures like form, background or foreground. On the next level, cell assemblies of several perceptions interact in a complex scenario. In this case, each cell-assem-bly is a firing unit, generating a cell assemcell-assem-bly of cell assemblies whose macro-dynamics is characterized by some order parameters. The order parameters may represent similar properties of the perceived objects.
There is no “mother neuron” which can feel, think, or at least coordinate the appropriate neurons. The binding problem of pixels and features in a perception is explained by cell assemblies of synchronously firing neurons dominated by learnt attractors of brain dynamics. The binding problem asked: How can the perception of entire objects be conceived without decay into millions of uncon-nected pixels and signals of firing neurons? Wolf Singer (1994) and others could confirm Donald Hebb’s concept of synchronously firing neurons by observations and measurements.
In this way, we get a hierarchy of emerging levels of cognition, starting with the microdynamics of firing neurons representing a visual perception. On the fol-lowing level, the observer becomes conscious of the perception. Then the cell as-sembly of perception is connected with the neural area that is responsible for states of consciousness. In a next step, a conscious perception can be the goal of planning activities. In this case, cell assemblies of cell assemblies are connect-ed with neural areas in the planning cortex, and so on. Even high-level concepts like self-consciousness can be explained by self-reflections of self-reflections, connected with a personal memory which is represented in corresponding cell assemblies of the brain. Brain states emerge, persist for a small fraction of time, then disappear and are replaced by other states. It is the flexibility and cre-ativeness of this process that makes a brain so successful in animals for their adaption to rapidly changing and unpredictable environments.
From Augmented Reality to the Internet of Things 29
Cell assemblies behave like individual neurons. Thus, an assembly of ran-domly interconnected neurons has a threshold firing level for the onset of global activity. If this level is not attained, the assembly will not ignite, falling back to a quiescent state. If the threshold level is exceeded, firing activity of an assembly will rise rapidly to a maximum level. These two conditions ensure that assem-blies of neurons can form assemassem-blies of assemassem-blies. Assemassem-blies emerge from the nonlinear interactions of individual neurons. Assemblies of assemblies emerge from the nonlinear interaction of assemblies. Repeated several times, one gets the model of the brain as an emergent dynamic hierarchy.
In brain research, it is assumed that all mental states are correlated to cell assemblies. The corresponding cell assemblies must empirically be identified by observational and measuring instruments. In brain reading, for example, ac-tive cell assemblies correlated with words and corresponding objects can be identified. A single neuron is not decisive and may differ among different per-sons. There are typical distribution patterns with fuzzy shapes which are repre-sented in computer simulations. Brain research is still far from observing the ac-tivities of each neuron in a brain. Nevertheless, the formal hierarchical scheme of dynamics, at least, allows an explaining model of complex mental states like, for instance, consciousness. In this model, conscious states mean that persons are aware of their activities. Self-awareness is realized by additional brain areas monitoring the neural correlates of these human activities (e.g., percep-tions, feeling, or thinking). This is a question of empirical tests, not of arm-chaired reflection (Chalmers 2010). For example, in medicine, physicians need clear criteria to determine different degrees of consciousness as mental states of patients, depending on states of their brain.
Thus, we aim at clear working hypotheses for certain applications and not at a “complete” understanding what “consciousness” means per se. Besides med-icine, the assumption of different degrees of self-awareness opens new perspec-tives of technical applications. Robots with a certain degree of self-awareness can be realized by self-monitoring and self-control which are useful for self-pro-tection and cooperation in robot teams. In technical terms, these robots have in-ternal representations of their own body and states. They can also be equipped with internal representations of other robots or humans which can be changed and adapted by learning processes. Thus, they have their own “theory of mind” with perspectives of first and second person. In this sense, even con-sciousness is no mysterious event, but observable, measurable, and explainable in appropriate research frameworks. The formal hierarchical model offers the op-portunity to build corresponding circuits and technical equipment for technical brains and robots with these abilities.
Obviously, patterns of cell assemblies in the brain are not identical with our perceptions, feeling, and thinking. But, it is well confirmed in modern brain re-search that neural patterns of firing cells are correlated with mental states. These correlates can be mathematically defined and modeled in state and parameter spaces with associated dynamical systems which allow us to test our models (Mainzer/Chua 2013). With the technology of brain reading, an analysis of cell assemblies was used to extract correlates of what is represented (e.g., pictures, words, phrases): Of course, there are only the first steps of research, but it seems to be possible at least in principle. Brain reading opens new avenues to Aug-mented Reality in neuropsychology: We learn to understand neural patterns of patients for better therapies.
Motor, cognitive, and mental abilities are stored in synaptic connections of cell assemblies. A hard core of synaptic network is already wired, when a mam-mal brain is born. But many synaptic connections are generated during growth, experience and learning phase of mammals. Firing states of neurons with repeat-ed action potentials enforce synaptic connections. Thus, during a learning phase, a cell assembly of simultaneously firing neurons creates a synaptic net-work storing the learnt information. Learning phases can be modeled technically by learning algorithms (Mainzer 2007). As we all know, the learnt information can be forgotten, when learning is not repeated and the synaptic connections decay. Thus, on the micro level, brain dynamics is determined by billions of fir-ing and not firfir-ing neurons, and, on the macro level, by emergfir-ing and changfir-ing cell assemblies of neural networks coding different neural information.
The efficiency of neural networks depends on their number of hierarchical layers. They enable the brain to connect different neural states of, e.g., visual, haptic, and auditory information. But, there are also layers monitoring perceptu-al procedures and generating visuperceptu-al consciousness: A person is aware and knows that she perceives something. Even our emotions depend on specified neural networks which are connected with all kinds of brain activity. It is a chal-lenge of brain research to identify the involved layers and networks of the brain during all kinds of mental and cognitive activities by AR technologies.
Compared with human brains, technical systems may be restricted, but they are sometimes much more effective with their specific solutions of cognitive and intelligent tasks. In Augmented Reality- and AI-technology, semantic webs and i-phones can already understand questions to some extent and even answer in natural languages. The technology of applied (speech analysis) algorithms may be different from biological procedures which were developed during evolution.
But, they solve the problem to some degree with their computer power, high speed, parallelism and storage which can be improved in the future.
From Augmented Reality to the Internet of Things 31
These procedures can be illustrated by automated translations of two lan-guages with Big Data algorithms (Mainzer 2014). A human translator must know grammar, vocabulary and the filigree meaning of both languages. The rea-son is the big number of words and phrases with multi-meaning depending on different contexts. Thus, a human translator must not only master all the nuan-ces of both languages, but also the contents of texts. This task can be managed by statistical methods on a very high level. It is not necessary to speak or to un-derstand both languages. Further on, you do not need a linguistic expert who, together with a programmer, feed a computer with linguistic knowledge or rules. You only need a mass of data in a pool with translated texts from a source language into a target language.
The Internet is an example of such a powerful store. In the meantime, nearly every group of words is translated by anyone and anywhere for several times.
Parallel texts are the basis of this kind of translations. The probability that a translation is close to a text increases with the frequency that a group of words in the data pool is translated in a certain context in a certain way. The con-text of words can be determined quantitatively by a computer with statistical measure numbers. Thus, from a technical point of view, we must not understand what “understanding” means in all its filigree meaning. May be that cognitive science and brain research will be successful someday to do that. In the mean-time, we are already mastering linguistic challenges by powerful data bases and algorithms in a better way than human linguistic experts ever did.