EXERCISES
3.4 REACTIVE APPROACH
103
FIGURE 3.14 Opportunism suppresses plans.The robot is foraging for food and has two types of sensing; laser for obstacle avoidance and olfactory to detect the food, and is driven by a serial algorithm. At point A, the robot must move to right in order to find the food item, which is not possible for serial execution.
commonplace. The robot programming should be designed in such a way that the robot performance should not break down due to such a failure. A parallel processing over multiple sensors gets around this issue.
These difficulties of a robot interacting in the real world separate it from number crunching systems. Efforts to have parallelism in sensor actuator pairs help in sustaining the robot in scenarios of sensor failure, effectively dealing with a dynamic real world, and to have an opportunistic outlook to attain a given task were realised in reactive approach which is also termed the nouvelle AI, as it was a paradigm shift from the traditional AI.
FIGURE 3.15 Checkmate ! In his research paper titled ‘Elephants Do Not Play Chess’, Brooks [51] advocated against representational knowledge and encouraged a reflexive, behaviour based approach to robots. Brooks illustrated his argument with a satirical example, that elephants are not appreciative of a representation and rule-based game of chess; however they do exhibit intelligent behaviour.
FIGURE 3.16 Reactive Approach or vertical decomposition. Layering of behaviours, each layer is a self contained unit that can function without the other layers, prevents bottleneck and failure of one module will not stop the functioning of the robot. Adapted from Hu and Brady [157]
105 quality processors, and it was believed that sufficient progress in technology could overcome this shortcoming. The shortcoming was not in hardware, but rather in the paradigm.
Walter’s turtles, Toda’s model of fungus eaters and Braitenberg’s vehicles proved to be the stepping stones for the development of the nouvelle AI. These three developments firmly grounded the principles for developing autonomous agents via representation free routes employing embodied AI. All of the three developments are motivated by, or closely mimic animal behaviours. Toda models the basis of a self sustaining hypothetical animal with goals and survival means and Braitenberg elucidates sensory-motor concepts with his gedanken experiments, Walter’s efforts prior to both Toda and Braitenberg confirmed to the reality of such theoretical models. While Brooks was influenced by Walter, ANIMAT and ALIFE research was a direct consequence of the Fungus Eater. Three typical characteristics of the behaviour-based approach;
1. Situatedness: The robots are situated in the real world. They do not deal with abstractions and models, viz. clouds are not spheres and buildings are not cuboids.
The information obtained from the internal state or the short-term memory which contributes to local knowledge are also not in the form of maps or world models.
2. Embodiment: The robots have perceptions which allow them to integrate into the environment. Their actions have immediate feedback on their own sensor readings and hence their perceptions. Therefore, the robot is an active part of the environment.
3. Emergence: The final behaviour of the robot is an emergent phenomenon which cannot be determined prior to run time. This is a boon and also a curse, as it often leads to novelty in performance but designing robots for a desired behaviour becomes difficult.
As discussed in the previous chapter, the nouvelle AI is modelled on low to no representation, and denies mentalist models and is often designed as sensorimotor. The reactive approach is a minimal paradigm designed as a bottom-up approach where the robot action is determined by the parallel working of a number of modules working in tandem.
This is also known as vertical decomposition, as shown inFigure 3.16. The nouvelle AI gets rid of the module of planning and couples together sensing and action more tightly and heavily relies on emergent functionality. Therefore, cognition is an emergent phenomenon, an overlap of perception and action.
Rodney Brooks’ subsumption architecture [50] heralded in the nouvelle AI for robots as shown inFigure 3.15. This new paradigm soon found newer implementations as Ronald Arkin’s motor schema and Pattie Mae’s action-selection and proved to be more effective than traditional approaches.
3.4.1 Subsumption architecture and the nouvelle AI
Subsumption architecture has a very special place in modern day AI robotics. It was the first implementation of the reactive paradigm. Ever since the mid 1980s, Reactive approach, vertical decomposition, behaviour based paradigm and subsumption architecture are often used as near synonyms. Rodney Brooks argued that intelligence in robots should not be symbolical and plan based, rather it should be reactive and instinct based. Brooks based his new theory on structuring intelligence without representation nor reason, but rather agent-environment interactions.
Brooks designed the architecture as a layering of each behaviour to develop his control structure. Each layer is a finite state machine where higher levels could suppress or inhibit lower levels of behaviour. Such inhibition of information was new in the field of AI, however
Considering a more involved scenario, a beetle, which moves straight ahead while in light and stops when it is dark. If the beetle encounters an obstacle it turns and runs backwards for 10 cm. After 10 cm, if it is in light then it will again switch directions and move straight ahead. The beetle can be expressed with 3 states and 5 rules.
The apparent shortcoming of this approach is that it is a compromise on the dynamism of the system, as each FSM can only represent a limited number of states. In subsumption architecture, Brooks employed each layer as an FSM.
The design considerations for the subsumption architecture were:
1. Employ evolution as the central principle in design methodology for robots, wherein more specialised performance can be designed by adding more modules on top of a pre-existing rudiment. These modules can be introduced as hardware, software or a hybrid entity of both. Brooks calls these modules circuitry or as the vernacular
107
FIGURE 3.17 Old AI. Cognition is the result of a cyclical process and the observer is a ‘mere spectator’ and is not really participative in the cognitive cycle. Intelligence of the process is coded-in by the programmer.
regarding subsumption has come to be — layers of control. Each added layer produces some observable new behaviour for the robot interacting with its environment.
2. Each layer is a binding between perception and actuation.
3. For the sake of design and predictibility of the system, it is necessary to minimise interaction between layers.
The reactive approach had immediate results and instead of a slow-moving robot such as the Stanford cart, Brooks’ laboratory had a plethora of gizmos running around in near real-time response. The notion of reducing intelligent action to mere reflexes to the environment was radical but it solved the issues of planning and world modelling.
The argument is not just restricted to planning versus reaction, but is rather between finite state computation where the programs use a constant amount of memory versus the more powerful computational models used in classical AI. These computational models often had to address problems such as search of infinite trees and graphs which are often more of an arbitration than contained to a limited amount of memory.
As another advantage, Shakey and Stanford Cart which employed deliberative approach were bulky and had a high performance computer onboard, in comparison robots developed by Brooks with reactive paradigm were far less bulkier as they did not need high level of computation
In old AI, the programmer was all important as shown inFigure 3.17. With no plans the robot never loses sync with the world as shown inFigure 3.18. The strengths of subsumption are its design methodology, parallelism and modularity. Design methodology makes it easy to build a system and add newer hardware with another layer of behaviour. With parallelism, the ability for each layer to run independently and asynchronously incorporates the working of a dynamic world and warrants for graceful degradation in case of the failure of any layer of
FIGURE 3.18 Nouvelle AI, action, perception and cognition not cyclical but rather continious processes and cognition happens as an overlap of the action and perception whence and as it is perceived by the observer. Reactive approach builds the model of cognition around the notion of emergence with the observer being central to the model.
control. Subsumption is clearly more modular than the sense-plan-act approach, but there has also been criticism regarding this feature, as I will discuss later in the chapter.
Other than these facets of design, three more salient features of subsumption architecture are low cost, reduced computational power and re-usability of code. Subsumption is modular and therefore cheaper to build and test and a single layer of behaviour is enough to get the robot running. Such results are most appreciable in walking robots, hexapods and humanoids.
A number of iconic robots were made using subsumption: Atilla, Herbert, Chengiz and later Cog. Connell developed the COLONY architecture by considering a number of individual subsumption units as shown in Figure 3.19, and connecting them together with further prioritisation among those units, to construct 15 different behaviours for 6 task-specific levels. COLONY architecture was meant for a robotic arm mounted on a mobile base meant to retrieve soft drinks cans. COLONY was the first behaviour-based paradigm developed for arm/hand manipulators. Later onwards, dedicated behaviour based architectures for manipulators, to manipulate pick & place to yield richer behaviours and tracking operation were developed by various research groups. Various more architectures were developed with subsumption in mind, two well known ones are, SSS Architecture and the Circuit Architecture.
3.4.2 Motor schema
Motor schema [17] was developed by Ronald Arkin in late 1980s. Though it is a behavioural paradigm, instead of layers of control it uses a schema-based vectorial approach. The output
109
FIGURE 3.19 SR diagram for a trash-collecting robot designed using subsumption architecture. Simple operations such as wandering are put at the lowest end, with more involved behaviours such as pickup and homing are higher up. The higher levels suppress the lower levels.
The robot can identify the trash item(s) on seeing them at a detectable distance. The robot
‘wanders’ as long as the trash is not seen; once the trash is seen by the robot’s sensors then
‘avoiding’ obstacles and reaching the trash suppresses ‘wandering’, at the site of the trash ‘pick up’ suppresses ‘avoiding’ and once the trash has been ‘picked up’ then ‘homing’ back to a dumping yard suppresses ‘pick up’. A trash collecting and foraging robot can be made from the same basic behaviours.
FIGURE 3.20 An example from the COLONY architecture,
of intelligence and rationality which hinted at the concepts of embodiment and situatedness. Later, Toda used the terminology of ‘behaviour program’ in his fungus eater model and he contends behaviour is a means of adaptability in an unknown and apparently unsophisticated environment. In his seminal paper, Brooks introduces vertical decomposition with ‘task achieving behaviours’ to develop the subsumption architecture. The works of Brooks, Payton and Arkin broadly define behaviour as basic building blocks for robot action, or alternatively the agent’s response to its environment. In reactive approaches a primitive behaviour would mean asensorymotor pair; combination of such primitives would lead to the phenomenon of emergence in the robot. A more formalised definition would be,behaviour is a function of the structure and dynamics of its physical design and construction, the attributes of the environment in which it works, and its internal control system.
However, since performance in reactive systems is emergent it is difficult to confirm the behaviour of the system. Oftentimes a subroutine in a robot controller is wrongly referred to as a ‘behaviour’. As an apparent way out, Gat distiguishes between code and the implements using ‘Behaviour’ for code and ‘behaviour’ for the implements, viz. A Behaviour is a piece of code that produces abehaviour when it is running.
This definition is not sufficient for hybrid systems which often have deliberative, reactive and adaptive components. In reactive systems behaviour means a purely reflexive action, whereas in hybrid systems it can mean reflexive, innate and learned actions and it is difficult to differentiate between the code and the implements as the system adheres to a high degree of emergence.
Unlike subsumption architecture, motor schema doesn’t have a preset heirarchy and doesn’t attempt to suppress or inhibit a given module but rather it works to blend behaviours together. Behaviours are generated as a vector at run time and are added together like vector summation. Motor schema are easily implemented in software using artificial potential fields developed by Khatib as shown inFigure 3.21.
Arkin classifies navigation primitives from individual schemas, more complicated behaviours can be generated by superimposition of these primitives as shown inFigures 3.22 and 3.23. At Georgia Tech, Arkin developed advanced foraging behaviours using schemas for robots, Callisto, Io and Ganymede.
111
What is schema?
Schema is traditional to psychology and neurology. Anthropomorphic motivations led Arkin to develop schema for robots. Neisser defines schema as“a pattern of action as well as a pattern for action”, while Arbib explains schema as“an adaptive controller that uses an identification procedure to update its representation of the object being controlled”. In robot concerns, schema relates motor behaviour in terms of various concurrent stimuli and it also determines how to react and how that reaction can be accomplished. Schemas are implemented using potential fields; individual schemas are often depicted as needle diagrams. Concurrently acting behaviours are coordinated using vector summations of the active schemas.
3.4.3 Action selection & bidding mechanisms
Action selection or action selection mechanism (ASM) was designed by Pattie Maes [?, 217] and it takes a different approach than suppressing, inhibiting or vectorially adding behaviours. Instead it employs a dynamic mechanism for behaviour selection. ASM is a selection based competitive implementation of behaviour-based paradigm and it overcomes the problem of the predefined priorities used in the subsumption architecture. Each behaviour has an associated activation level, which can be affected by the current sensor readings of the robot, its goals and the influence of other behaviours. The activation level can be interpreted as a corollary to the energy level for Beer’s cockroach simulation, the heuristic which determines the arousal and satiation level for food. Each behaviour also has some minimal requirements known as activation levels that have to be met in order to be active. These levels also decay over time. From all the active behaviours, the one with the highest activation level is chosen for actual execution. Maes was motivated by Minsky’s
‘society of mind’, arbitrary agents interacting locally to lead to an effective working of a society. However, Maes’ work also bears similarity to that of Tinberg and Baerends, and extends from biology to AI.
Maes designed every autonomous agent as a set of competence modules and, using list structures, implemented her algorithm. A competence modulei can be denoted as a tuple (ci, ai, di, αi). Whereciis a list of preconditions which have to be fulfilled before the agent can become active.aianddirepresent the expected effects of the agent’s action in terms of an add list and a delete list andαiis the activation level. A competence module isexecutable when all its preconditions are true, an executable module module whose activation level surpasses a threshold may be selected to perform some real world action.
Action selection solves the issues of loss of information in subsumption when a layer is suppressed or inhibited. It avoids the potential field implementation issues of motor schema and merits every single piece of information available to the robot. However in a dynamic implementation it is harder to predict the robot’s emergent behaviour at runtime. Maes’
action-selection has also been used in tandem with reinforcement learning and provided better results.
Action selection served as a motivation for development of control paradigms based on bidding mechanisms. In a bidding-based control architecture each behaviour bids according to the urgency for having the action executed, which can be compared in the ASM approach to preparing a list with a descending order of activation level in action-selection. However, unlike Maes’ approach bidding mechanisms do not employ a critical activation level to become active and neither do the behaviours have any preconditions to be met and they are always ready to bid. Also, the behaviours in ASM can influence the activation level of
FIGURE 3.21 Motor schema implemented using potential fields. Move forward schema (upper left), path tracking schema (upper right), obstacle schema (lower left) and superposition of two goal and one obstacle schemas (lower right).
113
FIGURE 3.22 Schema self modulation at run time. In this example the two concerned behaviours are — ‘wander’, ‘obstacle avoidance’ and ‘motion to goal’. The ‘wander’ behaviour is always acting, the other two behaviours occur due to the environment. Atpoint A the robot experiences maximum repulsion from the obstacle and feeble attraction from the goal; the vector sum doesn’t point to the goal point; and ‘obstacle avoidance’ is the dominant behaviour. Atpoint Bandpoint Cthe robot experiences less repulsion from the obstacle and more attraction to the goal point; the vector sum still doesn’t point to the goal. At point D, the repulsion from the obstacle is of very little significance and the vector sum points nearly to the goal point; ‘motion to goal’ is the dominant behaviour. In this manner, schemas implement behaviours in a dynamic manner. It is worth noting that the robot does not chart the shortest path frompoint Ato the goal.
FIGURE 3.23 Sigma diagram addition of 3 schemas. Unlike subsumption no behaviour is suppressed and throughout the previous example in Figure 3.22 these 3 schemas are acting in tandem. However obstacle avoidance is dominant atpoint A, while contributions from the other two schemas are insignificant. Nearpoint Dmotion to goal is dominant and the other two schemas contribute little to the robot’s emergent behaviour.
FIGURE 3.24 DAMN Architecture is an example of bidding-mechanisms where the behaviour module ‘votes’ to decides which action to be pursued.
other behaviours, whereas in bidding mechanisms, behaviours are totally independent of each other.
Rosenblatt developed the Distributed Architecture for Mobile Navigation project (DAMN). In this architecture, a set of behaviours which cooperate to control a robot’s path by voting for various possible actions such as control of heading angle and speed, and an arbiter decides which is the action to be performed. The action with more votes is the one which is executed. However, the set of actions is predefined and it had a grid-based path planning.
Behaviour-based architectures can be classified depending on how the coordination between behaviours occurs; (1) Competitive: in these architectures, the system selects an action and sends it to the actuators, it is a winner-take-all mechanism. Subsumption architecture, action-selection and bidding mechanisms are examples of competitive coordination, and (2) Cooperative: in these architectures the system combines the actions coming from several behaviours to produce a new one that is sent to the actuators. Motor schema and Braitenberg vehicle-3c are example of cooperative coordination.
Unified Behaviour Framework (UBF) was designed to allow a robot to use competitive as well as cooperative policies using a single architecture, as shown in Figure 3.25. UBF allows for seamlessly switching between disparate architectural policies by varying the arbitration techniques. Often a low-level controller and its behaviour is developed for a specific task; thus any future redesign or extending the utility appreciably to other tasks is severely limited. Since UBF allows for seamless switching across distinct architectural policies during runtime, it encourages reuse of behaviours, with various different modes, viz. random activation of an arbitrary behaviour, priority based selection, cooperative and semi-cooperative arbitration etc. UBFs modular architecture helps to simplify design and development by code reuse.
Bidding mechanisms and ASM attempt to put in some deliberative content in reactive architectures to design higher level behaviours. This can be seen as early attempts which led towards hybrid architectures which is discussed later in the chapter.