What are the neural pathways that support associative activation amongst various sensory, motor, and linguistic representations, and what functional properties do these pathways possess? In this section we review three somewhat different answers to this question, and review the motivation and evidence supporting each. We then offer a critical appraisal of the three views taken together, before summarizing the key conclusions.
The tripartite view: organization by modality and hemisphere
The first proposal derives from classical neurological perspectives on brain function, which have strongly emphasized gross functional dissociations among language, per- ception, and action following brain injury. Such dissociations provide the basis for the broadest relevant diagnostic categories in neuropsychology – the aphasias, agnosias, and apraxias – which are thought to reflect a coarse partitioning of cognitive function in the brain. In particular, the classical neurological view places language function in left hemisphere perisylvian regions (Mesulam et al., 2003); visual perception of objects in the ventral visual stream along the inferior surface of the temporal lobes, with per- haps greater contribution of the right hemisphere for some visual processes (Farah, 1990); and knowledge about action in the parietal and motor cortices (Goodale and Milner, 1992). Accordingly, such a view suggests a division of the conceptual knowledge network into three subnetworks: a “lexical semantic” system responsible for verbal semantic knowledge, a “visual” or “perceptual” network responsible for nonverbal knowledge about the visual or perceptual properties of objects, and an
“action” semantic system that encodes knowledge about how to interact with objects.
We will refer to this idea, illustrated in Figure 4.1, as the tripartite view.
The proposal that lexical semantics is supported by a left‐lateralized subsystem that is somewhat autonomous and separate from visual or action semantics has a long history in neuropsychology, and stems from case studies demonstrating that verbal assessments of everyday knowledge appear to doubly dissociate from visual recognition and action knowledge (Coltheart, 2004; McCarthy and Warrington, 1986). The functional dissoci- ation of lexical and perceptual semantic knowledge has been recently argued by Mesulam and colleagues (2003, 2013), who have conducted extensive case‐series analyses of patients with a progressive degenerative disorder that, in clinical assessments, appears to primarily affect language comprehension and production. Some such patients can pro- duce quite fluent speech that is largely devoid of specific content and is accompanied by a profound anomia, but nevertheless perform at or near ceiling on standard clinical tests of visual perception and object recognition. Mesulam (2001) dubbed this pattern fluent primary progressive aphasia (fPPA), and has shown that it is accompanied by gray matter atrophy that appears to be largely left‐lateralized. Taking both fluent and dysfluent cases into account, the atrophy in primary progressive aphasia is described as affecting perisyl- vian regions in the left temporal, parietal, and frontal lobes, consistent with the classical view that these regions form a left‐lateralized language network. Moreover, other cases of progressive dementia (see Chapter 20), such as the posterior cortical atrophy observed in rare cases of Alzheimer’s disease, can present with the reverse problems – severe diffi- culties in visual perception and recognition, but with language largely spared (Caine, 2004). Such case‐series studies thus reinforce the conclusions derived from single‐case studies that lexical and perceptual semantics are functionally independent.
The view that action knowledge is functionally independent of visual/perceptual and lexical knowledge also stems largely from neuropsychological dissociations among these abilities (Ochipa, Rothi, and Heilman, 1989). Perhaps most compelling in this litera- ture are the elegant studies of different forms of apraxia conducted by Buxbaum and colleagues (Binkofski and Buxbaum, 2013; Buxbaum, 2001; Buxbaum and Saffran, 2002; Buxbaum, Veramontil, and Schwartz, 2000). Working with large numbers patients with varying etiologies, these authors have identified a variety of qualitatively different disorders of action knowledge. In the case of action disorganization syndrome, patients can often retrieve the characteristic function associated with a single object, but show serious deficits organizing these simple actions into sequences that accomplish a familiar goal, such as wrapping a present or making a cup of tea (Schwartz et al., 1998).
In the case of ideational apraxia, even the simple action associations appear to be dis- rupted, with patients frequently confusing the actions associated with semantically related items (Ochipa, Rothi, and Heilman, 1989). For instance, such patients might mistakenly brush their hair with a toothbrush, or try to write with a paintbrush.
Nevertheless such patients typically show a good visual recognition and spared ability to name and verbally describe these items, suggesting that they have impaired knowledge specifically about actions associated with familiar objects. In still other cases, patients
4 7
6 6
7
1 2 3
8
Language Network 1. Output phonology 2. Input phonology 3. Verbal semantics 4. Visual word forms
Vision Network 5. Simple features 6. Color
7. Shape 8. Motion
8 4
9
11 10
8. Motion
9. Object function/action 10. Reach/grasp planning 11. Primary motor cortex 12. Supplementary motor area Action Network
5 12
Figure 4.1 Schematic illustrating the tripartite view. The left hemisphere perisylvian language network (blue) captures both classical and contemporary findings, with input phonology in the superior temporal gyrus (1), output phonology in Broca’s area (2), verbal semantics in Wernicke’s area (3), and visual word forms in the posterior ventral occipeto‐temporal cortex (4). The visual object knowledge‐network (red) includes simple visual features in occipital cortex (5), extends bilaterally along the ventral surface of the temporal lobes where color (6) and form (7) representations are thought to reside, and branches up to the human analog of area MT in posterior MTG, which encodes visual motion (8). The action network (green) incorporates the motion‐perception region in posterior MTG (8), inferior parietal cortex, which is thought to support object‐associated action knowledge (9), the superior parietal cortex, which supports visually‐guided action and grasping (10), motor and premotor cortex (11), and the supplementary motor area (12).
may show selective loss or sparing of knowledge about praxis, that is, about the particular motor actions necessary to engage an object in action (Buxbaum, 2001). For instance, such patients might not know to grasp a spoon by the handle.
The neuroanatomical correlates of such deficits have recently been laid bare by lesion‐symptom mapping methods, which suggest that impairments to knowledge about object use often arise from damage to regions in the left parietal cortex (Kalénine, Buxbaum, and Coslett, 2010). Visually guided reaching and grasping deficits appear to be associated with damage to more dorsal regions, while loss of knowledge about object‐associated action is best predicted by pathology in the left inferior parietal lobe.
Such studies thus suggest that there exist at least two dissociable components of the action‐knowledge network, differently situated within parietal cortex.
Functional imaging studies lend additional support to this idea. Boronat et al. (2005) asked subjects to judge if pairs of items were either (1) manipulated similarly or (2) used for the same purpose. The stimuli were selected so that form and function did not always correspond. For instance, the typewriter and piano engage similar praxis in the service of different functions, while the match and the lighter subserve similar functions but engage different praxis. Both kinds of judgments elicited activation within the posterior parietal cortex relative to a control baseline, but activation elicited by the manipulation judgment was located more superiorly to that elicited by the function judgment – suggesting, in concert with the lesion‐overlap data, that praxis and function knowledge may depend upon different subregions of parietal cortex.
Both functional neuroimaging and lesion‐overlap data further suggest that regions outside the parietal cortex are involved in representing knowledge about action. One such region is the posterior MTG, which appears to be strongly activated when participants retrieve semantic information about tools or other manipulable objects and when they perceive simple mechanical motion (Chao, Haxby, and Martin, 1999;
Martin et al., 1996). In a recent study of patients with impairments in action recog- nition, lesions in this region were found to correlate with impairment discriminating among semantically related manipulable objects, but not with impairment discrimi- nating objects that engage similar praxis, which tended to correlate instead with pathology in inferior parietal regions (Kalénine, Buxbaum, and Coslett, 2010).
Lesion‐symptom correlational studies also suggest that pathology in this region predicts deficits in naming highly manipulable objects (Campanella et al., 2010).
As noted in the introduction, there are also several studies suggesting that regions in the prefrontal cortex, including the supplementary motor area and premotor cortex, are important for supporting semantic knowledge about actions (Hauk, Johnsrude, and Pulvermüller, 2004). Taken together, these studies suggest the existence of an
“action‐knowledge” network that includes the left posterior MTG, several regions in the left parietal cortex, and premotor regions of the frontal lobe.
The many‐hubs view: organization by multiple domain‐specific convergence zones
A second proposal places emphasis, not on the functional independence of represen- tations in different modalities, but on the different pathways proposed to link various surface representations. The central idea is that there exist in cortex a number of cross‐modal “convergence zones,” each connecting two or more kinds of modality‐
specific representations (Damasio, 1989). Mappings among different sensory, motor,
and linguistic representations are encoded by the particular convergence zone that connects these. Thus, for instance, there may exist one convergence zone for connecting visual representations of object shape to representations of action plans;
another for connecting the same representations of object shape to visual representa- tions of motion patterns; another for connecting the object shape representations to word‐form representations; and so on. We will refer to this idea, illustrated in Figure 4.2, as the many‐hubs view.
Advocates of the many‐hubs view often additionally propose that different conver- gence zones may be especially important for the representation of different semantic categories. For instance, conceptual knowledge about tools may depend critically on the ability to retrieve information about their praxis from their visual appearance or their names. Knowledge of other categories, such as animals or buildings, may involve little information about praxis. Consequently the convergence zone that maps from visual images and/or words to praxic representations may come to predominantly encode representations of tools (Mahon, Schwarzbach, and Caramazza, 2010). Other regions might come to predominantly encode representations of animals, due to their reliance on motion and perceptual properties (Gainotti et al., 1995; Warrington and Shallice, 1984), or unique entities like Jennifer Anniston or the Eiffel Tower, due to their association with specific contexts and episodes encoded in medial temporal regions (Grabowski et al., 2001). Thus the many‐hubs view often goes hand in hand with the hypothesis that the cortical semantic network is partly organized by semantic class, with different cortical regions preferentially involved in the representations of particular concepts.
Evidence for multiple domain‐specific hubs stems partly from lesion‐overlap studies across groups of patients exhibiting apparent category‐specific loss of semantic knowledge. Gainotti (2000) has suggested that greater impairment of knowledge for animals/living things is associated with damage to ventral/inferior temporal cortex bilaterally, all along the rostral–caudal length, though with a greater likelihood of pathology in the left hemisphere. In contrast, a greater impairment of knowledge for
Unique hub Animal hub Tool hub
Figure 4.2 Schematic illustrating the many‐hubs view. The shading indicates the same modality‐specific subnetworks as Figure 4.1. In addition, possible locations of category‐specific hubs are shown, including a hub for unique entities in the temporal poles (orange), for animal representations in ventral anterior temporal cortex (red), and for tools/manmade objects in the left posterior MTG/OTP junction (green).
manmade objects/tools was likely to be associated with wide‐ranging left‐hemisphere pathology in the territory of the middle cerebral artery – that is, across frontoparietal regions and curling down around the occipito‐temporal‐parietal (OTP) junction to the posterior MTG. Other lesion‐symptom analyses have fleshed out these observa- tions, suggesting that loss of knowledge about individual people is correlated with pathology in the temporal poles; that loss of knowledge about animals is correlated with specific pathology in the anterior inferior temporal gyrus, the posterior fusiform, and parts of early visual cortex; and that loss of knowledge about tools/manmade objects is correlated with pathology in the vicinity of the OTP junction and posterior MTG (Campanella et al., 2010; Damasio et al., 1996; Rudrauf et al., 2008).
Support for the many‐hubs view also stems from functional neuroimaging. For in- stance, Damasio et al. (1996) used PET to measure functional activation elicited by naming pictures of people, animals, and manmade objects, and compared this to the results of a lesion‐symptom correlation analysis for the same categories. The imaging analysis both validated and extended the lesion‐symptom analysis: greater activation was observed in the left temporal pole for naming people, in the posterior MTG for naming manmade objects, and in the ventral occipitotemporal regions bilaterally for naming animals. The imaging also identified additional category‐specific patterns not found in the lesion‐symptom analysis, including a region in the anterior part of the left inferior temporal gyrus that appeared to respond more strongly to animals, and another region stretching down inferiorly from the posterior MTG that appeared to respond more strongly to artifacts.
The finding of tool/artifact selective responding in posterior MTG has been reported by several groups, as has the finding of greater activation in the temporal pole for person recognition, relative to object recognition (Gauthier et al., 1997;
Gorno‐Tempini et al., 2001; Gorno‐Tempini and Price, 2001). Reports of animal‐
selective responding in neuroimaging studies have been somewhat more variable.
Several authors have found greater activation for tools/artifacts than for animals/
living things in the medial posterior fusiform, and the reverse effect in the lateral aspect of the same region, in studies using both visual images and words (see Chouinard and Goodale, 2010, for review). Others have suggested, from lesion‐
overlap and functional imaging data, that animal concepts depend preferentially upon aspects of the ventral anterior temporal lobe (ATL), though different studies place the important region either laterally (Tranel, Damasio, and Damasio, 1997) or medially (Moss et al., 2005; Noppeney et al., 2007) in the left hemisphere, or even preferen- tially in the right hemisphere (Brambati et al., 2006). Several studies have also failed to find any category‐specific responding, an issue to which we return below.
The single‐hub view: organization by a bilateral domain‐general convergence zone
The third view proposes that communication amongst various sensory, motor, linguistic, and affective representations is mediated, at least in part, by a single domain‐
general cross‐modal “hub” situated bilaterally in the ATL (Patterson, Nestor, and Rogers, 2007). The hub is assumed to contribute to semantic processing for all kinds of concepts and all receptive and expressive modalities, though there may also exist other more direct pathways between various surface representations. This view is illustrated in Figure 4.3.
The single‐hub view originated with the study of a neurological syndrome called semantic dementia (SD), a progressive dementing illness that is unique in two respects (see Chapter 20). First, it produces a gradual but remarkably selective dissolution of conceptual knowledge that affects all conceptual domains and all modalities of testing (Hodges et al., 1999). Though such patients typically present with marked anomia and verbal comprehension impairment, their deficits are not purely verbal: they show serious problems discriminating real from chimeric objects (Rogers et al., 2003), recalling the characteristic colors of familiar objects (Rogers, Patterson, and Graham, 2007), matching both words and pictures with their characteristic sounds (Bozeat et al., 2000), sorting words or pictures into semantic categories (Hodges, Graham, and Patterson, 1995), retrieving knowledge about the characteristic praxis, function, and functional associates of common manipulable objects (Hodges et al., 2000), and even recognizing odors (Luzzi et al., 2007). Yet these often profound impairments coexist with otherwise relatively intact cognitive functioning: good performance on tests of episodic memory that do not depend upon intact semantics (such as the delayed Rey figure copy); intact working memory as assessed by forward and backward digit span; largely normal perception and attention; speech that, apart from word‐
finding difficulties, is fluent and grammatical; and good performance on tests of reasoning and problem solving such as the Raven’s progressive matrices (Hodges et al., 1999; Rogers et al., 2006). Patients with SD thus represent the purest form of semantic/conceptual impairment on record.
The second remarkable aspect of SD is the consistency of its neuropathology, which invariably affects the anterior temporal cortices, especially in their lateral and inferior aspects (Acosta‐Cabronero et al., 2011; Mummery et al., 2000). In about two‐thirds of the patients, the structural pathology appears to be more pronounced in the left hemisphere, but both hemispheres are hypometabolic in the disorder. The profound semantic impairment is thought to arise from this bilateral pathology, since unilateral damage in similar regions produces much milder patterns of impairment (Schapiro et al., 2013).
Corroborating evidence has amassed in recent years from other methods in cognitive neuroscience. One early puzzle for the single‐hub view was that the neuropsychological Figure 4.3 Schematic illustrating the single‐hub view. The shading indicates the same modality‐specific subnetworks as Figure 4.1. The purple region indicates the hypothesized semantic hub, situated on the ventral surface of the anterior temporal cortex bilaterally. Red arrows indicate incoming visual information.
evidence did not appear to align well with evidence from functional neuroimaging.
One reason is likely the predominance of functional magnetic resonance imaging (fMRI) in these studies. fMRI yields especially noisy signal in parts of the brain near the air‐filled sinuses, especially the inferior surface of the ATL and the orbitofrontal cortex. Other methods, most notably PET, do not encounter the same problem, and such studies are much more likely to show inferior ATL activation associated with semantic task performance (for review see Visser, Jefferies, and Lambon Ralph, 2010).
Recent technical innovations in fMRI have improved the signal‐to‐noise ratio in problem regions, and where these methods have been applied researchers have observed robust activation in inferior ATL regions associated with semantic task performance (Binney et al., 2010). Thus as the technical limitations of fMRI are being overcome, the literature is coming into better alignment with the neuropsycho- logical evidence.
Other emerging methods have likewise lent support to the single‐hub theory. For instance, Chan and colleagues (2011), using intracranial multi‐electrode recording in human participants, observed category‐selective patterns of activity in ventral anterior temporal regions that differentiated animals from manmade objects across different tasks and modalities. Such results provide strong evidence that ATL regions extract important cross‐modal semantic regularities. Studies employing TMS have further suggested that these evoked patterns of activation are not epiphenomenal, but play a causal role in the generation of responses in semantic tasks. Thus unilateral TMS stim- ulation to the ATL slows performance in tasks that rely on word meaning, such as synonym judgment, but not in comparably difficult tasks that do not rely on word meaning, such as numerical magnitude estimation (Pobric, Jefferies, and Lambon Ralph, 2007). Such effects are observed whether the stimulation is applied to the left or the right ATL, consistent with the view that the hub is bilaterally distributed; and it disrupts performance for living things, manmade objects, and abstract concepts, consistent with the view that the hub supports domain‐general processing. This pattern stands in contrast to stimulation of other brain regions such as the left inferior parietal lobe, which can produce category‐specific patterns of slowing (Pobric, Jefferies, and Lambon Ralph, 2010).
Finally, simulations with computer models have suggested why the cortical semantic system might adopt such an architecture (Rogers, et al. 2004a; Rogers and McClelland, 2004). The key insight is that the various different kinds of sensory, motor, linguistic, and affective representations that constitute important components of the semantic network do not, by and large, capture the core similarities that structure conceptual representations. Items that are similar in kind can nevertheless vary quite substantially in many of their surface details. Light bulbs and pears are similar in shape but are quite different kinds of things; ostriches and hummingbirds differ dramatically in size and in the way that the move, but nevertheless are conceived as quite similar kinds of things; tape and glue do not look similar, nor do they engage similar praxis in their use, but nevertheless we think of them as being similar kinds of things. Such examples seem to indicate that, in addition to representations of the particular surface prop- erties associated with familiar kinds, the semantic system must represent conceptual similarity structure that is not directly reflected in any single representational modality.
The single‐hub theory proposes that the functional role of the ATL is to extract this similarity structure. The central idea is that, although conceptual similarity structure may not be perfectly represented in any single representational modality taken on its