retrieval cue or with respect to its online significance. It is currently unclear how this and related ideas about retrieval processing (Rugg and Wilding, 2000) should be incorporated into the framework outlined in Figure 5.1.
3 The framework outlined in Figure 5.1 implies that retrieval consists of little more than the “replaying” of the processing engaged by the original experience. If this were so, then recollection would be “all or nothing”; either everything that was registered in the brain as an event unfolded would be retrieved, or nothing would be. Moreover, memories would be largely veridical. Clearly, neither of these sce- narios is accurate. Memories are a very imperfect mirror of experience: they are invariably partial, and often highly distorted, records of the original event (e.g., Bartlett, 1932; Loftus and Palmer, 1974; Schacter, 2002; see also Chapter 8).
Among the many factors contributing to this imperfect relation between an event and our later memory of it, two stand out. First, the different features of an event are not equally likely to be successfully encoded. Other things being equal, those aspects of the event that are attended to the most fully are most likely to be later remembered (see below, and Moscovitch, 1992). Second, there is a wealth of evi- dence that episodic retrieval is a constructive process, in which retrieved information is combined with other knowledge about the event and the result interpreted in light of current expectations and biases (e.g., Bransford and Franks, 1971; Brewer, 1987; Brewer and Treyens, 1981; Schacter, Norman, and Koutstaal, 1998). Thus, retrieved episodic information may only partially determine the content of the resulting memory representation. Together, these two factors will act to reduce the amount of overlap between encoding‐ and retrieval‐related neural activity to well below the 100% illustrated in Figure 5.1. Specifically, as we discuss below, cortical activity elicited during an event by attended information is more likely to be incorporated into the resulting hippocampal representation than is the activity elicited by unattended information. And the constructive nature of memory retrieval means that retrieval‐related neural activity will reflect not only the reinstatement of activity elicited at encoding, but also any non‐episodic information that was incorporated into the memory representation.
Empirical findings
to other cognitive operations also modulated by the task manipulation. A second problem is that in task‐based designs, encoding is manipulated through qualitative changes in the cognitive processes engaged by study tasks. Thus, task‐based designs cannot address the question of what modulates the efficacy of encoding of stimulus events that seemingly engage the same cognitive processes as they are experienced.
An alternative approach to the study of encoding is to identify brain regions where within‐task item‐related activity predicts successful memory on a later retrieval test.
Such “subsequent memory” comparisons, which were originally developed in the context of event‐related potential (ERP) research (Rugg et al., 1995; Sanquist et al., 1980), identify activity at the time of encoding that is in some sense “predictive” of later memory performance (such activity is referred to as a “subsequent memory effect”). The subsequent memory procedure is now near‐ubiquitous in fMRI studies of memory encoding.
The logic of the subsequent memory procedure is straightforward: (1) brain activity elicited by a series of study items is acquired; (2) memory for the study items is later tested; (3) the brain activity that was elicited at study is “backsorted” according to performance on the subsequent memory test. Differences in the activity elicited by subsequently remembered and subsequently forgotten items are taken to be corre- lates of differences in the efficacy of encoding, and regions exhibiting these differ- ences are regarded as candidate loci of operations supporting successful encoding.
These differences can take one of two forms: contrasts can be employed that identify regions where the activity elicited by study items that go on to be successfully remem- bered is greater than it is for items that subsequently fail to be remembered. The result of such a contrast is what is typically meant by the term subsequent memory effect. Here, we also use the term “positive” subsequent memory effect, to distin- guish these effects from “negative” subsequent memory effects, when remembered study items elicit less activity than items that fail to be accurately remembered.
Positive subsequent memory effects
As was discussed above, according to the TAP principle episodic encoding is a
“byproduct” of online processing. Therefore, no cognitive system is expected to be uniquely dedicated to memory encoding, and encoding‐related cortical activity should be evident in the same regions that support the cognitive operations engaged during a given study task. In other words, cortical subsequent memory effects should vary in their locations according to the processes engaged during study. By contrast, given the centrality of the role of the hippocampus in capturing the patterns of cortical activity elicited by events as they are experienced, hippocampal subsequent memory effects should be evident regardless of the nature of the study processing. As described below, fMRI evidence supports these two predictions.
In a study by Otten and Rugg (2001), for example, subjects were cued on a trial‐by‐trial basis to make either a semantic (living/nonliving) or a phonological (odd/even number of syllables) judgment on a series of study words. For both tasks, items that were recognized with high confidence on a memory test undertaken some 15 minutes later elicited greater cortical activity during study than items that failed to be recog- nized. As is illustrated in Figure 5.2, the loci of these subsequent memory effects differed according to the study task. Whereas the effects associated with the semantic task were found primarily in medial and left inferior frontal cortex, effects for the
syllable task predominated in posterior cortical regions (see Otten, Henson, and Rugg, 2002; and Park, Uncapher, and Rugg, 2008, for similar findings). Importantly, and in accordance with the TAP principle, the subsequent memory effects overlapped with regions selectively engaged by each study task (as identified by contrasting the activity elicited by all items belonging to each task).
Evidence in support of the TAP principle is not confined to experiments where study processing was varied through a task manipulation. For example, Gottlieb and colleagues investigated the subsequent memory effects elicited during the encoding of pairs of study items that comprised a picture of an object and its name, presented either visually or auditorily (Gottlieb, Uncapher, and Rugg, 2010; see also Gottlieb et al., 2012). The requirement on the subsequent memory test was to discriminate between studied and unstudied pictures and, for each picture judged as studied, to indicate whether its name had been presented visually or auditorily. It was thus pos- sible to contrast the activity elicited on auditory and visual study trials according to whether successful recognition of the picture was accompanied by accurate memory for the modality of its name. Unsurprisingly, contrasts between the two classes of study trial revealed widespread regions where activity was either visually or auditorily selective. Crucially, modality‐selective subsequent memory effects were evident in a subset of the same regions (Figure 5.3). As is evident from the figure, visually selective subsequent effects were identified in ventral temporal cortex whereas auditorily selective effects were evident in the superior temporal sulcus. Analogous findings were reported by Park and Rugg (2011) for the encoding of word–word and picture–
picture associations.
Other findings in support of the TAP principle come from studies reporting dissociable subsequent memory effects for the different contextual features of a single study episode. In Uncapher, Otten, and Rugg (2006), subjects were scanned while they studied words presented in one of four possible locations and in one of four possible colors. A subsequent memory test required the subjects to discriminate studied from unstudied items and, for each recognized item, to recall the color and location in which it had been studied. Using as a baseline the study activity elicited by items that were later recognized but for which neither color nor location information were correctly recalled (“recognition‐only” items), we investigated whether successful retrieval of either of the two contextual features was associated with distinct subsequent memory effects. In addition, we asked whether there were any brain regions where subsequent memory effects were specific for study items for which both features were later retrieved. The key findings are illustrated in Figure 5.4. Relative to items attracting recognition‐only judgments, study items for which location was accurately
Animacy Syllable
figure 5.2 Data from Otten and Rugg (2001) illustrating the subsequent memory effects associated with words subjected to animacy judgments (left) and syllable judgments (right).
remembered elicited additional activity in retrosplenial cortex, whereas items whose color was remembered elicited greater study activity in posterior inferior temporal cortex. These regions have been implicated previously in the processing of location information and color knowledge respectively (Chao and Martin, 1999; Frings et al., 2006; Kellenbach, Brett, and Patterson, 2001).
Figure 5.4 also illustrates a region – the right intraparietal sulcus (IPS) – where subsequent memory effects were uniquely associated with items for which both features were remembered on the later test. These effects indicate that conjoint encoding of the two features was not simply the result of the concurrent engagement of processes supporting memory for each feature alone. Rather, it appears that conjoint encoding engaged processes that operated across color and location information, perhaps binding the two features into a single perceptual representation (Cusack, 2005; Humphreys, 1998; for review, see Robertson, 2003). Uncapher, Otten, and Rugg (2006) suggested that the IPS subsequent memory effect in Figure 5.4 might reflect the benefit of allocating attention at the “object level” rather than to individual features of a stimulus event, allowing the features to be integrated in a unified perceptual representation, and hence more likely to be incorporated into a cohesive memory representation.
In a follow‐up study we investigated whether the encoding of a contextual feature is sensitive to whether it is explicitly attended to during study (Uncapher and Rugg, 2009).
Subjects were required to selectively attend either to color or to location information
Visual>Auditory
Auditory>Visual Auditory sub mem
Visual sub mem
figure 5.3 Data from Gottlieb, Uncapher, and Rugg (2010). Left: outcome of the contrasts between study trials containing auditorily or visually presented words. Right: regions where subsequent memory effects for auditory (upper) or visual (lower) study trials overlapped with the relevant contrast between the two classes of study trial.
(so as to detect infrequent “targets” defined by a specific color or location, depending on the task) while incidentally encoding pictures of objects. If, as we had previously argued (Uncapher, Otten, and Rugg, 2006), feature‐selective subsequent memory effects reflect the benefit to encoding that accrues when a feature is strongly attended, the effects should be greater when the feature was task‐relevant (and hence attended) than when it was not. As is illustrated in Figure 5.5, this prediction was borne out.
As was outlined earlier, the hippocampus is held to be responsible for encoding the patterns of cortical activity engaged during the online processing of an event. Since attention modulates both the magnitude of cortical activity (Corbetta et al., 1990) and the likelihood of later successful memory (e.g., Chun and Turk‐Browne, 2007), we conjectured that hippocampal activity, like activity in the cortex, should be atten- tionally modulated. As can be seen in Figure 5.5, a region in the hippocampus did indeed respond selectively to the attended rather than the unattended feature. This finding suggests that one mechanism by which attention influences encoding is through the modulation of cortical–hippocampal interactions.
The findings reviewed in this section provide strong evidence for a central tenet of TAP – namely, that encoding is a “byproduct” of online processing. The findings sug- gest that the distinction between processes supporting the online processing of an event, and those supporting its encoding into memory, are not honored at the cortical level. Rather, cortical subsequent memory effects appear to reflect the amount of attentional resources allocated to (or, perhaps, captured by) a particular feature of an
Color Location
Retrosplenial cortex Inferior temporal
cortex
Both
figure 5.4 Data from Uncapher, Otten, and Rugg (2006) illustrating feature‐selective and multi‐featural subsequent memory effects. Top panel: subsequent memory effects (relative to correctly recognized study words for which no contextual feature could be accurately recalled) associated with accurate memory for the location (left) and color (right) of the study word.
Bottom panel: subsequent memory effects in the right intraparietal sulcus (IPS) uniquely associated with accurate memory for both color and location.
event and, consequently, the likelihood that the cortical activity engaged by the processing of that feature will form part of a hippocampally mediated, offline repre- sentation of the event.
Negative subsequent memory effects
As was noted earlier, “positive” subsequent memory effects can be accompanied by so‐called “negative” effects, when later forgotten study events elicit higher activity than do events that are later remembered (Otten and Rugg, 2001; Wagner and Davachi, 2001). Negative effects are typically found in regions belonging to what is often referred to as the “default mode network”: posterior and frontal midline cortex, along with inferior parietal and dorsolateral prefrontal regions. The default mode network is so named because its constituent regions exhibit higher activity during task‐independent periods (sometimes referred to as a “resting” state) than when an externally imposed task is being performed (Shulman et al., 1997; for reviews, see Buckner, Andrews‐Hanna, and Schacter, 2008; and Gusnard and Raichle, 2001).
From the time they were first identified, it has been proposed that negative subsequent memory effects reflect competition for cognitive resources between study events and elements of the internal and external environment, such as task‐irrelevant thoughts or stimulus features (e.g., Wagner and Davachi, 2001). Such a view is in keeping with the notion that default mode activity reflects the allocation of processing resources toward internal cognitive events at the expense of events originating from the environment (such as the study events in an encoding experiment). Relatively
Color encoding
3.5 2.5 1.5 0.5
2.0 1.6 1.2 0.8 0.4 0.0
Attend color
Attend color
Attend location
Attend location
Color Location Color Location
Attend color Attend
location
Location encoding
figure 5.5 Data from Uncapher and Rugg (2009). Upper: regions where color‐ (left) and location‐selective (right) subsequent memory effects were modulated by the direction of attention. Lower: right hippocampal region where color and location subsequent memory effects were reciprocally modulated by attention.
little research, however, has been directed toward a more mechanistic understanding of these effects, or to attempts to dissociate them anatomically.
Cabeza (2008) suggested that negative subsequent memory effects may reflect an encoding impairment caused by the reflexive (“bottom‐up”) capture of attention by task‐irrelevant information. This suggestion received support from a review of studies reporting subsequent memory effects in lateral parietal cortex (Uncapher and Wagner, 2009), a region heavily implicated in the control of attention (Corbetta, Patel, and Shulman, 2008). Uncapher and Wagner reported that negative subsequent memory effects are localized exclusively to ventral parietal cortex, with the majority of the effects falling in the vicinity of the temporoparietal junction (TPJ), a region held to be a key component of a network mediating the bottom‐up capture of attention (Corbetta, Patel, and Shulman, 2008). Negative subsequent memory effects were also evident, however, in the adjacent angular gyrus, a component of the default mode network (Buckner, Andrews‐Hanna, and Schacter, 2008). These observations suggest that negative subsequent memory effects are unlikely to reflect the modulation of a single, functionally homogeneous network. Rather, the loci of the effects suggest that there are at least two mechanisms by which encoding can be impaired: attentional capture and the failure to disengage fully from “default” (task‐independent) cognitive processing.
In a test of the attentional capture hypothesis, Uncapher, Hutchinson, and Wagner (2011) manipulated top‐down and bottom‐up attention within a subsequent memory paradigm using a Posner cueing procedure. Subjects were cued on a trial‐by‐trial basis to focus their attention on one of two locations, in anticipation of an upcoming study item. The majority of the items (82%) appeared in the cued location, forcing items appearing in the uncued location to gain attention through a “bottom‐up” re‐orienting process (Posner, 1980). Thus, bottom‐up attention effects could be identified by contrasting activity elicited by objects appearing in uncued relative to cued locations, a contrast that revealed robust effects in the TPJ bilaterally. Crucially, negative subsequent memory effects associated with correctly cued study items overlapped these attentional effects, strongly supporting the proposal that negative subsequent memory effects in this region reflect impairment of encoding arising from the capture of attention by task‐
irrelevant input. The nature of this input – for example, whether it can take the form of both internal and external events – remains to be characterized.
Retrieval
In this section we review evidence for the existence of a network of “core” regions where activity is enhanced when a retrieval cue elicits recollection of a prior event.
We then go on to discuss studies addressing the question of whether any neural cor- relates of recollection are content‐sensitive and whether, as would be predicted from the theoretical framework outlined previously, these correlates reflect rein- statement of activity elicited during encoding. We focus on effects associated with successful recollection, making only passing reference to work on retrieval cue or post‐retrieval processing (Rugg and Wilding, 2000). Thus, we concentrate on studies contrasting activity elicited by retrieval cues (mainly, but not exclusively, recognition memory test items) according to whether the cues were or were not associated with recollection of details of the study episode, as this is operationalized either by successful retrieval of contextual (source) information or by phenomenal report (a “remember” judgment).
As in the case of studies of encoding, the earliest functional neuroimaging studies of retrieval (see Rugg, Otten, and Henson, 2002, for review) employed blocked designs and hence did not directly contrast activity elicited by recollected and unrec- ollected test items. Two of the earliest event‐related studies were those of Henson et al. (1999) and Eldridge et al. (2000). In both studies the authors employed the remember/know (R/K) test procedure and contrasted the activity elicited by cor- rectly recognized items according to whether the items were endorsed as recollected or as familiar only. Among the regions where activity was enhanced for recollected items were the hippocampus, the posterior cingulate, and lateral parietal cortex.
Numerous subsequent studies employing either variants of the R/K procedure or tests of source memory reported similar results, and also identified regions additional to those reported in the early studies. Although the findings across individual studies differ in detail, the consensus is that successful recollection is consistently associated with enhanced activity in the MTL, especially the hippocampus and parahippocampal cortex, along with retrosplenial, posterior cingulate, medial prefrontal, and ventral posterior parietal cortex (angular gyrus) (Figure 5.6).
We (Hayama, Vilberg, and Rugg, 2012; Johnson and Rugg, 2007; Rugg and Vilberg, 2013) have proposed that these regions form part of a “core” recollection network engaged when recollection is successful regardless of how memory is cued, or the nature of the recollected content. Consistent with this proposal, the same regions identified as recollection‐sensitive in studies employing variants of recognition memory tests have also been identified in studies of cued recall (e.g., Hayama, Vilberg, and Rugg, 2012; Okada, Vilberg, and Rugg, 2012; Schott et al., 2005), a memory test held to rely heavily on recollection. In a recent study from our laboratory, cued recall and source memory tests were employed in combination (Hayama, Vilberg, and Rugg, 2012). Subjects first studied a series of words presented on the left or right side of a display monitor. Test items comprised three‐letter word stems, with the requirement
Angular gyrus Retrosplenial/posterior cingulate cortex
Medial prefrontal cortex
Hippocampus Parahippocampal
cortex
figure 5.6 The putative core recollection network. The outcome of the contrast between correctly recognized test words endorsed as “remember” or “know” in an unpublished study (n = 19) by Wang and Rugg. The words had been studied either as pictures or as words in the context of two different encoding tasks. Illustrated are regions where recollection was associated with enhanced activity at test regardless of the encoding condition.