The Representational Similarity between Visual Perception and Recent Perceptual History
- Integrative Neuroscience and Cognition Center, Université Paris Cité and Centre Nationale de la Recherche Scientifique, Paris 75006, France
-
Author contributions: T.C. designed research; J.L. performed research; J.L. and T.C. analyzed data; T.C. wrote the first draft of the paper; J.L. and T.C. edited the paper; T.C. wrote the paper.
Abstract
From moment to moment, the visual properties of objects in the world fluctuate because of external factors like ambient lighting, occlusion and eye movements, and internal (proximal) noise. Despite this variability in the incoming information, our perception is stable. Serial dependence, the behavioral attraction of current perceptual responses toward previously seen stimuli, may reveal a mechanism underlying stability: a spatiotemporally tuned operator that smooths over spurious fluctuations. The current study examined the neural underpinnings of serial dependence by recording the electroencephalographic (EEG) brain response of female and male human observers to prototypical objects (faces, cars, and houses) and morphs that mixed properties of two prototypes. Behavior was biased toward previously seen objects. Representational similarity analysis (RSA) revealed that responses evoked by visual objects contained information about the previous stimulus. The trace of previous representations in the response to the current object occurred immediately on object appearance, suggesting that serial dependence arises from a brain state or set that precedes processing of new input. However, the brain response to current visual objects was not representationally similar to the trace they leave on subsequent object representations. These results reveal that while past stimulus history influences current representations, this influence does not imply a shared neural code between the previous trial (memory) and the current trial (perception).
SIGNIFICANCE STATEMENT The perception of visual objects is pulled toward instances of that object seen in the recent past. The neural underpinnings of this serial dependence remain to be fully investigated. The present study examined electroencephalographic (EEG) responses to faces, cars, and houses, and ambiguous between-category morphs. With representational similarity analysis (RSA), we showed (1) object-specific neural patterns that differentiate the three categories; (2) that the response to the current object contains information about the previous object, mirroring behavioral serial dependence; (3) that the object-specific neural pattern about the past was different from that in the current response, revealing that while past stimulus history influences current representations, this does not imply a shared neural code between the previous trial (memory) and the current trial (perception).
Introduction
One of the fundamental tenets of cognitive psychology is the idea that mental activity and states involve representations. Representations are internal codes for information, and their content and intrinsic value are causally related to behavior and subjective experience (Baker et al., 2022; Cleeremans and Tallon-Baudry, 2022). The question of the nature of mental representations is thus one of the crucial questions of cognitive psychology. Behavioral measures such as reaction time are an indirect measure of representations insofar as RT differences may result from specific properties of or differences between the representations involved in different experimental conditions. A data analysis technique known as representational similarity analysis (RSA) enables psychologists to characterize the information carried by a brain representation, and to compare representations across conditions (Kriegeskorte et al., 2008). RSA provides a more direct measure of psychological representations. Of course, RSA has its limitations in that it is constrained by the signal-to-noise ratio and is by essence correlational. However, it has the advantage of avoiding the correspondence problem when investigating the brain correlates of psychological models because it does not require specific mappings between brain activity units (voxels, electrodes, clusters) and units in a psychological model. Rather, RSA looks at whole-brain activity and quantifies the similarity between different conditions. Similarities or dissimilarities are informative about shared or modular representations. For example, studies using RSA have been successful in demonstrating the hierarchical structure of visual object representations (Kiani et al., 2007) and between-participant proximity of different emotional representations (Nummenmaa et al., 2012). In the current study, we show that RSA captures a unique feature of visual object representations: their dependence on past stimulus history.
Current perception depends on what was experienced in the immediate past. History effects are ubiquitous in the psychology of perception, and interest in a particular history effect, serial dependence, has been rekindled in the last decade (Cicchini et al., 2014; Fischer and Whitney, 2014). Serial dependence is the attraction of current perception toward instances of a stimulus seen in the recent past. It occurs across a wide range of stimuli, from low-level random dot patterns (Cicchini et al., 2014) and Gabor patches (Fischer and Whitney, 2014) to high-level visual objects and attributes such as shape (Manassi et al., 2019; Collins, 2022), facial identity and emotional expression (Liberman et al., 2014, 2018; Collins, 2021), and even facial attractiveness (Taubert et al., 2016).
Here, we provide evidence for the representational similarity between visual objects that share a common recent past, relative to visual objects that do not have a common recent past. Our results demonstrate that the recent past is integrated into current visual object representations, and may thus contribute to perceptual experience. However, the trace of past representations in current responses is not a recurrence of the past representation, because they are representationally dissimilar to the response to current objects. These results provide a first step toward understanding the neural correlates of serial dependence.
Materials and Methods
Participants
A total of 26 participants (18 women; mean age 26 years old, range 19–41; 24 right-handed; all with self-reported normal vision and no neurologic disorders) were recruited from (1) the cohort of subjects who have participated in EEG studies before, maintained by the host laboratory, and (2) a national subject pool maintained by the RISC (Relais d'Information sur les Sciences de la Cognition). All participants signed a consent form and were compensated 15€/h, as per laboratory practice. The experiment was approved by the French national ethics board (Comité de Protection des Personnes).
Stimuli
Photographs of a face, a car and a house were selected as prototypes (Fig. 1). The face prototype was taken from the Radboud faces database (Langner et al., 2010), car and house prototypes were found online. All prototypes were in black and white, and their contrast level was equalized. A total of 48 morphs were then created between each category, for a total of 147 images (48 × 3 morphs + 3 prototypes). Morphs were created by using Sqirlz Morph 2.0 software. In order to perform the morphing, control points were defined manually and matched to corresponding contours between two prototypes (e.g., control points surrounding car lights were mapped to control points surrounding the eyes on the face prototype). After control point definition, the software generated the series of morphs with different proportions of characteristics from each prototype. For example, a morph could contain 50% car-like and 50% face-like features, generated from the car and face prototypes. Such a morph is likely to be highly ambiguous. Image size was 9 by 6 degrees of visual angle (dva) and the edges were blurred by superimposing a Gaussian patch (SD = 4 dva). Images were presented behind semi-transparent pixel-noise (a random value from 0, black, to 1, white was selected for each pixel). Stimuli presentation was controlled with Psychtoolbox for MATLAB (MathWorks; Brainard, 1997; Pelli, 1997; Kleiner et al., 2007).
Stimuli. Three prototypes (face, car, house) and some example intermediate morphs. The wedges correspond to categories defined as ambiguous (light gray) or unambiguous (dark gray); these were defined individually such that the exact position and size of the wedges depends on participant.
Apparatus
Stimuli were back-projected by a ProPixx projector (VPixx Technologies) onto a wide screen (107 × 60 dva) with a resolution of 1920 × 1080 pixels and a refresh rate of 120 Hz. Participants sat ∼1 m from the screen in an otherwise dark room. Viewing was binocular. Movements of the right eye were monitored with an Eyelink 1k (SR Research) at 1000 Hz sampling rate. At the beginning of a session, the Eyelink was calibrated with the standard five-point Eyelink procedure. Before each trial, fixation was checked, and if the measured value was >3 dva, a new calibration was initiated. Calibration was also automatically renewed every 100 trials. Participants were required to maintain their gaze within a tolerance window of 4 dva from the center of the visual images. If their gaze deviated from this window, the trial was aborted and rerun at the end of the experiment, and a warning was shown to encourage fixation.
Procedure
Participants viewed one image at a time for 300 ms, and pressed on one of three keyboard keys to report the category. The correspondence between keys and categories was counterbalanced between participants. There were 1617 trials (11 repetitions of each of the 147 morphs), except for the following participants whose total number of trials differed for technical reasons: participant 1 (1176 trials), participant 2 (1582 trials), participant 8 (1598 trials), and participant 9 (1612 trials). An entire experimental session took ∼120 min (including EEG set-up).
EEG recording and preprocessing
EEG data were recorded with 128 Ag/AgC1 electrodes mounted on a fabric cap and amplified by an ActiCHamp amplifier (Brain Products). The sampling rate of the EEG recording was set to 1000 Hz. Electrodes were arranged according to the international 10–20 system. The right mastoid (M2) was used as the online reference. Preprocessing of the EEG data were performed with the MATLAB-based FieldTrip Toolbox (Oostenveld et al., 2011). The following processing steps were applied: high-pass filtering at 0.4 Hz and stop-band at 50 Hz with deletion of the linear component, performed on the continuous signal; visual detection of electrodes with poor signal and reconstruction by interpolation of neighboring electrodes; epoching around the onsets of the morph image, from −500 to +1000 ms; re-referencing to the average of the electrodes; baseline correction over the 500 ms preceding the onset of the morph image.
Data analysis
For all analyses (behavioral and RSA), we divided object images into six categories, as illustrated in Figure 1: unambiguous faces, unambiguous cars, unambiguous houses, ambiguous face-cars, ambiguous car-houses and ambiguous house-faces. Ambiguous versus unambiguous labels were determined individually as follows. Performance was quantified by psychometric functions fit to each of the object pairs (face-house, face-car and car-house), using the psignifit toolbox for MATLAB which implements the maximum-likelihood method described by Wichmann and Hill (2001). Points of subjective equality (PSE) correspond to the morph image that looks as much like one prototype as the other. Individual unambiguous categories were bounded by the morph values corresponding to 10% and 90% on each of the three morph continua (face-car, car-house, and house-car). For example, on the face-car continuum, all morph images to which a participant responded “face” >90% of the time were labeled as an unambiguous face, and all morph images to which a participant responded “car” >90% of the time were labeled as an unambiguous car. Morph values in the middle (giving rise to 45–55% of a particular response) were labeled ambiguous. The ambiguous and unambiguous categories are illustrated in Figure 1 as light and dark gray wedges; the size of each wedge could differ between participants.
Serial dependence was quantified by the proportion of responses to ambiguous morphs coherent with the previous morph, when the previous morph was unambiguous. For example, the number of ambiguous face-car morphs that were categorized as a face when they were preceded by an unambiguous face, relative to the total number of ambiguous face-car morphs that were categorized as a face (whatever the preceding unambiguous stimulus). This definition of serial dependence corresponds to the classic characteristics reported in the literature: uncertain stimuli are most influenced by serial dependence (Cicchini et al., 2018; van Bergen and Jehee, 2019; Gallagher and Benton, 2022), unambiguous stimuli exert the largest effect on subsequent stimuli (Cicchini et al., 2018), and the greatest interaction occurs between objects nearby in feature space (Fischer and Whitney, 2014). If the total number of unambiguous faces, car and houses were equivalent, then if the preceding morph has no effect on the current response, this proportion should be 33%. However, because the number of unambiguous objects was defined individually (see above), the relevant reference level may not be 33%. A baseline was thus estimated by calculating the average percent serial dependence for 25–50 trials in the past, and 1–50 trials in the future. At these distances in the past and future, no serial dependence is expected (Collins, 2021), but the proportion of unambiguous trials is equivalent to the one-back trial. For ease of interpretation, baseline was subtracted from the raw values: no serial dependence is thus 0%, positive values correspond to serial dependence, and negative values correspond to a repulsive effect.
To quantify the extent to which past stimuli influenced the current brain representation, we performed representational similarity analysis (RSA) on concurrently measured electroencephalographic (EEG) data. RSA quantifies the correlation between the spatial patterns of activity evoked in different experimental conditions (Kriegeskorte, 2008). Here, patterns correspond to the voltage recorded at each electrode (n = 127) and at each time point.
We performed three RSA. The first examined the representational similarity between currently viewed unambiguous faces, cars, and houses. The second examined the representational similarity between subsets of morphs preceded by the same versus different objects. The third examined representational similarity between currently viewed unambiguous objects and morphs preceded by that object.
We proceeded in four steps. First, we examined average similarity matrices. For each participant and each time point from stimulus onset to 1000 ms poststimulus, we computed Spearman rank correlations between patterns on a random subset of half of the trials for each pair of conditions. The number of trials was matched between subsets, owing to potential differences in the total number of unambiguous faces, cars, and houses because of the individual determination described above. We repeated this 10 times, each time taking a different random split of trials. By adopting such a bootstrapping procedure, we expected values on the diagonal of the representational similarity matrix to be high, but not strictly equivalent to 1. The correlations were averaged across all time points from stimulus onset to 1000 ms poststimulus. We expected the correlation between two subsets of same-object morphs (e.g., faces) to be higher than the correlation between a subset of one object (e.g., faces) and a subset of another object (e.g., cars or houses). Significance was ascertained by comparing each of the three off-diagonal correlations to the mean correlation on the diagonal with a one-tailed paired t test (α level = 0.025), Bonferroni corrected for multiple comparisons.
Second, we examined the time course from 250 ms before stimulus onset to 1000 ms after. The procedure was identical to the average similarity matrix except that we did not average over all time points. To ascertain significance, we used permutation tests with the false discovery rate (FDR) procedure described previously (Storey, 2002); significant time points are identified by a black dot on the abscissa of Figures 3 and 6. Our hypothesis was that a set of unambiguous face morphs would be more similar to another set of face morphs than to a set of car or house morphs (with corresponding hypotheses for car and house morphs), but we had no specific hypothesis about the time at which this category-specific effect would emerge.
Third, we examined which electrodes contributed the most to representational similarity, as a proxy for brain localization. A searchlight analysis was performed as follows. The correlation between conditions was calculated within clusters of nearby electrodes (n = 4–6) in 200-ms time windows from −200 to +800 ms around stimulus onset. As before, a bootstrap procedure was used on split-halves of trials from each condition. Thus, the correlation reported at each electrode in the topographies in Figures 3 and 6 was the average correlation including that electrode and the surrounding cluster.
Fourth, we examined to what extent the representation, as indexed by the distribution of activity across the scalp, correlated from one moment to the next. This analysis (temporal generalization) identifies when and for how long a particular piece of information is represented in the brain, and how the underlying neural code evolves with time (King and Dehaene, 2014). The activity between categories at time t is correlated with the activity between categories at all other time points; and the same is done for within-category correlations. By analyzing whether the neural representations that support above-chance correlation are stable or dynamically evolving, this reveals how mental representations unfold with time. (The third analysis described above is thus a special case of the temporal generalization matrix, i.e., the diagonal). When the representation observed at time t correlates with the representation occurring at time t', this suggests that the neural code that was identified at time t recurred at time t'. The width of the temporal generalization also carries information about the stability of brain representations across time.
Results
Currently seen ambiguous morphs were reported to look more like the previously seen unambiguous object than that which would be expected by chance alone. Average serial dependence was 6.0% (bootstrapped 95% confidence intervals, [2.0–10.1]). A t test confirmed that this amount of serial dependence, although small, was significantly different from null (t(25) = 2.8, p < 0.001).
To quantify the extent to which past stimuli influenced the current brain representation, we performed RSA on concurrently measured electroencephalographic (EEG) data. Figure 2 shows the representational similarity matrix within and across object categories. The top left cell (0.79 ± 0.02; mean ± SEM) corresponds to the correlation between subsets of unambiguous face morphs. The cell just beneath that (0.76 ± 0.02) corresponds to the correlation between a subset of unambiguous face morphs and a subset of unambiguous car morphs. All off-diagonal correlations were significantly lower than the average diagonal correlation of r = 0.80 ± 0.02 (car-face vs diagonal, t(25) = 4.47, corrected p < 0.0005; house-face vs diagonal, t(25) = 5.6, corrected p < 0.0001; house-car vs diagonal, t(25) = 7.1, corrected p < 0.0001).
Representational similarity matrix within and across categories. Correlations between pairs of conditions (face, car, house). Stronger correlations are in warmer colors. Stars correspond to off-diagonal correlations that are significantly different from the average diagonal correlation.
Figure 3 shows the time course. At each time point, the correlation within categories (in red) is compared with the correlation across categories (in blue). The results confirm the hypothesis that sets of unambiguous morphs from the same category are more similar to each other than to sets of morphs from a different category. This category-specific information emerged ∼200 ms after stimulus onset, and persisted for several hundred milliseconds.
Representational similarity time course within and across categories. Red curve, Correlation between subsets of same category objects. Blue curve, Correlation between subsets of objects from two different categories. Black curve, Difference between within-category and across-category conditions. Shaded areas around each curve correspond to 95% bootstrapped confidence intervals. Black dots along the abscissa correspond to time points for which the difference between the two colored curves was statistically significant. The topographies show the average correlation across six successive 200-ms time windows. Warm colors correspond to electrodes where the local difference between within and across category was high.
The topographies in Figure 3 show which electrodes contributed the most to the representational similarity in the results, and confirm a positive difference between within-and across-categories starting around 200 ms after stimulus onset. The category-specific activity appears mainly driven by occipito-parietal electrodes.
Figure 4 shows the temporal generalization matrix for the correlation within versus across categories. The signature of category-specific information in the brain is the difference between the two, thus Figure 4E,F presents the subtraction of across-category from within-category. For all conditions (within, across and difference), significance was ascertained as before (Storey, 2002). Nonsignificant correlations appear in Figure 4, bottom panels, dark blue.
Temporal generalization of RSA on the current visual object. A, Within-category correlations. B, Significant (FDR-corrected) within-category correlations. C, Across-category correlations. D, Significant (FDR-corrected) across-category correlations. E, Difference between within and across-category correlations. F, Significant (FDR-corrected) differences.
Category-specific information, as indexed by the difference illustrated in Figure 4F, emerges in the brain response ∼200 ms after stimulus onset, and is largely confined around the diagonal, meaning that the spatial pattern of brain activity is specific to a given time point.
The second RSA examined the similarity between subsets of morphs preceded by the same versus different objects. Figure 5 presents the average similarity matrix for subsets of morphs preceded by one of three unambiguous object categories, averaged across all time points. Serial dependence would manifest here as higher diagonal correlations relative to off-diagonal: if a set of morphs preceded by faces is more similar to another set of morphs also preceded by faces relative to a set of morphs preceded by another object (e.g., cars), then that would point toward a trace of the previous object (a face) in the response to the current morph. If the diagonal correlation is not higher than off-diagonal correlations, then that means that the similarity between subsets of identical morphs does not contain any information about the previous stimulus. Two of three off-diagonal correlations were significantly lower than the average diagonal correlation of r = 0.78 ± 0.02; the face-house correlation was not significantly different from the average diagonal (car-face vs diagonal, t(25) = 11.4, corrected p < 0.0001; house-car vs diagonal, t(25) = 12.5, corrected p < 0.0001; face-house vs diagonal, t(25) = 1.1, corrected p > 0.8). This suggests that the representation of visual objects was more similar to the representation of other visual objects when both were preceded by the same object than when they were preceded by different objects (car vs face and car vs house). The influence of previous faces and houses on current representations was, however, statistically indistinguishable, potentially because of face pareidolia occurring with the house stimuli.
Representational similarity matrix within and across categories of previously seen objects. Correlations between subsets of morphs preceded by faces, cars, or houses. Stronger correlations are in warmer colors. Stars correspond to off-diagonal correlations that are significantly different from the average diagonal correlation.
Figure 6 examines the time course. The red curve shows the correlation at each time point between two subsets of morphs preceded by objects of the same category. For example, between two subsets of morphs preceded by unambiguous faces, two subsets of morphs preceded by unambiguous cars, or two subsets of morphs preceded by unambiguous houses. The blue curve shows the correlation at each time point between two subsets of morphs preceded by objects of different categories. For example, the correlation between a subset of morphs preceded by unambiguous cars and another subset of morphs preceded by unambiguous houses. It is important to note that the morph subsets are identical. Thus, if previously seen objects did not influence the brain representation of the current object, correlations should be high and, crucially, should not differ between conditions. Timepoints at which the difference between the two conditions was significant are indicated by the black dots on the abscissa (FDR procedure). Previously seen objects influenced current representations immediately after the ambiguous morph onset. There was actually a difference between the two conditions before the visual object was presented, arguing in favor of a predetermined set or state onto which the current response builds.
Representational similarity time course as a function of previously seen objects. Red curve, Correlation between subsets of morphs preceded by objects of the same category. Blue curve, Correlation between subsets of ambiguous morphs preceded by objects of a different category. Other conventions as in Figure 3.
The topographies in Figure 6 illustrate the electrodes that contributed the most to the representational similarity, based on the searchlight analysis described in Materials and Methods. The positive difference between the two conditions is broadly distributed across the scalp, with a peak at frontal electrodes in the 400- to 800-ms range, when the effect is descriptively strongest.
The temporal generalization analysis is in Figure 7. The top panels represent the raw cross-correlations, and the bottom panels highlight only the FDR-corrected significant timepoints. Figure 7F shows that the difference between within-object and across-object conditions is largely confined to and around the diagonal, meaning that the spatial pattern of brain activity is specific to a given time point.
Temporal generalization of RSA as a function of the previous visual object. Conventions as in Figure 4.
The final analysis examined whether there was any representational similarity between past and present. The previous analysis showed that there was a trace of the past in the current representation. This trace could take several forms: it could be a reactivation or recurrence of the representation of the previously seen object, or it could be a trace that was entirely different from the brain representation that occurred at the time that previous object was viewed. The third RSA thus looked at the correlation between the representation of the previous object in the current response and that of the current object. For example, the correlation between the representation of morphs preceded by a face and the representation of a face when it was the current object versus when it was a car or house. If the signal driving the trace of previously seen faces in the representation of a current morph is a reactivation of a face representation, then we should find a higher correlation between morphs preceded by a face and faces than between morphs preceded by a face and cars. If the trace of the past is not representationally similar to current perception, there should be no such difference.
Figure 8 shows the analysis. There was no difference between the conditions: the trace of the past in the current response is not representationally similar to what was presented in the past. This null result renders examining topography and temporal generalization meaningless and we therefore did not perform these analyses.
Representational similarity time course between past and present. Red curve, Correlation between a subset of morphs preceded by an object category and a subset of that object category when it was the current object. Blue curve, Correlation between a subset of morphs preceded by an object category (e.g., faces) and a subset of another object category when it was the current object (e.g., cars or houses). Other conventions as in Figures 3 and 6.
Discussion
Participants categorized images of visual objects: cars, houses, and faces. Their behavioral responses showed attraction to stimuli seen on the previous trial. These results replicate a large number of studies on various stimuli, using a categorical task inspired by Manassi et al. (2019). The spatial pattern of brain activity in response to the current visual object was influenced by the previous stimulus. The representational similarity between a set of morphs preceded by an unambiguous car and another set of morphs preceded by an unambiguous face or house was lower than that between two sets of morphs both preceded by cars.
Several previous studies have used multivariate analyses similar to RSA in rationale to examine whether a trace of the previous stimulus could be found in the response to the current stimulus. For example, using multivariate pattern analysis (MVPA), Bae and Luck (2019) trained a support vector machine (SVM) to differentiate two categories of trials: those preceded by object A versus those preceded by object B, and then successfully categorized novel trials as a function of the preceding stimulus. These results, like the present results, show that the current trial contains information about the previous trial. Fornaciai and Park (2018) examined numerosity judgments and confirmed that the amplitude of the ERP evoked by an ensemble of dots scales with numerosity. Furthermore, the numerosity seen on the previous trial influenced the ERP amplitude: when the current numerosity was preceded by smaller one, the ERP was smaller than when that same numerosity was preceded by larger one. Fornaciai and Park (2018) then successfully trained an SVM to categorize numerosities of 200 as a function of whether they had been preceded by a smaller (100 dots) or larger (400 dots) numerosity.
The current results are in line with both the Bae and Luck (2019) and Fornaciai and Park (2018) studies in that we all show that there is a trace of the previous stimulus in the response to the current stimulus. However, the MVPA approach is agnostic as to whether the neural code is shared between previous (memory) and current (perception), because the features that drive the successful decoding are unknown. In other words, it does not say whether that part of the representation that persists is similar to the representation of the currently seen object. One way to adapt decoding methods to address the question would be to train the SVM on current trials of object A versus B, and then to see whether the algorithm could subsequently classify novel objects as a function of the object that had been presented on previous trials. If a decoder able to differentiate object A versus B on the current trial can then classify AX versus BX, then this would suggest that there was A-like or B-like activity in the current (X) trial.
The RSA analysis presented here does examine the proximity between brain representations of a given stimulus and the trace of that stimulus in the current response. We did not find such representational similarity, suggesting that the trace of the previous stimulus in the response to the current object is not a recurrence of the past representation. Rather, the influence of the previous stimulus likely involves a neural code that is different from that which supports the initial registration of the current object. If serial dependence suggests that object representations are spatiotemporally continuous (Fischer and Whitney, 2014), our results show that the temporal aspect is not like a running average of sensory input across a brief moment in time. The topographies further support this interpretation, since the loci that contributed the most to category-specific information in the current response were occipito-parietal, while those that contributed the most to the trace of the category seen in the past were frontal. This analysis examining representational similarity between object representations and their trace on subsequent representations also addresses a potential alternative explanation for our results: that the response to the previously seen stimulus simply lingers for an extended period of time, and spills over into the next trial. If that were the case, then it would be erroneous to characterize our finding as a trace of the past in the present representation and to relate it to the behavioral phenomenon of serial dependence. However, the representational dissimilarity between an object representation and the trace it leaves on subsequent processing argues firmly against this interpretation.
Despite the dissimilarity between the trace of the past objects in current representations and current object representations, we believe that one cannot conclude from our data that serial dependence is a decisional or memory effect rather than a perceptual effect. This question has been debated and is, in our view, still open (Cicchini et al., 2017, 2018; Fritsche et al., 2017, 2020; Collins, 2020; Sheehan and Serences, 2023). Memory, perception, decision and many other psychological constructs manifest in a variety of different brain responses and patterns, and finding two dissimilar patterns does not mean that the underlying construct is different.
It is also important to note that the dissimilarity between the trace of a past object in current representations and that object's current representation is a negative result, and has to be interpreted with regards to the level of behavioral serial dependence, which was modest in the current study. Perhaps a larger effect would lead to representational similarity between past and current representations.
The finding that the past leaves a trace in subsequent electrophysiological neural signals suggests that serial dependence is the manifestation of an active mechanism. Indeed, learning at short time scales in the brain can result from changes in the synaptic connections between neurons (short-term synaptic facilitation; Mongillo et al., 2008), but such changes would not lead to modifications in electrophysiological activity although of course they could influence behavior. We therefore suggest that our results are indicative of an active representation of the previous stimulus that interacts with the representation of the current stimulus. Such an active representation would produce recordable neural signals that differ reliably between object category conditions.
Recently, Sheehan and Serences (2022) proposed a model that aimed to reconcile attractive biases in behavior and repulsive adaptation in the cortex. They showed that the attractive biases in behavior may be because of read-out mechanisms that overcompensate for cortical adaptation. Their results suggest a mechanism by which the visual system both improves coding efficiency through cortical adaptation and optimizes perceptual behavior by integrating across time. The similarity in the present results between representations that share a similar past could arise either because of similar adaptation between conditions, or because of attractive biases in representation; one must not confound representational similarity and attractive effects. Another limit of the RSA method is that of course it can only identify similarities that may be related to preselected differences between experimental conditions. Thus, the glimpse into the dimensions of subjective experience (Tallon-Baudry, 2022) addressed by this method are only partial. Nevertheless, our results show that the representation of visual objects incorporates information about the recent past, suggesting that temporal integration of sensory information across a brief period in time may play a role in stable perception.
Footnotes
- Received November 8, 2022.
- Revision received January 19, 2023.
- Accepted February 18, 2023.
-
The authors declare no competing financial interests.
- Correspondence should be addressed to Thérèse Collins at therese.collins{at}u-paris.fr
- Copyright © 2023 the authors
