Large-Scale Color Biases in the Retinotopic Functional Architecture Are Region Specific and Shared across Human Brains
- 1Werner Reichardt Centre for Integrative Neuroscience, University of Tübingen, Tübingen 72076, Germany
- 2Max Planck Institute for Intelligent Systems, Tübingen 72076, Germany
- 3Max Planck Institute for Biological Cybernetics, Tübingen 72076, Germany
- 4Bernstein Center for Computational Neuroscience Tübingen, Tübingen 72076, Germany
- 5Department of Psychology, University of Tübingen, Tübingen 72076, Germany
-
Author contributions: M.M.B. and A.B. designed research; M.M.B. performed research; M.M.B. analyzed data; M.M.B. and A.B. wrote the paper.
Abstract
Despite the functional specialization in visual cortex, there is growing evidence that the processing of chromatic and spatial visual features is intertwined. While past studies focused on visual field biases in retina and behavior, large-scale dependencies between coding of color and retinotopic space are largely unexplored in the cortex. Using a sample of male and female volunteers, we asked whether spatial color biases are shared across different human observers and whether they are idiosyncratic for distinct areas. We tested this by predicting the color a person was seeing using a linear classifier that has never been trained on chromatic responses from that same brain, solely by taking into account: (1) the chromatic responses in other individuals’ brains and (2) commonalities between the spatial coding in brains used for training and the test brain. We were able to predict the color (and luminance) of stimuli seen by an observer based on other subjects’ activity patterns in areas V1–V3, hV4, and LO1. In addition, we found that different colors elicited systematic, large-scale retinotopic biases that were idiosyncratic for distinct areas and common across brains. The area-specific spatial color codes and their conservation across individuals suggest functional or evolutionary organization pressures that remain to be elucidated.
- color vision
- fMRI
- pattern classification
- retinotopic bias
- shared response modeling
- visual field maps
Significance Statement
Does a given color elicit comparable neural activity in two different observers? Do colors elicit area-specific response patterns? To address these questions, we predicted what color someone is seeing based on their brain activity, using only knowledge of color responses from other observers’ brains. Estimating the commonalities across brains in the way they respond to achromatic, spatial stimulation allowed us to retinotopically align different brain responses to each other in a common response space. In this space derived without any color responses, we could decode across individuals what color an observer was seeing and found spatial color biases that differed between areas. Our results demonstrate systematic dependencies between chromatic and visual field representations that are area-specific and preserved across observers.
Introduction
It is known that the primate visual cortex contains multiple representations of the visual field in different functional areas (Felleman and Van Essen, 1991; Sereno et al., 1995; Wandell et al., 2007). This multiplicity has been explained in terms of differences between areas in what cognitive and behavioral purpose their computations fulfill (Mishkin and Ungerleider, 1982; Goodale and Milner, 1992), what perceptual processes they reflect (Gauthier et al., 2000), and what content or object properties they encode (Zeki et al., 1991; Tanaka, 1993; Malach et al., 1995; Kanwisher et al., 1997).
While differences between brain areas are widely accepted, within-area variability that can be explained by the underlying cortical retinotopy received less attention (Sasaki et al., 2006; Freeman et al., 2011, 2013; Roth et al., 2018). Since these few studies focused on orientation tuning only, even less is known about color.
Traditionally, human color vision has been thought to be mediated by functionally segregated mechanisms already at earliest processing stages (De Valois and Jacobs, 1968; Zeki, 1973; Derrington et al., 1984; Schiller et al., 1990; Field and Chichilnisky, 2007). Independent processing could explain dissociations in LGN and visual cortex, with a specialization for color processing in the cytochrome oxidase-rich “blob” and “thin stripe” regions in areas V1 and V2, respectively (Livingstone and Hubel, 1984, 1988; Ts’o and Gilbert, 1988; Schiller et al., 1990; Lu and Roe, 2008; Denison et al., 2014; Nasr et al., 2016), and in V4 (Zeki, 1978, 1980; Lueck et al., 1989; Zeki et al., 1991; Beauchamp et al., 1999; Bartels and Zeki, 2000), especially its “globs” (Conway et al., 2007; Conway and Tsao, 2009).
There are few neuroimaging experiments investigating how cortical color processing differs across the visual field, but they have been limited to V1 and investigated the radial field coordinate only (Vanni et al., 2006; Mullen et al., 2007) or nonhuman primates (Loggia et al., 2025). This is surprising if one considers the inhomogeneities in retinal cone distributions (Roorda and Williams, 1999) and behavioral chromatic sensitivity differences across visual space (Mullen, 1991; Mullen and Kingdom, 2002; Levine and McAnany, 2005; Danilova and Mollon, 2009; Horiguchi et al., 2013). Computational analysis furthermore suggested that the difference in color statistics between objects and surrounding background contributes to color tuning in object recognition regions (Rosenthal et al., 2018). It thus makes sense that objects, which are typically foveated, might benefit from different chromatic processing than the peripheral background. Finally, simulations showed that, to explain visual performance differences across the visual field, retinal factors are insufficient, suggesting cortical contributions are plausible (Kupers et al., 2019).
Our research aim was to examine functional differences in color processing as a function of visual space in retinotopic areas. Specifically, we were interested to learn how consistent such dependencies would be across different brains and across distinct visual areas.
We addressed these questions by aligning subject-specific fMRI response spaces to each other using shared response modeling. In contrast to previous similar cross-subject alignment approaches (Haxby et al., 2011; Guntupalli et al., 2016), we used brain responses evoked by the spatially changing position of slowly moving achromatic checkerboard stimuli used for retinotopic mapping. Individual brain activity was hence sampled in response to a highly restricted subset of stimulus inputs composed of purely spatial variability in the absence of chromatic stimulation. This ensured that the stimulus properties that were used for the estimation of shared neural responses were based solely on spatial information. If neural corepresentations of space and color are indeed shared across observers, it should be possible to predict from an observer's pattern of brain activity what color the person was seeing using a classifier that was trained solely on other observers’ brain responses to color, i.e., to classify color across the brains of different participants. Our findings showed this to be true and furthermore revealed large-scale retinotopic biases idiosyncratic for distinct regions in the visual processing hierarchy that are shared between brains.
Materials and Methods
Participants
We analyzed fMRI data from N = 15 (2 male, 13 female) participants aged between 22 and 35 years (mean, 25.5) who took part in a previously published fMRI study about color vision (Bannert and Bartels, 2018). The participants were the subset from the prior study for whom the cortical retinotopic representations of the visual field were measured along both the polar and the eccentricity axis of the visual field. All participants had normal or corrected-to-normal visual acuity and were tested for normal color vision using Ishihara color plates (Ishihara, 2011). Each participant gave written informed consent before the first study session. The experiment was approved by the local ethics committee of the Tübingen University Hospital.
Experimental setup
All stimuli were shown to the volunteers while they were lying in the scanner using a projector (NEC PE401H) that was gamma-calibrated with a Photo Research PR-670 spectroradiometer running the Psychtoolbox-3 display calibration script. Experimental stimuli were presented at a resolution of 800 × 600 pixels and a refresh rate of 60 Hz. The display on the projection screen had a size of 21.8° by 16.2° of visual angle. Stimuli were presented using Psychtoolbox-3 (Kleiner et al., 2007) and MATLAB running on Windows XP.
Color stimuli
Brain responses obtained from a retinotopic mapping session (described in a separate section) and a color vision experiment were used for the present analysis. Stimuli of both sessions have been described in detail before (Bannert and Bartels, 2018) and are again briefly described below.
In the color vision experiment, we showed stimuli with each belonging to one of three different color categories: red (mean chromaticities x = 0.39, y = 0.35), green (mean chromaticities x = 0.34, y = 0.41), and yellow (mean chromaticities x = 0.41, y = 0.43). Each color category was shown at a high or low luminance, resulting in a total of six trial types. Each stimulus consisted of concentric rings presented against a midlevel gray background (154 cd/m2). Stimulus intensities were psychophysically matched (see below) for the high and low luminance conditions, respectively, yielding mean luminance values (SD in brackets) of 241.7 (19.8) cd/m2 and 199.8 (9.0) cd/m2 for red, 242.4 (11.4) cd/m2 and 190.0 (8.8) cd/m2 for green, and 230.4 (11.6) cd/m2 and 179.6 (9.8) cd/m2 for yellow. Transparency of the stimulus color was sinusoidally modulated as a function of radial distance from the center (thereby conferring the appearance of multiple concentric rings). The radius of the largest ring was 8.61° of visual angle, and the cycle size of the modulation was 2.16° of visual angle with its phase changing continuously at a speed of 2.47°/s in an outward direction. To ensure that the three colors appeared isoluminant, we used the minimum flicker method (Kaiser, 1991) inside the scanner prior to data acquisition: a color rectangle of size 3.28° by 2.46° of visual angle in vertical and horizontal directions from a given color category was shown foveally to the participant while continuously replacing it every second frame with an achromatic reference stimulus of a given luminance. By adjusting the luminance of the color patch by button press in steps of 11–12 cd/m2, the participant chose the stimulus intensity that minimized the amount of perceived flicker. Since the three color conditions were presented in either low or high luminance (151.3 or 184.9 cd/m2), each participant individually performed adjustments with reference stimuli at two different luminance levels. Our color experiment thus had a 3-by-2 factorial design.
Experimental design and task
The observers had to foveate the fixation dot in the middle of the screen while paying attention to the expanding color rings. Each stimulus was presented for 8.5 s followed by an intertrial interval (ITI) of 1.5 s. There were 36 trials per imaging run for a total of 216 trials presented across the six runs. The trial sequence was pseudorandomized across runs to ensure that each of the six conditions was preceded equally often by every condition (Brooks, 2012). The last trial in every run was repeated at the beginning of the subsequent run to ensure that the trial sequence remained counterbalanced across all runs. The first trial of each run was excluded from the analysis.
The task was to indicate by button press the occurrence of a brief (0.3 s) luminance increment or decrement of the color stimulus. Specifically, when presenting a high luminance color stimulus, the target was the low luminance stimulus and vice versa. Observers were instructed to respond as quickly and accurately as possible and received visual feedback about mean reaction time (RT) and errors at the end of each imaging run.
Retinotopic mapping details
In the retinotopic mapping experiment, each participant underwent four runs of polar angle and two runs of eccentricity mapping following standard protocol (Sereno et al., 1995; Wandell and Winawer, 2011). In polar runs, observers viewed an achromatic contrast-reversing checkerboard stimulus through a wedge-shaped aperture in a midlevel gray layer occluding the checkerboard. The wedge had an angle of 45° and extended to the edge of the display. The aperture slowly rotated in a clockwise direction in half of the runs or counterclockwise direction in the remaining half. In the two eccentricity runs, the checkerboard stimulus was viewed through an annulus that cyclically expanded in one run whereas it contracted in the other. The check sizes in the polar and eccentricity runs as well as the annulus width increased logarithmically with radial distance to accommodate cortical magnification. Participants were instructed to keep their eyes on the fixation cross in the middle of the screen while paying attention to the wedge or annulus apertures, respectively, and to press a button each time a red dot briefly appeared randomly anywhere in the visible checkerboard.
Per run, most participants saw 10 cycles of the rotating wedge and expanding or contracting annulus with each cycle lasting 55.68 s. For Subjects 2, 8, and 14, the stimulus period was 50.46 s and 12 cycles were presented. Subject 1 saw only nine cycles. To be able to use shared response modeling (see below), we made sure that the stimulus input was the same for each time point across participants by including only fMRI measurements for the first nine cycles and temporally resampling data from Subjects, 2, 8, and 14 to obtain 576 fMRI volumes for each participant. However, we used each participant's original dataset to define retinotopic regions of interest (ROIs) and determine polar coordinates for each voxel.
fMRI scan details
Imaging was carried out on a Siemens Prisma scanner at 3 T magnetic field strength using a 64-channel head coil. The 56 slices were positioned, without gaps between them, approximately parallel to the AC-PC line for whole brain coverage. We employed a multiband factor 2 and GRAPPA factor 2 to obtain a fourfold accelerated parallel imaging sequence. T2*-weighted functional images were recorded at a repetition time (TR) of 0.87 s and an in-plane matrix resolution of 96 × 96. Slice thickness and voxel size were 2 mm isotropic. Echo time (TE) was 30 ms and flip angle was 57°. Anatomical images were recorded for each participant at a voxel size of 1 mm isotropic using a T1-weighted MP-RAGE ADNI sequence. To correct for magnetic field inhomogeneities, we measured gradient field maps which were included in the motion correction step of the fMRI preprocessing pipeline.
fMRI data preprocessing
Data from the main experiment were preprocessed using SPM8 (https://www.fil.ion.ucl.ac.uk/spm) running on MATLAB 2014b (The MathWorks). We excluded the first 11 images of each run to allow the magnetic field to reach equilibrium. To correct for head motion, each participant's functional images were realigned to the first recorded image. The gradient field maps were used to unwarp the image sequence in order to take into account magnetic field distortions. We corrected for differences in slice acquisition times by shifting the phase of each frequency in the signal's Fourier representation to the middle of the volume. Every participant's image sequence was then coregistered to their respective anatomical scan. Anatomical scans were spatially normalized to MNI space using SPM's segmentation-based method, and the ensuing transformations were also applied to normalize functional images to MNI space.
The data from the retinotopic mapping experiment were motion corrected, coregistered, and slice time corrected with SPM8 in the same way as the data from the main experiment. We used FreeSurfer's recon-all pipeline (https://surfer.nmr.mgh.harvard.edu) to reconstruct each participant's cortical surface from their individual anatomical scan (Fischl, 2012). The functional images were then spatially smoothed on the cortical surface with a 4 mm Gaussian kernel.
Retinotopic ROI definition
We used standard retinotopic procedures to functionally identify visual areas in each participant separately (Sereno et al., 1995; Wandell and Winawer, 2011). After applying Fourier transformation to the time series of each vertex, we plotted the corresponding phase of each vertex at the frequency of the polar angle mapping stimulus onto the inflated cortex. The reversals in the angle map indicate the boundaries between visual areas. In this way, we delineated boundaries for areas V1, V2, V3, hV4, VO1, LO1, and LO2. The definition of ROIs is shown in Figure S1.
Pattern estimation and preparation
To extract patterns of brain activity, we fit each voxel time series with a general linear model (GLM) in SPM8. The design matrix contained one boxcar regressor for each of the 216 trials (36 trials per run, six runs in total), a regressor each for the first trials in every run, which were excluded from the analysis, a constant intercept for each run, and the six parameters from the motion correction step to model out motion-induced artifacts in the voxel time series. To account for the hemodynamic lag of the BOLD response, we shifted each boxcar 5 s forward in time. The beta weights were obtained using restricted maximum likelihood estimation and were then used to form vectors of brain responses to their respective color stimulus. Finally, before a sequence of vectors were entered into pattern classification analyses, each voxel time series within this sequence was—for each run separately—temporally detrended by removing the fit of a second-order polynomial (effectively high-pass filtering the data), followed by scaling to zero mean and unit variance.
Pattern classification
Pattern classification was performed in Python 3 using scikit-learn 0.19.1 (Pedregosa et al., 2012). We used linear discriminant analysis (LDA) to train classifiers based on a shrinkage estimate of the covariance matrix (Ledoit and Wolf, 2004). In a first step, we tested how well color (and luminance) information could be decoded from brain signals when the training and test data came from the same participant. In the second step, we examined how well color (or luminance) patterns generalized across participants. We refer to the analyses in the first and second steps as within-subject classification (WSC) and between-subject classification (BSC), respectively (Haxby et al., 2011).
In WSC analyses, we trained LDA classifiers to predict from vectors of neural responses from which of the three color categories (or the two luminance conditions in the luminance classification) they came. For the classification of color, we pooled the conditions across luminance and, for the classification of luminance, we pooled across color. We obtained unbiased estimates of classification performance by cross-validating them following a leave-one-run-out cross-validation procedure. There were six runs with 36 response vectors each. The classifier was thus trained on 180 vectors (5 times 36) and tested on 36 vectors in every iteration. Each vector element corresponded to the response estimate of a single voxel within a given ROI. Individual results were averaged across all participants.
In BSC analyses, however, we were interested to learn how well color (or luminance) responses generalized across participants. Rather than cross-validating classifiers across runs (within each participant separately) and then averaging results over participants, we now cross-validated across participants and obtained a single value for the whole group. Another difference was that vector elements used for classification came from the k-dimensional common space estimated through shared response modeling (see below; instead of voxels used in WSC) for each ROI. With 216 response vectors (one for each trial) for each of our 15 participants, this leave-one-participant-out cross-validation approach resulted in classifiers being trained on 3,024 (14 times 216) response vectors and tested on the 216 response vectors of the left-out participant in every iteration. The k vector elements corresponded to the common space dimensions, i.e., weighted sums of the voxel estimates from individual observers’ ROIs. The value of k was determined via grid search on the training set for each ROI (see below, Shared response modeling). Since shared response modeling implicitly already performs dimensionality reduction, we did not use any additional feature selection such as recursive feature elimination.
Since color decoding was a three-way classification whereas luminance decoding was a two-way classification, classification
accuracies were transformed to z-values using the normal approximation to the binomial distribution:<mml:math display="block"><mml:mi>z</mml:mi><mml:mo>=</mml:mo><mml:mstyle displaystyle="true" scriptlevel="0"><mml:mrow><mml:mfrac><mml:mrow><mml:mi>n</mml:mi><mml:mo
stretchy="false">(</mml:mo><mml:mrow><mml:mi>a</mml:mi><mml:mo>−</mml:mo><mml:mi>p</mml:mi></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow><mml:mrow><mml:msqrt><mml:mi>n</mml:mi><mml:mi>p</mml:mi><mml:mo
stretchy="false">(</mml:mo><mml:mrow><mml:mn>1</mml:mn><mml:mo>−</mml:mo><mml:mi>p</mml:mi></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:msqrt></mml:mrow></mml:mfrac></mml:mrow><mml:mo>,</mml:mo><mml:mspace
width="2em"></mml:mspace><mml:mo stretchy="false">(</mml:mo><mml:mn>1</mml:mn><mml:mo stretchy="false">)</mml:mo></mml:mstyle></mml:math>z=n(a−p)np(1−p),(1)
n is the number of predictions in the classification problem (i.e., Bernoulli trials), a is the fraction of correct predictions, and p is the probability of a correct prediction expected by chance, i.e., the reciprocal of the number of classes. Intuitively,
this transformation simply expresses the extent to which classification accuracy exceeded chance, scaled by the standard deviation
expected by chance. After z-transformation, chance level for both classification problems was hence zero.
Shared response modeling
Our hypothesis was that the purely achromatically defined functional retinotopic architecture shared across brains also contained color representations that were equally shared across participants. We therefore used shared response modeling (SRM; Haxby et al., 2011; Anderson et al., 2016; Cohen et al., 2017) to identify the shared functional architecture from responses to the achromatic retinotopic mapping stimulus. For all our SRM analyses, we used the implementation that is part of the Brain Imaging Analysis Kit, https://brainiak.org (version 0.7.1). An important advantage of SRM over a related method called hyperalignment (Haxby et al., 2011; Guntupalli et al., 2016) is that it allows for different dimensionalities of individual data matrices (i.e., number of voxels per ROI). This is because SRM estimates linear mappings between Xi and S whereas hyperalignment uses Procrustes transformation to align representational spaces by rotating and scaling vectors between voxel spaces of the same dimensionality. For intelligibility of this manuscript, in the following we briefly describe the mathematical concept of the SRM implementation used.
Let Xi be a v-by-d matrix of fMRI responses <mml:math display="inline"><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">X</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>∈</mml:mo><mml:msup><mml:mrow><mml:mrow><mml:mrow><mml:mi
mathvariant="double-struck">R</mml:mi></mml:mrow></mml:mrow></mml:mrow><mml:mrow><mml:mi>v</mml:mi><mml:mspace width="0.25em"></mml:mspace><mml:mo>×</mml:mo><mml:mspace
width="0.25em"></mml:mspace><mml:mi>d</mml:mi></mml:mrow></mml:msup></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:math>(Xi∈Rv×d)
measured for participant i with v denoting the number of voxels (e.g., within a ROI) and d the number of measurements (in our case number of volumes recorded in the retinotopic mapping experiment). In SRM each participant's
response matrix Xi is then modeled as a linear transformation of a common response <mml:math display="inline"><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:mspace width="0.25em"></mml:mspace><mml:mrow><mml:mi
mathvariant="bold-italic">S</mml:mi></mml:mrow><mml:mo>∈</mml:mo><mml:msup><mml:mrow><mml:mrow><mml:mrow><mml:mi mathvariant="double-struck">R</mml:mi></mml:mrow></mml:mrow></mml:mrow><mml:mrow><mml:mi>k</mml:mi><mml:mspace
width="0.25em"></mml:mspace><mml:mo>×</mml:mo><mml:mspace width="0.25em"></mml:mspace><mml:mi>d</mml:mi></mml:mrow></mml:msup></mml:mrow><mml:mo
stretchy="false">)</mml:mo></mml:math>(S∈Rk×d)
that is shared by all participants:<mml:math display="block"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">X</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:msub><mml:mrow><mml:mi
mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mrow><mml:mi mathvariant="bold-italic">i</mml:mi></mml:mrow></mml:msub><mml:mrow><mml:mi
mathvariant="bold-italic">S</mml:mi></mml:mrow><mml:mo>+</mml:mo><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">E</mml:mi></mml:mrow><mml:mrow><mml:mi
mathvariant="bold-italic">i</mml:mi></mml:mrow></mml:msub><mml:mo>,</mml:mo><mml:mspace width="2em"></mml:mspace><mml:mo stretchy="false">(</mml:mo><mml:mn>2</mml:mn><mml:mo
stretchy="false">)</mml:mo></mml:math>Xi=WiS+Ei,(2)
k is the number of components in the shared common space. <mml:math display="inline"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>∈</mml:mo><mml:mspace
width="0.25em"></mml:mspace><mml:msup><mml:mrow><mml:mrow><mml:mrow><mml:mi mathvariant="double-struck">R</mml:mi></mml:mrow></mml:mrow></mml:mrow><mml:mrow><mml:mi>v</mml:mi><mml:mspace
width="0.25em"></mml:mspace><mml:mo>×</mml:mo><mml:mspace width="0.25em"></mml:mspace><mml:mi>k</mml:mi></mml:mrow></mml:msup></mml:math>Wi∈Rv×k
is the transformation matrix that describes the relationship between the shared responses in the common space S and each individual's original response space with matrix <mml:math display="inline"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">E</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>∈</mml:mo><mml:msup><mml:mrow><mml:mrow><mml:mrow><mml:mi
mathvariant="double-struck">R</mml:mi></mml:mrow></mml:mrow></mml:mrow><mml:mrow><mml:mi>v</mml:mi><mml:mspace width="0.25em"></mml:mspace><mml:mo>×</mml:mo><mml:mspace
width="0.25em"></mml:mspace><mml:mi>d</mml:mi></mml:mrow></mml:msup></mml:math>Ei∈Rv×d
modeling the error between fitted and observed individual data.
SRM estimates the common space matrix S and individual transformation matrices <mml:math display="inline"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mn>1</mml:mn></mml:msub></mml:math>W1
, <mml:math display="inline"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mn>2</mml:mn></mml:msub></mml:math>W2
,…, <mml:math display="inline"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub></mml:math>Wi
,…, <mml:math display="inline"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>m</mml:mi></mml:msub></mml:math>Wm
for each of the m participants such that they minimize the Frobenius norm <mml:math display="inline"><mml:mo fence="false" stretchy="false">‖</mml:mo><mml:mo>⋅</mml:mo><mml:msub><mml:mo fence="false"
stretchy="false">‖</mml:mo><mml:mi>F</mml:mi></mml:msub></mml:math>‖⋅‖F
of each error matrix Ei summed over all participants:<mml:math display="block"><mml:munder><mml:mrow><mml:mo form="prefix">min</mml:mo></mml:mrow><mml:mrow><mml:msub><mml:mrow><mml:mi
mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>,</mml:mo><mml:mrow><mml:mi mathvariant="bold-italic">S</mml:mi></mml:mrow></mml:mrow></mml:munder><mml:mspace
width="0.2em"></mml:mspace><mml:munderover><mml:mrow><mml:mo movablelimits="false">∑</mml:mo></mml:mrow><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>m</mml:mi></mml:munderover><mml:mspace
width="0.2em"></mml:mspace><mml:mo fence="false" stretchy="false">‖</mml:mo><mml:mrow><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">X</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>−</mml:mo><mml:msub><mml:mrow><mml:mi
mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">S</mml:mi></mml:mrow><mml:msubsup><mml:mo
fence="false" stretchy="false">‖</mml:mo><mml:mi>F</mml:mi><mml:mn>2</mml:mn></mml:msubsup></mml:mrow><mml:mo>,</mml:mo></mml:math>minWi,S∑i=1m‖Xi−WiS‖F2,
<mml:math display="block"><mml:mi>s</mml:mi><mml:mo>.</mml:mo><mml:mi>t</mml:mi><mml:mo>.</mml:mo><mml:mspace width="0.25em"></mml:mspace><mml:mspace
width="0.25em"></mml:mspace><mml:mspace width="0.25em"></mml:mspace><mml:msubsup><mml:mrow><mml:mi mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>i</mml:mi><mml:mi>T</mml:mi></mml:msubsup><mml:msub><mml:mrow><mml:mi
mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>=</mml:mo><mml:mspace width="0.25em"></mml:mspace><mml:msub><mml:mrow><mml:mi
mathvariant="bold-italic">I</mml:mi></mml:mrow><mml:mi>k</mml:mi></mml:msub><mml:mo>.</mml:mo><mml:mspace width="2em"></mml:mspace><mml:mo
stretchy="false">(</mml:mo><mml:mn>3</mml:mn><mml:mo stretchy="false">)</mml:mo></mml:math>s.t.WiTWi=Ik.(3)
This optimization is performed subject to an orthonormality constraint on each transformation matrix Wi. The estimated solutions for Wi are therefore similar in interpretation to orthogonality in PCA. Importantly, we can use Wi to map patterns of fMRI responses between individual spaces and the common space.
We fit SRMs to data from the retinotopic mapping experiment (Fig. 1). This yielded one transformation matrix Wi per participant and ROI describing the linear mappings between each individual dataset and the common space of shared responses
to purely achromatic, spatially defined stimulation. Let <mml:math display="inline"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">D</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>∈</mml:mo><mml:mspace
width="0.25em"></mml:mspace><mml:msup><mml:mrow><mml:mrow><mml:mrow><mml:mi mathvariant="double-struck">R</mml:mi></mml:mrow></mml:mrow></mml:mrow><mml:mrow><mml:mi>v</mml:mi><mml:mspace
width="0.25em"></mml:mspace><mml:mo>×</mml:mo><mml:mspace width="0.25em"></mml:mspace><mml:mn>216</mml:mn></mml:mrow></mml:msup></mml:math>Di∈Rv×216
be the matrix of v-dimensional patterns of responses to the color and luminance stimuli from the main experiment. The transformation matrices
<mml:math display="inline"><mml:msub><mml:mrow><mml:mi mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub><mml:mo>∈</mml:mo><mml:msup><mml:mrow><mml:mrow><mml:mrow><mml:mi
mathvariant="double-struck">R</mml:mi></mml:mrow></mml:mrow></mml:mrow><mml:mrow><mml:mi>v</mml:mi><mml:mspace width="0.25em"></mml:mspace><mml:mo>×</mml:mo><mml:mspace
width="0.25em"></mml:mspace><mml:mi>k</mml:mi></mml:mrow></mml:msup></mml:math>Wi∈Rv×k
for participant i can then be used to map the individual color responses to the k-dimensional common space as <mml:math display="inline"><mml:msubsup><mml:mrow><mml:mi mathvariant="bold-italic">W</mml:mi></mml:mrow><mml:mi>i</mml:mi><mml:mi>T</mml:mi></mml:msubsup><mml:msub><mml:mrow><mml:mi
mathvariant="bold-italic">D</mml:mi></mml:mrow><mml:mi>i</mml:mi></mml:msub></mml:math>WiTDi
. We repeatedly trained LDA classifiers to distinguish between color (or luminance) categories leaving out one participant's
transformed dataset each time for cha.
Voxel preference prediction across subjects
The results of the BSC analysis suggested a systematic relationship between the retinotopic organization and voxel responses to color. To examine the nature of this relationship in more detail, we tested if SRMs could predict which color elicits the strongest response in single voxels: in other words, can we use information from N − 1 brains (SRM/retinotopy and color information) to predict single-voxel color preference from the left-out, Nth brain (only SRM/retinotopy information)? Note that, whereas the previous BSC described above predicted what color a person was seeing based on patterns of brain activity, we here aimed to predict the response preference for individual voxels.
This analysis extends previous research showing that the location of category-selective visual regions FFA and PPA could be predicted across brains using a common model space (Haxby et al., 2011) in that it examines the generalizability for more fine-grained visual information than basic level concept distinctions (houses vs faces).
As in the BSC analysis, SRMs were fit to the retinotopic mapping data of the whole group of brains. The goal is to predict, for each brain's ROI, the stimulus preferences for all individual voxels in a leave-one-brain-out cross-validation procedure. To this end, the individual mean patterns of responses to every stimulus class (e.g., red) from all training brains were mapped into the common space using the individual transformation matrices from the SRM. The transformed patterns were then averaged per stimulus class in the common space yielding one group average vector per class. The vector dimensionality depended on the optimal number of shared features as determined by grid-search (see above, Shared response modeling). The group averages were transformed into the individual voxel space of the left-out test brain. The predicted stimulus preference for a voxel was the stimulus class with the largest predicted value, which could then be compared with its true preference based on the main experiment data from that brain, which was used at no point in the prediction pipeline. Figure 4 shows an illustration of the procedure.
Prediction accuracy was quantified using a balanced score metric because the preference for a given color in a ROI's voxels could not be assumed to be balanced across classes (see balanced_accuracy_score in scikit-learn; see below, Statistical inference, for details).
Voxel preference: visualization
Finally, we visualized the dependence of stimulus preference and retinotopic location. The voxel preference predictions from each cross-validation fold were subjected to Gaussian kernel density estimation (KDE) using least-squares cross-validation for a data-driven bandwidth selection (Racine, 2007) in order to obtain a two-dimensional density function per stimulus class expressing preference for that class for any visual field location. To obtain a probabilistic interpretation of stimulus preference at any location, we applied a softmax function to the values of each function per stimulus class. This resulted in a probability distribution for every location over the stimulus categories describing how likely it was that a voxel in that location would prefer a given stimulus. We subtracted from each value the probability that a given stimulus class would be preferred under the assumption of a uniform probability distribution, i.e., one-third and one-half everywhere for color and luminance, respectively. After converting the probability differences to percentages, a value of x therefore means that the probability of a given class being preferred at this location is increased by x % points. Note that the preference maps are based on predictions that are cross-validated across brains. The prediction of voxel preferences in a given brain therefore requires that any spatial biases in color or luminance processing are reliable across brains used for fitting the model and the brain used for model testing. As an additional check, we performed an odd–even split on the sample of participants and created the same maps based on independent subsets of data to assess split-half reliability.
ROI specificity of SRMs for voxel preference prediction
Since SRMs trained on retinotopic mapping data could successfully be used to predict stimulus preference across different brains, this indicated that they captured to some extent the relationship between retinotopy and color/luminance representation. How specific are these relationships to individual visual areas? Do SRMs trained on different visual areas capture these relationships differently?
To answer these questions, we repeated the voxel preference prediction analysis detailed above using the SRM approach based on the mean patterns of univariate responses to each category. This time, however, we used SRMs that were fit either on data from the same or different ROIs. More precisely, the transformation matrices used to, on the one hand, map individual patterns from individual spaces into the common space and, on the other hand, predict stimulus preference in the space of the left-out test brain were based on SRMs fit on data from either the same or different ROIs. All combinations of ROIs were examined for the transformation of the training and test brains and the classification accuracy between congruent (SRMs fit on the same ROI) and incongruent (SRMs fit on different ROIs) were compared. For the incongruent condition, the accuracies were averaged across all combinations where SRMs came from different ROIs. We tested if the resulting accuracies were significantly larger than chance and if they differed between congruency conditions. Complementing our comparison by congruency, we also examined in more detail the accuracy of BSC for specific pairs of individual ROIs used for training and testing.
SRM searchlight analysis
In addition to ROI analyses, we also applied SRM in a whole-brain analysis to local patterns at every location in the brain normalized to MNI standard space. The purpose of this analysis was to test how specific the shared commonalities were to the exact individual locations of visual areas compared with anatomically aligned patterns of brain responses. Please note that anatomical alignment here refers to the alignment of response patterns which are then subjected to SRM. We do not assume that anatomically aligned voxels already exhibit similar stimulus tunings. Rather, we were interested in determining the difference between modeling shared responses between retinotopically versus anatomically aligned patterns of brain activity.
We used SRM in combination with a searchlight technique that analyzed only local patterns of brain activity within a radius of three voxels (“searchlight sphere”) at every location in the whole brain (Kriegeskorte et al., 2006). The number of SRM components k was 50 in every searchlight sphere or, if the total number of voxels within the sphere was smaller (e.g., near edges in the brain mask), the number of available voxels. LDA was then used in the same manner as in the BSC ROI analyses to calculate an unbiased estimate of classification performance for every location in the brain, which was then assigned to the center voxel of the sphere. The resulting brain map was then spatially smoothed with a Gaussian kernel of size 6 mm FWHM.
Exploration of model robustness
Dependence of between-subject classification on number of shared features
The number of shared features assumed in fitting the SRM determines the dimensionality of common space representations. Note that, in the main analysis (see above, Shared response modeling), the value for this parameter was chosen by grid search using only the training data to learn the optimal value in an unbiased fashion. The BSC analysis was repeated with values for the number of shared features that ranged from 1 to 100 to understand how this affects classification accuracy and how much variance the shared features explain in the data. For each parameter value, we calculated the variance of every voxel time series in the original datasets from the retinotopic mapping session and the main experiment and summed them over voxels and brains. After mapping each participant's responses from the common space back into original space (using the estimated transformation matrix), the same calculation was conducted on the resulting datasets, thereby representing how much variance was explained by the model. Explained variance was quantified as the percentage of this variance relative to the variance computed using the original data. Classification accuracy was calculated as detailed in the paragraph describing the BSC analysis. The 95% confidence bands are determined as Jeffreys intervals.
Dependence of BSC accuracy on number of voxels
Classification accuracies may differ between visual areas because they are different in size and hence contain different numbers of voxels. In order to check to what extent classification accuracies are robust against these differences, we repeated the BSC analysis using the same number of voxels across visual areas. This meant that the number of voxels had to be reduced because the smaller areas impose an upper limit on the maximum number of voxels that can be used for classification. To decide which voxels to select, all voxels were ranked per ROI based on their signal strength in the Fourier analysis of the retinotopic mapping data. The F value of the test comparing voxel amplitude at the frequency of stimulus presentation to the remaining amplitudes was used as an index for the visual sensitivity of a voxel. All voxels were ranked in a descending order and the top 50, 100, …, 500 voxels (i.e., in steps of 50) were selected. Note that, for any visual area, the smallest ROI across observers limited the largest number of voxels that could be selected from each participant. Thus, only areas V1, V2, and V3 contained 500 voxels in every participant for example. The number of multiple comparisons that needed to be corrected for accordingly was sometimes lower than in the original BSC procedure. Otherwise, it was identical to the main BSC analysis described above.
Retinotopic binning analysis
The SRM approach is both, novel and methodologically complex. We used it to map retinotopic properties of distinct brains into a common space and then test whether color preferences are also shared across participants in this space. The method also allowed us to visualize color category preferences of the shared responses to individual visual field maps.
We wanted to assure ourselves of the validity of our conclusions from our analysis by complementing it with a more conventional procedure—retinotopic binning—that had been applied previously to test for retinotopic biases (Freeman et al., 2011; Beckett et al., 2012; Larsson et al., 2017). Although the SRM-based analysis provides greater richness of information, it makes testable predictions for the retinotopic binning approach as well. In retinotopic binning, WSC of color is conducted as follows: all voxels in a given ROI are binned into a certain number of bins, and the voxel responses to the colors are averaged per bin. The dimensionality of each response pattern in a ROI containing v voxels binned into b bins is thus reduced from v to b. Now, to test for a retinotopic bias in this scenario, binning is carried out in two different ways. In one analysis, the phase in which voxels respond to the retinotopic mapping stimulus (polar angle or eccentricity) and which ranges from 0 to 2π is used to assign voxels into bins consisting only of voxels with similar phases. This preserves some aspect of the retinotopic structure in the data. In the control analysis, voxels are binned randomly. If classification is mediated at least in part by retinotopic bias, one would expect that retinotopy preserving binning yields higher classification accuracy than random binning.
The two analyses (binning for retinotopic angle or eccentricity, respectively) are conducted for several values of b chosen from a base-10 logarithmic scale ranging from 0 to 3 in steps of 0.2: <mml:math display="inline"><mml:mi>b</mml:mi><mml:mspace width="0.25em"></mml:mspace><mml:mo>∈</mml:mo><mml:mspace width="0.25em"></mml:mspace><mml:mo
stretchy="false">[</mml:mo><mml:mrow><mml:msup><mml:mn>10</mml:mn><mml:mn>0</mml:mn></mml:msup><mml:mo>,</mml:mo><mml:mspace
width="0.25em"></mml:mspace><mml:msup><mml:mn>10</mml:mn><mml:mrow><mml:mn>.2</mml:mn></mml:mrow></mml:msup><mml:mo>,</mml:mo><mml:mspace
width="0.25em"></mml:mspace><mml:mo>…</mml:mo><mml:mo>,</mml:mo><mml:mspace width="0.25em"></mml:mspace><mml:msup><mml:mn>10</mml:mn><mml:mi>b</mml:mi></mml:msup><mml:mo>,</mml:mo><mml:mspace
width="0.25em"></mml:mspace><mml:mo>…</mml:mo><mml:mo>,</mml:mo><mml:mspace width="0.25em"></mml:mspace><mml:msup><mml:mn>10</mml:mn><mml:mn>3</mml:mn></mml:msup></mml:mrow><mml:mo
stretchy="false">]</mml:mo></mml:math>b∈[100,10.2,…,10b,…,103]
. The analysis was carried out separately for every participant and ROI. If b exceeded the number of voxels in any of the participants in a given ROI, this binning step was skipped. We statistically
compared retinotopic and random binning with paired t tests corrected per ROI for multiple comparisons across 15 binning steps (or fewer, see above) using the Holm–Šidák procedure.
Statistical inference
Within-subject and between-subject classification
All our statistical decisions were based on permutation tests. For WSC we tested the one-tailed null hypothesis that the sample average across all participants’ classification accuracies was equal to or below chance. Since there were three classes in the color classification problem, chance level was one-third. Likewise, chance level in the luminance classification was one-half as there were two luminance levels.
We created 2,000 new label assignments in the following way: For every participant, the sequence of training labels was shuffled with the restrictions that labels were only permuted within cross-validation folds, i.e., functional runs, and that permutations were identical across ROIs. Both restrictions were implemented by setting the groups and random_state arguments in scikit-learn's permutation_test_score accordingly. A group average of classification accuracies was calculated for each ROI and iteration. This way we obtained a null distribution of average classification accuracies expected under the hypothesis that there was no relationship between labels and neural activity patterns.
It was important that permutations were identical across ROIs because they constituted the test family for which we controlled the familywise error (FWE). We formed a new null distribution for all ROIs by taking the maximum of group average classification accuracies across ROIs in every iteration (Nichols and Holmes, 2002). p values were computed as the fraction of permutations that resulted in accuracies that were larger than or equal to the observed classification accuracy. We declared results significant if p was below 0.05, thereby keeping the type I error probability of falsely rejecting at least one null hypothesis at α = 0.05. For each ROI, the lower and upper limits of the 95% confidence interval (CI) were calculated parametrically using the standard error of the mean. In the BSC analysis, upper CI limits were obtained by adding the difference between the mean of the (uncorrected) null distribution and its 2.5th percentile to the observed accuracy. Lower CI limits were obtained by subtracting the difference between the 97.5th percentile and the mean from the observed accuracy.
The permutation test for BSC was identical to that for WSC. However, since data from all participants were now combined in the k-dimensional functional common space, classifiers were now cross-validated leaving out one participant at a time. Again, labels were permuted only within cross-validation folds, which in this analysis were individual participants.
Voxel preference prediction and ROI specificity analysis
We tested how well we could predict the preference of voxels for different stimulus classes based on SRMs and whether there was a difference in prediction accuracy depending on whether the SRMs used for the training and test brains came from the same or different ROIs (congruent vs incongruent). In both analyses, it was first tested for each ROI whether classification accuracy was significantly larger than chance (1 divided by number of classes) using binomial tests. For pairwise comparisons between congruency conditions, we used independent z-tests (Python module statsmodels 0.11.0). The resulting p values for each comparison (against chance and pairwise) were corrected for multiple comparisons across seven ROIs using Holm–Šidák correction.
SRM searchlight analysis
The searchlight analysis yielded a brain map of cross-validated estimates of classification accuracies. In order to test if classification accuracies were significantly larger than what would be expected by chance, we used a one-tailed binomial test. The number of correctly classified trials were assumed to come from a sequence of 216 × 15 = 3,240 Bernoulli trials with success probability equal to one divided by the number of classes. We applied multiple-comparisons correction to the resulting map of p values by keeping the false discovery rate at q = 0.05 (Benjamini and Hochberg, 1995).
Behavioral control analysis
Our classification analysis of neural representations of color assumes that the fMRI data were collected under conditions that differed in color (or luminance) exclusively. In order to rule out the possibility that neural differences were related to differences in task performance between conditions, we therefore conducted a two-way repeated-measures ANOVA of the observers’ reaction times (RT) in the target detection task under the three color and two luminance fixed-effect conditions (3-by-2 design) specifying subject as random factor. A major disadvantage of classical ANOVA is that it only considers empirical evidence against the null hypothesis (i.e., indicating that “RTs are different across conditions”) but not against the alternative hypothesis (i.e., indicating that “RTs do not differ across conditions”). Bayesian statistics, however, simultaneously take into account evidence in support of the null and alternative hypothesis in a balanced way. Hence, we used Bayesian ANOVA (Rouder et al., 2017; Morey and Rouder, 2018) where Bayes factors (BFs) quantify the compatibility of the observed data with the alternative hypothesis relative to the null. Specifically, each effect was tested by computing the BF ratio of the globally best fitting model with the best model lacking the effect in question (Rouder et al., 2017). If the globally best model did not include a given effect, the ratio was taken with respect to the best model that did include the effect in question (resulting in BFs favoring the null). As an example, a BF of 3 means that the data are three times more likely to have occurred under the alternative hypothesis than the null. Fixed-effect priors were specified for “medium” sizes by applying the default r scale of 0.5. Normality assumptions were checked with Shapiro–Wilk's tests and by visual inspection of Q–Q plots of the residuals (data not shown). Sphericity was tested using Mauchly's test.
Code
Code accompanying the paper is available at https://osf.io/yfxhe.
Results
The goals of the experiment were to examine whether representations of chromatic signals were shared with a spatial neural code, whether spatial color specificity was shared across human brains, and whether spatial color biases were area specific. For this reason, we used brain responses to achromatic, spatial stimuli to construct a neural space that was shared between observers (Fig. 1). We then tested whether color responses projected into this shared space allowed a classifier to decode the color a given participant viewed, even when this participant had not been used in training the classifier—in other words—to predict colors across brains. We then probed whether the shared color codes were area specific, first by comparing decoding across brains using SRMs from either the same or from different visual areas for training and test brains, respectively, and second by visualizing spatial color biases across distinct visual areas.
Shared response modeling procedure. Retinotopic mapping data (top) were recorded while participants watched a rotating wedge or an expanding/contracting ring stimulus. Datasets from every participant's ROI were resampled to the same temporal resolution and entered into a shared response model (Chen et al., 2015; Anderson et al., 2016). SRM estimates a transformation matrix Wn for every participant that maps voxel response spaces from individual ROIs into a k-dimensional common space (see Materials and Methods). Note that only retinotopic mapping data were used to estimate transformation matrices. Color responses (bottom) were measured with fMRI from participants performing a luminance change detection task on red, green, and yellow ring stimuli presented at two luminance levels. Stimuli were shown for 8.5 s with ITI = 1.5 s (see Materials and Methods). These fMRI color responses, which were recorded independently from the retinotopic mapping data, were mapped from individual response spaces into the common space using the Wn matrices for every ROI and participant.
Within-subject classification of color and luminance (WSC)
Decoding color across different brains requires that color can be decoded within one and the same individual brain. For this reason, we first performed pattern classification analyses on fMRI activity from ROIs at the level of individual subjects. The same logic applied to luminance decoding. Figure 2 shows classification accuracies z-transformed (n = 216) individually and then averaged across participants (n = 15). Color as well as luminance could be decoded significantly better than chance in all ROIs under investigation (permutation tests, each p < 0.01, FWE-corrected for seven ROIs). For color (chance level: 33%), the following classification accuracies applied for each ROI, averaged across participants. V1: 57%, 95% CI [55.1%, 60%] (z = 7.38, 95% CI [6.51, 8.25]), V2: 55.4%, 95% CI [53.5%, 57.3%] (z = 6.89, 95% CI [5.94, 7.84]), V3: 52.84%, 95% CI [50.8%, 54.7%] (z = 6.08, 95% CI [5.13, 7.03]), hV4: 51.2%, 95% CI [49.2%, 53.1%] (z = 5.56, 95% CI [4.21, 6.91]), VO1: 46.3%, 95% CI [44.4%, 48.2%] (z = 4.03, 95% CI [3.01, 5.05]), LO1: 44.8%, 95% CI [42.8%, 46.7%] (z = 3.56, 95% CI [2.66, 4.46]), LO2: 42.5%, 95% CI [40.5%, 44.5%] (z = 2.86, 95% CI [1.72, 4]). The confusion matrices (Fig. S2) suggest that residual luminance differences cannot easily explain the results as discussed in Text S1.
Multivoxel pattern classification results for within-subject classification (WSC). For every ROI and participant, cross-validated (leave-one-run-out) classification accuracies were obtained for the prediction of color (dark dots) and luminance (light dots). Black diamonds indicate mean accuracy averaged across participants and error bars represent the parametric 95% CI. Asterisks mark accuracies with p values below 0.01 (permutation tests, 2,000 iterations, FWE corrected across 7 ROIs). As shown, classification accuracies were significantly larger than chance for both color and luminance classification.
For luminance (chance level: 50%), classification accuracies were as follows. V1: 61.9%, 95% CI [59.9%, 64.1%] (z = 3.51, 95% CI [2.32, 4.7]), V2: 65%, 95% CI [62.8%, 67.2%] (z = 4.41, 95% CI [2.81, 6.01]), V3: 66.7%, 95% CI [64.5%, 68.9%] (z = 4.92, 95% CI [3.45, 6.38]), hV4: 59.1%, 95% CI [56.9%, 61.1%] (z = 2.67, 95% CI [1.84, 3.5]), VO1: 56.7%, 95% CI [54.6%, 58.9%] (z = 1.98, 95% CI [1.16, 2.8]), LO1: 63.5%, 95% CI [61.4%, 65.7%] (z = 3.96, 95% CI [2.71, 5.22]), LO2: 59.9%, 95% CI [57.6%, 62%] (z = 2.9, 95% CI [1.69, 4.11]). Individual response patterns in every ROI thus were reliable enough across fMRI runs to linearly predict color and luminance conditions.
Between-subject classification of color and luminance (BSC)
Next, we quantified how consistent chromatic neural processing was across individual brains. We did this within a neural space that was shared across subjects and defined by responses elicited by a solely achromatic and spatially defined retinotopic mapping stimulus. Individual color responses were mapped from independent measurements to that shared response space, and the classifiers were trained to predict the color (or luminance) of the stimulus a person was seeing. In contrast to WSC, fitted classification models were cross-validated across participants instead of runs. As can be seen in Figure 3, accuracies for luminance and color classification across different brains significantly exceeded chance in areas V1–V3. Additionally, color could be decoded across participants significantly better than chance in areas hV4 and LO1 (permutation tests, p < 0.01, FWE corrected for seven ROIs). For each ROI, significant BSC accuracies for color were as follows. V1: 44.7%, 95% CI [40.2%, 49.1%] (z = 13.7, 95% CI [8.23, 19.0]), V2: 39.8%, 95% CI [36.8%, 42.5%] (z = 7.75, 95% CI [4.23, 11.09]), V3: 39.6%, 95% CI [36.7%, 42.2%] (z = 7.57, 95% CI [4.11, 10.75]), hV4: 39.5%, 95% CI [36.0%, 42.8%] (z = 7.42, 95% CI [3.27, 11.47]), LO1: 38.8%, 95% CI [35.1%, 42.3%] (z = 6.6, 95% CI [2.1, 10.9]). Accuracy in VO1 was 34.0%, 95% CI [31.6%, 36.3%] (z = 0.78, 95% CI [−2.11, 3.6], p = 0.873, FWE corrected) and thus not significant. In LO2 accuracy was 36.0%, 95% CI [33.5%, 38.4%], but did not survive FWE correction (p = 0.021, uncorrected): z = 3.17, 95% CI [0.14, 6.14], p = 0.285, FWE corrected. For luminance, significant classification accuracies were as follows. V1: 56.2%, 95% CI [52.5%, 60.0%] (z = 7.03, 95% CI [2.9, 11.36]), V2: 56.4%, 95% CI [52.4%, 60.3%] (z = 7.24, 95% CI [4.23, 11.09]), V3: 55.2%, 95% CI [52.0%, 58.3%] (z = 5.94, 95% CI [22.22, 9.46]). This suggests that measurements of how purely spatially defined stimulation activates the brains of different individuals is sufficient to predict colors and luminance from brain activity in any one of them using a classifier trained on data from the remaining participants.
Multivoxel pattern classification results for between-subject classification (BSC). Individual responses were transformed to the common space. Transformation matrices were estimated using the shared response model fit to the independent set of retinotopic mapping data. Classification accuracies were cross-validated by leaving out the participant during classifier training whose data were used for testing (leave-one-subject-out). Bars represent prediction accuracies averaged across participants. Error bars denote the 2.5th and the 97.5th percentile of the permuted null distributions. Asterisks and plus sign denote accuracies significantly exceeding chance at p < 0.01 and p < 0.05, respectively (permutation tests, 2,000 iterations, FWE corrected across 7 ROIs). Both color and luminance could be predicted across subjects from areas V1–V3. Additionally, color could be predicted from hV4 and LO1.
Voxel preference prediction: large-scale retinotopic color biases
Since our BSC results were based on shared responses to retinotopic mapping experiments, we were interested in a closer examination of any large-scale retinotopic biases mediating across-subject color decoding. We therefore tested how accurately we could predict to which color (or to which luminance level) each voxel in every ROI of a given test brain would respond most strongly, i.e., which it would prefer. Importantly, this prediction would be based only on the retinotopic mapping data and on color/luminance data from brains that were independent from the test brain. Figure 4 demonstrates how we used SRMs in this procedure.
Preference prediction analysis. Illustration of the SRM approach to predict and visualize stimulus class preferences (e.g., in case of color: red, green or yellow) of individual voxels within a ROI in an independent, withheld brain based on activity observed in the remaining brains. A, Prediction procedure. SRM provides direct mappings between individual and common space representations of brain activity. The retinotopic mapping data were used to obtain a shared response model for the whole group of participants. The W matrices obtained from the SRM were used to map every vector of voxel responses for each stimulus category from all individual ROI voxel spaces to the common space. The vectors in common space were averaged across participants for each category, and the result was mapped into the individual ROI voxel space of the left-out brain. The preference prediction for each voxel in the test brain was determined as the stimulus category with the strongest predicted response. B, Visualization of predictions. To visualize the stimulus preference across retinotopic space, we estimated the preference for each visual field location from the predicted voxel preferences using kernel density estimation (KDE). This yielded one 2D density function per stimulus category. At each visual field location, a softmax function was applied to the different function values from each category to obtain a probabilistic interpretation in percent of stimulus preference at that location. Finally, the baseline percentage (100%/number of categories) was subtracted from each value to facilitate the distinction between positive and negative deviations from uniformity.
As Figure 5 (left) shows, color preference could be predicted significantly better than chance in several ROIs. This means that individual areas do indeed have large-scale, retinotopically mapped color biases. The five significant ROIs in this analysis were the same as those where significant classification of color across brains (BSC) was observed (all p < 0.01): V1 (38.4%, 95% CI [37.8%, 39%]), V2 (36%, 95% CI [35.4%, 36.6%]), V3 (36.9%, 95% CI [36.2%, 37.5%]), hV4 (35.3%, 95% CI [34.4%, 36.2%]), and LO1 (37.7%, 95% CI [36.7%, 38.7%]). Figure 5 (right) shows the corresponding results for luminance. Here, voxel preference could be predicted with SRMs better than chance in area V1 (54.6%, 95% CI [54.0%, 55.2%]), in area V2 (55.0%, 95% CI [54.4%, 55.6%]), in area hV4 (51.7%, 95% CI [50.8%, 52.6%]), and in area LO1 (53.9%, 95% CI [52.9%, 55.0%]), all p < 0.01. Comparing the results with those from the BSC analysis (Fig. 3), we find that, except for areas V3 and LO1, voxel preference prediction results and BSC results are in line with each other in the remaining areas.
Preference prediction results. SRM-based predictions of voxel preferences across different brains for color (left) and luminance (right). Accuracy is the fraction of correctly classified voxels across all brains and error bars represent 95% CIs. Binomial tests were used to test if preference prediction accuracy was significantly larger than chance. Pairwise comparisons were made using independent z-tests. Asterisks denote significance at p < 0.05, Holm–Šidák corrected for seven ROIs.
To summarize across color and luminance, the results between this analysis and BSC are largely compatible. We therefore proceed with this SRM approach to visualize the retinotopic biases across the visual field that could explain the observed cross-brain decoding.
Stimulus preference across the visual field
Given the above statistical evidence for large scale, retinotopically mapped color biases in individual visual areas, we aimed to visualize them as illustrated in Figure 4B. We visualized the SRM-based predictions for individual voxels at different locations across the visual field by applying KDE on the cross-validated predictions made for each brain (see Materials and Methods, Voxel preference prediction) and converting them to probabilities using a softmax operation.
The resulting color preference maps can be seen in Figure 6. The maps in the large polar plots are based on the whole sample of brains. Note that these maps are based on models that were fit to and evaluated on independent datasets (i.e., cross-validated across brains) and that successful prediction of voxel preferences demonstrates the presence of reliable regional biases across datasets (i.e., brains). The small plots serve as an additional assessment of reliability. They are based on data from only the two halves of the sample after an odd–even split. This means that two SRMs were fit to the group data from the retinotopic mapping experiment, one for each half. Within each, the same leave-one-brain-out cross-validation, which was used previously for the whole group, then produced the maps in the insets. This reliability estimate is more conservative in that it is based on a split at the earliest stage of the analysis pipeline and thus complements the main results, which already show results that were cross-validated across participants from the whole-group SRM. Descriptively, the results from the whole sample showed a systematic relationship between visual field location and color preference. In V3, for instance, voxels representing parafoveal locations showed a preference for yellow. At intermediate eccentricities along the horizontal meridian, the analysis revealed a preference for green. More peripherally, our analysis showed a preference for red in the upper and lower visual fields. Visual inspection of the pairs of maps resulting from the odd–even split show good agreement in the ROIs where above-chance BSC was observed (Fig. 3), indicating sufficient reliability.
Color preference across the visual field. Color preference was visualized as explained in Figure 4B. Each of the three columns indicates the probability of a voxel to prefer the green, red, and yellow stimulus for every visual field location. The plotted values are the difference between this probability and the “baseline probability,” i.e., the probability that would be expected if it were distributed uniformly across the three stimulus categories, i.e., 33.3%. Positive (colored) values thus represent regions where the preference probability for a given color is larger than baseline, negative (gray) values represent regions where it is lower. Large plots show results for analysis using all participants’ datasets; small plots show results using only subsets of the data based on an odd–even split. Visual areas show characteristic dependencies between retinotopic location and color preference: V1, for instance, exhibits a preference for green or red near the fovea and less for yellow while the opposite is true at higher eccentricities. Note that color bars differ across ROIs to highlight changes in preference across the visual field at the level of individual brain regions.
Importantly, the maps differed between ROIs. For example, whereas in V3 the yellow stimulus was preferred at parafoveal locations, this was not the case in V1, where preference for yellow was more pronounced at more peripheral locations. These results speak against inheritance of large-scale color biases along the processing hierarchy and for area-specific properties, which will be examined in more detail in the following section.
Figure 7 shows the same results for luminance. Preference maps again differed across brain regions: While in V1 and V2 the preference for stimulation at high luminance decreased with increasing eccentricity for instance, the preference for high luminance stimulation in V3 was more pronounced in the lower visual field than in the upper visual field.
Luminance preference across the visual field. Same conventions as in Figure 6 except that preference is shown for high versus low intensity luminance. Since classification was binary, there is only a single column. Hot areas mark regions in the visual field where high luminance was preferred and gray areas mark regions where low luminance was preferred. In early areas V1–V3, for instance, luminance preference changed as a function of eccentricity. Note that although VO1 showed a difference between left and right visual field descriptively, preference classification in this area was not significant (Fig. 5). Note that color bars differ across ROIs to highlight changes in preference across the visual field at the level of individual brain regions.
In sum, the results showed that voxels at different retinotopic locations were predicted to prefer different stimulus classes. These differences were on the order of a few percent points. These results indicate a systematic relationship between a voxel's retinotopic spatial preference and its response pattern to color and luminance that was area specific and that could at least partially mediate between-subject classification.
ROI specificity of SRMs for voxel preference prediction
The topographies presented in Figures 6 and 7 showed variability between the different ROIs. This raises the question to what extent the retinotopic biases in stimulus preference captured by the SRMs were unique to each visual area and to what extent they were of a more generic nature and therefore interchangeable across areas for the prediction of stimulus preference. We examined this by comparing two analyses that used SRMs trained on retinotopic data: in the first, the SRMs used to transform the training brain data into the common space and to transform the predictions into the individual space of the left-out test brain differed in terms of the ROIs they were trained on, and results from all combinations of ROIs were averaged (incongruent condition). In the second, the SRMs for both transformations were trained on the same ROI (congruent condition—identical to our original analysis).
If SRMs were specific to each visual area, one would expect that the prediction of voxel stimulus preferences is better when SRMs from the same ROI were used for both transformations (congruent condition) than when each were based on SRMs from different ROIs (incongruent condition). If the models merely capture general retinotopic biases, one would expect that prediction accuracies would not differ. As is evident from Figure 8A (left), prediction accuracies for color were significantly higher when using congruent SRM pairings than when using incongruent ones (all p < 0.01) in all ROIs where prediction accuracies were significantly better than chance. The only exception to this was hV4 where, although the pairwise test was not significant (p = 0.1653), accuracy was significantly better than chance only for congruent pairings (p < 0.01) but not incongruent pairings (p = 0.0695). V2 was the only ROI where color preference prediction was significantly better than chance also for incongruent pairings (p < 0.01) while all remaining ROIs showed no significant accuracies. The same pattern of results was obtained for luminance (Fig. 8A, right). In all ROIs where luminance preference prediction was significantly above chance, accuracy was also significantly lower when incongruent SRM pairings were used (p < 0.01). Area hV4 again differed in this regard because, although the two conditions did not significantly differ (p = 0.2905), preference prediction accuracy was only significantly better than chance in the congruent condition (p < 0.01), but not the incongruent condition (p = 0.2218). For luminance, there was no ROI with significant above-chance prediction accuracy in the incongruent condition. A detailed examination of all possible pairs of training and test ROIs furthermore confirms that, if classification was significantly better than chance for a congruent pair, its accuracy value was higher than in any of the remaining pairs (Fig. 8B).
A, ROI specificity of SRMs for preference prediction. To examine whether the retinotopic biases for color or luminance captured in SRMs were specific for individual visual areas, we repeated the voxel preference prediction analysis (Fig. 4A) but compared transformation matrices from all visual areas in the test brain with respect to how well they could predict voxel preference in a given area. Dark bars show how well voxel preference could be classified when the transformation from the SRM of the same visual area was used for prediction as for training (“congruent,” same as bars in Fig. 5). Light bars show accuracy averaged across all classifications in which transformation into the individual space of the test brain was based on an SRM from a different visual area (“incongruent”). Accuracy is the fraction of correctly classified voxels across all brains and error bars represent 95% CIs. Binomial tests were used to test if observed accuracies exceeded chance, z-tests were used for comparisons between congruency conditions (asterisks denote p < 0.05, Holm–Šidák corrected for seven ROIs). B, Voxel preference prediction accuracies for all ROI pairs.
The discrepancies in prediction accuracies between congruent and incongruent SRM pairings demonstrate that SRMs capture retinotopic biases in stimulus preferences that are largely specific to individual visual areas.
SRM searchlight BSC analysis
We next tested how well common shared responses could be estimated from only anatomically aligned activity patterns for subsequent BSC (in contrast to our ROI BSC analysis where visual areas had been delineated functionally through retinotopy). Individual datasets were warped to MNI space separately and SRM was fit to only local patterns of retinotopic mapping responses in a searchlight analysis to estimate common space representations for every brain location. After transforming individual color responses to this common space, BSC could be carried out on those local patterns. This approach had the additional advantage that BSC could be applied to the whole brain.
Applying a false discovery rate of q < 0.05, we found that classification accuracies in a region within the early visual cortex was significantly larger than chance (one-tailed binomial test based on 3,240 Bernoulli trials). Figure 9 shows the location of this region relative to the hV4 group ROI. This ROI comprised all the voxels that were identified as being part of individual hV4 ROIs in at least 25% of the participants (green) or at least one participant (blue). As can be seen, the voxels where classification was significantly better than chance were located near the calcarine sulcus but did not overlap with the hV4 group ROI.
SRM searchlight map. SRMs were fit to patterns within local spheres (3 voxel radius) of brain responses to the retinotopic mapping stimulus. After transforming every participant's dataset into the common space, classifiers were used to predict the color of the viewed image using only the information in the local activity pattern (BSC). Cross-validation was performed by leaving out data from a different participant for testing on every iteration. Classification accuracies are shown in warm colors on a standard MNI template for brain regions that survived the whole-brain significance threshold of a binomial test (q < 0.05, FDR corrected). Green marks all voxels falling within area hV4 in at least 25% of the participants; blue marks all voxels that were in hV4 in at least a single participant. The regions where BSC of color yielded accuracies significantly above chance, namely, near the calcarine sulcus, were distinctly different from the location of hV4. This indicates that anatomical alignment was not sufficient beyond earliest visual areas for BSC (e.g., in hV4).
In sum, the searchlight analysis suggests that while anatomical alignment of voxels was sufficiently high for BSC of color in early visual cortex, this alignment was too coarse in more anterior regions.
Behavioral control analysis
In order to ascertain that the classification results were not driven by differences in task performance, we performed a Bayesian two-way repeated-measures ANOVA (Rouder et al., 2017) of the reaction time data under the factorially crossed three-level factor color and the two-level factor luminance. Reaction time datasets from Participants 2 and 9 were not complete due to programming error and had to be discarded from the analysis, yielding 13 participants in total. Residuals were normally distributed (six Shapiro–Wilk's tests, smallest test statistic across six conditions: W = 0.925, p = 0.2931), and no violation of sphericity was detected (Mauchly's test, p = 0.0845 for color, p = 0.6332 for the interaction). The Bayes analysis suggested that behavioral data were not affected by color or luminance: the resulting Bayes factors indicated that the data were BF10 = 1.273 times more compatible with the model assuming a main effect of color relative to no effect (red: M = 0.574 s, SD = 0.052 s, green: M = 0.582 s, SD = 0.058 s, yellow: M = 0.601 s SD = 0.067 s). As for luminance, the data were BF10 = 1.523 more likely to have occurred under the model assuming a difference between conditions compared with no difference (high: M = 0.595 s, SD = 0.054 s, low: M = 0.576 s, SD = 0.064 s). The data provided BF01 = 1.478 times more support to the model without interaction. Thus, even if a difference between color (and likewise luminance) conditions existed, it is too small for the present data to detect with sufficient certainty [1/3 < BF < 3 are considered “anecdotal evidence” (Jeffreys, 1961)].
Exploration of model robustness
Dependence of between-subject classification on number of shared features
Shared response modeling requires a parameter that controls the dimensionality of the shared common space. To understand how much variance needs to be explained by the model to enable above-chance BSC accuracy, we repeated the analysis using different numbers of shared components between 1 and 100. Figure 10 shows results for the classification of color (top panel) and luminance (bottom panel). The blue curves show how much variance is explained by the SRM components as a function of number of features, for retinotopic mapping data (solid lines) and for data of the main experiment (dashed lines). The former are generally higher than the latter because the model features are determined based on the retinotopic mapping data. More components naturally explain more variance in the training set. This also applies to the data from the main experiment, which were not used to fit the components. Classification accuracies (orange lines) increase with the number of shared features but plateau or decrease slightly from typically between 10 and 20 features onward. The peak of classification (orange) hence occurs approximately where 15–30% variance is explained in the training data (blue curve). This is also where a weakly pronounced “elbow” in the blue curves is visible. The results indicate that using more features does not necessarily lead to better classification accuracy and that accuracy peaks are typically found with low numbers of features that explain between 15 and 30% of the variance in the training set. The dependence of classification accuracy on the number of features emphasizes the need for an unbiased selection procedure of this parameter like the one we used in the main analysis.
Dependence of BSC accuracy on number of shared features. Top, Explained variance and decoding accuracy for the color data. Bottom, Same for luminance data. The x-axis denotes the number features k used for the SRMs fit to the retinotopic mapping data. The left y-axis (blue) denotes the percent variance explained by the features of the fitted models for the retinotopic mapping dataset (solid blue lines) and the color (luminance) responses from the main experiment (dashed blue lines). The right y-axis (orange) denotes the accuracy of the between-subject classification of color (luminance) and its 95% CI (Jeffreys interval; solid orange, see also Fig. 3). Chance level is indicated by the dotted orange line. A higher number of shared features resulted in more explained variance for the retinotopic dataset. This applies to the main experiment as well although it was not used for model fitting. Classification accuracy only increased with more shared features for low values but shows a downward trend beyond ∼10–20 features. This turning point coincides with a weakly pronounced “elbow” in the explained variance curves. Few shared features are thus sufficient to capture the information shared across observers. Note that the number of shared features in the main analysis (Fig. 3) was selected by grid-search on the training set.
Dependence of BSC accuracy on number of voxels
We examined to which extent the results from the BSC analysis were robust against changes in the number of voxels used for classification. That way the number of voxels could be held equal across ROIs, which otherwise typically differ due to differences in size between visual areas. Figure 11 shows the results for BSC of color (top) and luminance (bottom). Descriptively, classification accuracies tend to be lower compared with the results from the original BSC analysis (Fig. 3), as can be expected given the lower number of voxels. Classification accuracy in V1 is fairly robust in that it is significantly larger than chance across all numbers of voxels tested. In the majority of selection steps, classification accuracy was likewise significantly larger than chance in hV4 (4 out of 7) and LO1 (6 out of 7). Classification accuracy was more dependent on the number of voxels in V2 and V3. It exceeded chance level at a more lenient uncorrected significance threshold (V2: 6 out of 10; and V3: 5 out of 10). Accuracy was not above chance at any selection step in VO1 and in LO2 only at one step (uncorrected). BSC accuracy was thus largely robust across selection levels in V1, hV4, and LO1. In V2 and V3, above-chance classification was also possible with fewer voxels but it was more contingent on the number of voxels used for the analysis.
Dependence of BSC accuracy on number of voxels. Since ROIs differ in terms of the number of voxels they contain, an analysis was conducted with different numbers of voxels. Voxels within ROIs were ranked according to signal quality (F value from the Fourier-based analysis of the retinotopic mapping data) and only the best few voxels were included in the analysis. Curves represent classification accuracies as the mean percent correct classifications across the group and confidence bands are the 95% CI. Asterisks mark accuracies that are significantly larger than chance (p < 0.05, Holm–Šidák corrected for the number of ROIs, which could be smaller than seven because some ROIs did not contain enough voxels in all tested subjects) and daggers denote p < 0.05 (uncorrected).
As for luminance, classification accuracy in V1 similarly exceeds chance significantly at 9 out of 10 selection steps. In V2 it is significantly above chance in 6 out of 10. V3 showed less robustness with accuracy surpassing a more lenient uncorrected significance threshold at 5 out of 10 steps. In hV4 significant classification was observed at 3 out of 7 and in LO2 at 1 out of 9 voxel selection steps (uncorrected). Decoding thus was robust across selection steps in V1 and V2, whereas V3 decoding required larger numbers of voxels.
Retinotopic binning analysis
Finally, we carried out a more traditional analysis that had previously been used to test whether retinotopic bias might partially underlie the accuracy with which sensory input can be decoded from patterns of fMRI signals (Freeman et al., 2011; Beckett et al., 2012; Larsson et al., 2017). We therefore repeated the WSC analysis of color yet this time binned voxels based on (1) their eccentricity preference and (2) their polar angle preference, each of which preserves to some extent the retinotopic structure in the respective coordinates. Voxel values were averaged within bins and classification was carried out on these bin averages. Accuracies were compared with accuracies obtained with random binning of voxels. The p values from the resulting paired t tests were corrected for multiple comparisons across binning steps using Holm–Šidák correction.
Figure 12A shows that for binning based on eccentricity, decoding accuracies were significantly better for at least some binning steps in all examined ROIs except VO1 compared with random binning. For binning based on polar angle (Fig. 12B), we observed significantly higher classification accuracies in V1 and V2 only. We note that these retinotopic binning-based results are overall consistent with those of the SRM analysis shown in Figure 3. In particular, all ROIs where prediction of color from shared response patterns was significantly better than chance also showed better classification after retinotopic binning than random binning, be it based on polar angle, on eccentricity, or on both. Retinotopic binning yielded stronger results only in LO2 where classification accuracy was higher than random binning at two binning steps (after correction) whereas SRM-based results in LO2 between brains was not significantly better than chance after FWE correction across ROIs. To conclude, retinotopic binning analysis is in good agreement with the results from our BSC analysis and confirms the presence of spatial retinotopic biases in color and luminance encoding in the human brain.
Binning analysis. Voxels within a ROI were assigned into a number of bins based on their polar coordinates, i.e., either eccentricity or polar angle. Bin limits were chosen such that they were equally spaced across the range of voxel coordinates. Different numbers of bins were used. Within-subject classification was performed except that, instead of training and testing classifiers on vectors of voxel responses, vectors of bin-wise averages of voxel responses were used. In a control analysis, voxels were randomly assigned to the bins. If a retinotopic bias exists, one would expect that retinotopically based binning would yield higher classification accuracy than random binning. Paired t tests (df = 14) were used, and error bands represent 95% bootstrapped CI. Asterisks denote significance at p < 0.05 (Holm–Šidák corrected for 15 binning steps). In all ROIs where color could successfully be classified across subjects in the BSC analysis, the binning analysis also showed a retinotopic bias when binning based on eccentricity or polar angle (or both, see V1 and V2). This confirms the results from our own analysis. In addition, retinotopic binning yielded higher accuracies than random binning also in LO2.
Discussion
Shared response modeling allows experimenters to test for between-subject commonalities in the neural representations of sensory stimuli. Typical applications involve sampling large neural response spaces from individual observers by presenting them with a vast variety of stimulus input (e.g., movies). Knowing how different brains respond to such stimulation makes it possible to match idiosyncratic response spaces to each other and to identify neural representations that are shared across observers. We applied SRM to brain activity measured while observers viewed simple retinotopic mapping stimuli, not movies. We show that controlling for how different brains respond to the same, purely spatially defined mapping stimulus is sufficient to preserve color responses across brains while being specific for distinct areas across the visual processing hierarchy. Our results suggest a relationship between visual space and color that is shared across observers and unique to each visual area. This conclusion is supported by three key observations.
First, we fit SRMs to retinotopic mapping data to map individual fMRI patterns into a shared common space. Classification analysis in this space showed that, in areas V1, V2, V3, hV4, and LO1, it was possible to predict what color an observer was seeing even though classifiers were trained exclusively on responses from other observers’ brains (Fig. 3). This is in line with previous reports of their role in color vision (Lueck et al., 1989; Engel et al., 1997; Beauchamp et al., 1999; Bartels and Zeki, 2000; Brouwer and Heeger, 2009). We found no behavioral differences between stimulus conditions that could explain these results. In conclusion, this demonstrates that aligning individual response spaces across brains with respect to their retinotopic functional organization is sufficient to afford classification of color in multiple visual areas across different brains.
Second, we hypothesized that the systematic relationship between retinotopic and color responses captured by SRMs would allow them to predict how individual voxels would respond to the color stimuli in an fMRI dataset recorded from a completely independent brain. Similar methods had previously been used to predict broader brain topographies like the location of FFA and PPA (Haxby et al., 2011). SRM-based prediction accuracies at this task were indeed above chance in various ROIs (Fig. 5). Similarly, a previous study presenting “color-preference images [similar to the orientation-preference maps of Kamitani and Tong (2005)]” found no evidence for regional bias either (Fig. 11; Parkes et al., 2009). This seemingly contradicts our observation that voxel preferences could be predicted across brains. This disparity is easily explained if one considers that SRM separates individual participants’ fMRI responses into shared and idiosyncratic components. While retinotopic biases exist in the shared component, it is conceivable that at the individual level they remain obscured by idiosyncratic response characteristics. Likewise, retinotopic biases as an explanation of orientation decoding (Freeman et al., 2011) were demonstrated with an analysis like our binning test (Fig. 12) whereas they had not been apparent from individual weight maps (Kamitani and Tong, 2005).
Third, we quantitatively investigated if the relationships between retinotopy and color responses captured by SRMs were specific to different ROIs. We therefore compared how accurately voxel responses to color could be predicted when transformation matrices from SRMs belonging to either the same or different areas were used for model fitting and prediction, respectively. We found that prediction accuracies dropped considerably when SRMs based on different visual areas were used (Fig. 8). This specificity is consistent with visual inspection showing marked differences between topographies from different brain areas (Fig. 6). These inter-area differences are important as they rule out alternative explanations like artefactual spatial inhomogeneities, e.g., in the stimulus projection, because, in that case, one would expect identical topographies across ROIs.
The decreasing preference for red and green stimuli in V1 at larger eccentricity is consistent with previous fMRI findings reporting decreasing responsiveness to modulations in the red/green direction in color space, but not for the blue/yellow direction (Vanni et al., 2006; Mullen et al., 2007). Similarly, behavioral measurements showed that, compared with blue-yellow and achromatic mechanisms, contrast sensitivities of the red-green cone mechanism are highest at the fovea, yet show the strongest decline as eccentricity increases (Mullen, 1991; Mullen and Kingdom, 2002; Hansen et al., 2009). Although our stimuli did not systematically sample color space and did not include blue, our V1 findings are compatible with behavior. Such eccentricity differences may be related to the observation that cone density decreases with eccentricity (Curcio et al., 1990) or that a fourth (melanopsin) photopigment may contribute to peripheral color vision (Horiguchi et al., 2013). Furthermore, S cones are absent at foveal eccentricities <0.15° (Williams et al., 1981; Curcio et al., 1991), which, however, when compared with foveal voxel bandwidths (∼0.5°), suggests only small effects on fMRI (Dumoulin and Wandell, 2008). However, the distributions of L and M cones (in contrast to S cones) show large variability across individuals (Brainard, 2015) so that commonalities at the receptor level likely cannot fully explain the cortical effects.
V2, unlike V1, shows no preference for green in the fovea but only to the left and the right of it around the horizontal meridian while green is relatively less preferred along the upper vertical meridian and yellow is less preferred along the horizontal meridian. In V3, the fovea has less preference for green while, in every quadrant, there is a patch each surrounding it where green is preferred. The opposite pattern is found for red and, to lesser extent, for yellow, which are both more preferred in the center and more peripheral regions of the visual field. Area hV4 shows higher preference for red than green or yellow in the center, which drops off ventrally and to its left and right where yellow and green are preferred. Finally, LO1 shows more heterogeneity in preference in the upper visual field. These descriptive differences between visual areas are consistent with our finding that stimulus preference can be predicted more accurately when SRMs from the same visual area are used to train models and derive predictions.
Similar variability between areas has been observed using fMRI with respect to spatial sensitivity for chromatic and achromatic input which differentially covaries with eccentricity (Welbourne et al., 2018). In addition to eccentricity, human neuroimaging studies—although not specifically addressing color vision—demonstrated that polar angle influences visual processing (Liu et al., 2006; Benson et al., 2012; Silson et al., 2018; Silva et al., 2018). It is hence conceivable that the topographies in extrastriate areas are related to different behavioral and cone mechanisms than V1, involving a combination of both eccentricity and polar angle differences. While it is known that visual perception of spatial frequency and orientation depends on polar angle (Karim and Kojima, 2010; Jóhannesson et al., 2018), the empirical findings for color are mixed (Levine and McAnany, 2005; Danilova and Mollon, 2009). At the retinal level, cone densities change depending on polar angle in that they are higher along the horizontal than the vertical meridians (Curcio et al., 1990; Song et al., 2011). But at least for orientation discrimination, simulations suggested that these cone differences cannot fully explain that polar angle affects visual performance (Kupers et al., 2019).
The color biases presented here complement similar fMRI findings involving other visual features like orientation (Sasaki et al., 2006; Freeman et al., 2011, 2013; Beckett et al., 2012; Larsson et al., 2017), and we predict that retinotopy-based SRM would also allow between-subject classification of these features. Concerning color, spatial biases in the visual system are known from research on mice in which stimuli appearing above or below the horizon carry different behavioral meaning (Baden et al., 2013; Rhim et al., 2017; Denman et al., 2018). Environmental regularities like the chromaticities of natural daylight (Lafer-Sousa et al., 2012) or of objects versus their backgrounds (Rosenthal et al., 2018) shape color vision in primates, too. Which environmental adaptations may underlie our findings requires further research.
We point out that our stimuli do not only probe the retinotopic layout of cortical field maps. That is because they are composed of more low-level spatial features like contrast, motion, and orientation, which may have shared representations with color (Leventhal et al., 1995; Johnson et al., 2001, 2008; Friedman et al., 2003; Sincich and Horton, 2005; Chang et al., 2017; Garg et al., 2019). SRM may therefore also exploit commonalities in the corepresentations of these features, thereby contributing to our results.
Finally, it has been demonstrated that response biases for orientation can be changed by embedding a grating stimulus in a modulator, which could render the orientation either radial or tangential (Roth et al., 2018). Color vision, on the other hand, shows some robustness in that different colors elicit patterns of brain activity that are similar between sensory-driven colors and memory colors (Bannert and Bartels, 2013; Vandenbroucke et al., 2016; Teichmann et al., 2019) and across tasks (Serences et al., 2009; Bannert and Bartels, 2018). In sum, given the evidence for both flexibility and stability of color representations, it would be interesting for future studies to probe the plasticity of the observed large-scale response biases.
Footnotes
- Received October 24, 2020.
- Revision received July 30, 2025.
- Accepted August 1, 2025.
-
This work was supported by the German Excellence Initiative of the German Research Foundation (DFG) grant number EXC307, by the Max Planck Society, Germany, and by DFG grant SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, project 9, project number: 276693517.
-
The authors declare no competing financial interests.
-
This paper contains supplemental material available at: https://doi.org/10.1523/JNEUROSCI.2717-20.2025
- Correspondence should be addressed to Michael M. Bannert at michael.bannert{at}tuebingen.mpg.de or Andreas Bartels at andreas.bartels{at}tuebingen.mpg.de.
- Copyright © 2025 the authors
