Introduction
To successfully deal with everyday activities, people must adaptively
monitor the environment, focus on goal-relevant information, and shield
irrelevant interferences. The cognitive function to select and keep the
relevant representation and process as the focus of consciousness is
attention (Posner, 1989). Attention is fragile to brain
dysfunctions , such as autism spectrum disorder (Mash et al. ,
2020), attention deficit hyperactivity disorder (Arora et al. ,
2020), schizophrenia (Backes et al. , 2011), and Parkinson’s
disease (Boord et al. , 2017; Yang et al. , 2021).
Accurately assessing attention deficits is thus fundamental for
developmental and clinical neuroscientific research and practice.
The attention network test (ANT), a popular tool to studyattention deficits (de Souza Almeida et al. , 2021), builds upon
the triple-network taxonomy proposed by Posner and colleagues (Posner,
1989; Fan et al. , 2002; Fossella et al. , 2002). The
triple-network taxonomy considers the attention system to comprise three
isolable networks: alerting, orienting, and the executive network. Since
the original ANT developed by Fan and colleagues in 2002, the behavior
task form has evolved into several variants in the last twenty years
(see de Souza Almeida et al., 2021 for a comprehensive review). The
classical fMRI version of ANT was published in 2005, demonstrating that
the three task contrasts corresponded to distinct brain activations (Fanet al. , 2005). Xuan and collaborators used the revised ANT task
(ANT-R) in 2016 to investigate the interaction among the attention
networks (Xuan et al. , 2016). However, due to its simplicity and
interpretability, the original fMRI ANT task is still prevalent in
neuroscience studies (Backes et al. , 2011; Fan et al. ,
2012; Daamen et al. , 2015; Hubbard et al. , 2015; Neufanget al. , 2015; Boord et al. , 2017; Ikeda et al. ,
2017; Dash et al. , 2019; Kwak et al. , 2019; Neufanget al. , 2019; Hao et al. , 2021; Yang et al. , 2021).
According to the classical test theory, reliability refers to the
proportion of variance due to the “real” value of the interested
dimension in the raw score variance (Hedge et al., 2018). As the
“real” value is unknown, reliability is often estimated through the
test-retest or split-half method (Zhang et al., 2023).Intra-class correlation
coefficient (ICC), which estimates the ratio of between-subject
variance, is a popular measure of reliability (Stoffel et al.,
2017). Higher ICC values mean less measurement error and better
stability over time. Thus, reliability is crucial for a measure used in
individual difference studies. Previous studies have suggested that the
reliability of behavioral attentional network scores falls below
expectations. For example, a psychometric analysis of 15 unique studies
suggested that split-half reliability is moderate-to-high for executive
control but low for alerting and orienting scores (Macleod et
al. , 2010). Hahn et al. (2011) reported similar test-retest reliability
estimates using a sample of schizophrenia and healthy controls (Hahnet al. , 2011). Ishigami and collaborators compared the stability
and reliability of network scores between ANT and ANT-I (i.e., Attention
Network Test-Interactions). Their study found that despite the
group-level result being stable and robust after ten sessions, the
reliability of ANT scores was unsatisfactory (Ishigami & Klein, 2010).
The inadequate reliability of the
attention network scores might be mainly due to the nature of the
difference score approach (Draheim et al. , 2019). The
reliability of the difference score can be poor even when the
experimental effect is robust at the group level (Hedge et al.,
2018) . There are two reasons. First, researchers often design a
cognitive task for testing the overall experiment effect rather than
individual differences. Thus, minimizing between-subject variance is
often the goal of design optimization. Second, the experiment and
control conditions are usually correlated. The variance of difference
score is the sum of the unique variance of each condition and their
error variances. Because the shared variance is eliminated due to
subtraction, the variance of difference scores reflected mainly the
measurement errors if the two conditions were highly correlated.
Similarly, fMRI studies often used the general linear model to
measure task-specific brain activations by contrasting experimental and
control conditions. The difference score approach and its drawback thus
also apply to task-fMRI. Even if the spatial contrast map can be robust
and reproducible, the high collinearity between experimental and control
conditions might restrict the reliability of fMRI contrast (Infantolinoet al. , 2018). In addition, the cross-trial variability is
commonly larger than between-subject variability in fMRI studies, which
might further undermine the reliability of fMRI estimates (Chen et
al. , 2021). Mounting evidence has indicated that the reliability of
fMRI measurement is inadequate to be used as sensitive biomarkers
(Herting et al. , 2018; Elliott et al. , 2021; Nobleet al. , 2021; Klingelhofer-Jens et al. , 2022). In a
meta-analysis of 90 experiments, authors concluded that the overall
reliability is poor for frequently used task-fMRI measures (Elliottet al. , 2020). Meta-analysis on brain connectivity fMRI measures
yielded similar findings (Noble et al. , 2019). Recent years have
manifested increasing research efforts to examine the test-retest
reliability for widely used fMRI tasks (Korucuoglu et al. , 2020;
McDermott et al. , 2021; Haller et al. , 2022; Kennedyet al. , 2022; Wuthrich et al. , 2023). However, regarding
its widespread usage in developmental and clinical neuroscience, the
reliability of the fMRI measurement of ANT receives insufficient
research attention. Characterizing the reliability profile of ANT is
fundamental for further research interested in using ANT to reveal
individual or group differences.
The three attention network components, defined by the original
ANT fMRI task, parallel the well-known intrinsic networks defined by
resting-state fMRI (Yeo et al. , 2011; Smith et al. , 2013;
Schaefer et al. , 2018). The intrinsic network structure
corresponds to the functional domains and task-evoked co-activation
network (Smith et al. , 2009; Nickerson, 2018). Several intrinsic
network components, especially the frontoparietal control, dorsal
attention, and ventral attention, are related to attention and executive
control function (Dixon et al. , 2018). However, how the
“attention network” defined by ANT is related to the attention-related
“intrinsic network” components remains elusive. A recent study
investigates the correspondence between ANT attention network contrast
(alerting, orienting, and executive) and intrinsic network components
(Markett et al. , 2022). That study demonstrated that the
three-attention network of ANT recruited multiple and partly overlapping
intrinsic networks and converged in the dorsal fronto-parietal and
midcingulo-insular network (Markett et al. , 2022). As far
as we know , no study has examined whether reliability estimates
varied by intrinsic network components. If the ANT fMRI task could
measure attention, the reliability estimates for ANT fMRI should vary
according to the relevance to visual attention.
The present study investigates
the test-retest reliability of the classical fMRI version attention
network test from the perspective of intrinsic network structure.Because of the relevance of basal ganglia (van Schouwenburg
et al., 2015) and the thalamus (Fan et al., 2005) in
attention function, we
also include a separate analysis of subcortical structures. We utilized
an openly available ANT fMRI dataset for participants with Parkinson’s
disease and healthy elderly (Day et al. , 2019). Firstly, we
estimated the test-retest reliability for the behavioral measures
because no study has reported the ICC estimates for the attention
network scores of the fMRI ANT task. Then, the reproducibility of
group-level activations across sessions and participant groups and the
test-retest reliability at the individual level were examined at the
voxel, region, and network levels. The intrinsic network was defined
using the Yeo-Schaefer atlas. The present study can provide
comprehensive reliability information for researchers and clinicians
interested in using the classical fMRI version of ANT.