\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage{natbib}




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[ngerman,english]{babel}



\usepackage{float}








\begin{document}

\title{External Validity in Distributed Data Networks}



\author[1]{Michael Webster-Clark}%
\author[2]{Sengwee Toh}%
\author[3]{Jonathan Arnold}%
\author[3]{Kathleen M. McTigue}%
\author[4]{Thomas Carton}%
\author[1]{Robert Platt}%
\affil[1]{McGill University Department of Epidemiology Biostatistics and Occupational Health}%
\affil[2]{Harvard Medical School Department of Population Medicine}%
\affil[3]{University of Pittsburgh Department of Medicine}%
\affil[4]{Louisiana Public Health Institute}%


\vspace{-1em}



  
  \date{June 10, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup





\selectlanguage{english}
\begin{abstract}
Purpose: While much has been written about how distributed networks
address internal validity, external validity is rarely discussed. We
aimed to define key terms related to external validity, discuss how they
relate to distributed networks, and identify how three networks (the US
Food and Drug Administration's Sentinel System, the Canadian Network for
Observational Drug Effect Studies {[}CNODES{]}, and PCORnet, the
National Patient Centered Clinical Research Network, initiated and
supported by the Patient-Centered Outcomes Research Institute. Methods:
We define external validity, target populations, target validity,
generalizability, and transportability and describe how each relates to
distributed networks. We then describe Sentinel, CNODES, and PCORnet and
how each approaches these concepts. Results: Each network approaches
external validity differently Sentinel answers regulatory questions in
the general US population using data from commercial health plans and
Medicare fee-for-service beneficiaries and considers external validity
when exploring outliers or performing subgroup analyses to examine
potential heterogeneity of treatment effects. CNODES focuses on a
Canadian target population but includes UK and US data and thus has to
make decisions about which partners can be included in each analysis.
PCORnet supports a wider array of studies including randomized trials
and often assesses whether a given study will be representative of the
wider US population. Conclusions: There is no one-size-fits-all approach
to external validity within distributed networks. With these networks
and comparisons between their findings becoming a key part of
pharmacoepidemiology, there is a need to adapt tools for improving
external validity to the distributed network setting.%
\end{abstract}%



\sloppy


\textbf{Title:} External Validity in Distributed Data Networks

\textbf{Type of article:} Original article

\textbf{Target journal:} Pharmacoepidemiology and Drug Safety

\textbf{Authors:} Michael Webster-Clark,\textsuperscript{1,2} Sengwee
Toh,\textsuperscript{3} Jonathan Arnold,\textsuperscript{4} Kathleen M.
McTigue,\textsuperscript{4} Thomas Carton,\textsuperscript{5} Robert
Platt\textsuperscript{1}

\textbf{Affiliations:}

\begin{enumerate}
\tightlist
\item
  Department of Epidemiology and Biostatistics, McGill University,
  Montreal, QC
\item
  Department of Epidemiology, Gillings Schools of Global Public Health,
  UNC Chapel Hill, NC
\item
  Department of Population Medicine, Harvard T.H. Chan School of Public
  Health, Boston, MA
\item
  Department of Medicine, University of Pittsburg, Pittsburgh, PA
\item
  Louisiana Public Health Institute, New Orleans, LA
\end{enumerate}

\textbf{Corresponding author information:}

Michael Webster-Clark, PharmD, PhD

Department of Biostatistics, Epidemiology, and Occupational Health,
McGill University

Montreal, QC H3A 1G1

Email: mawc@live.unc.edu

Phone: 1 919 966 7433

Fax: 1 919 966 2089

\textbf{Word count: 2906/3000}

\textbf{Funding statement:} This project had no direct funding.

\textbf{Conflicts of interest:} No conflicts of interest to report.

\textbf{Ethics approval and patient consent:} This project did not
involve any patient data.

\textbf{Key points: (max 100 words)}

1. Distributed networks are rapidly becoming a cornerstone of
pharmacoepidemiologic research.

2. Given their use of multiple real-world study populations, external
validity is an essential but under-investigated part of how these
networks generate data on drug safety and effectiveness.

3. How Sentinel, CNODES, PCORnet, and other networks approach external
validity is a function of their official purposes, the study questions
they answer, and the data partners they include.

4. Formal statistical methods to improve external validity of estimates
are rarely implemented within these networks for a variety of reasons.

\textbf{Abstract: 247/250}

\textbf{Purpose:} While much has been written about how distributed
networks address internal validity, external validity is rarely
discussed. We aimed to define key terms related to external validity,
discuss how they relate to distributed networks, and identify how three
networks (the US Food and Drug Administration's Sentinel System, the
Canadian Network for Observational Drug Effect Studies {[}CNODES{]}, and
PCORnet, the National Patient Centered Clinical Research Network,
initiated and supported by the Patient-Centered Outcomes Research
Institute.

\textbf{Methods:} We define external validity, target populations,
target validity, generalizability, and transportability and describe how
each relates to distributed networks. We then describe Sentinel, CNODES,
and PCORnet and how each approaches these concepts.

\textbf{Results:} Each network approaches external validity differently
Sentinel answers regulatory questions in the general US population using
data from commercial health plans and Medicare fee-for-service
beneficiaries and considers external validity when exploring outliers or
performing subgroup analyses to examine potential heterogeneity of
treatment effects. CNODES focuses on a Canadian target population but
includes UK and US data and thus has to make decisions about which
partners can be included in each analysis. PCORnet supports a wider
array of studies including randomized trials and often assesses whether
a given study will be representative of the wider US population.

\textbf{Conclusions:} There is no one-size-fits-all approach to external
validity within distributed networks. With these networks and
comparisons between their findings becoming a key part of
pharmacoepidemiology, there is a need to adapt tools for improving
external validity to the distributed network setting.

\textbf{Plain language summary:}

Much work has discussed how studies that partner with multiple separate
databases can address things that create bias for estimates from
specific databases. Little attention has been paid, however, to how
these studies consider differences in characteristics between the
databases and specific groups of people treatment effects may be
relevant to (sometimes referred to as ``external validity''). This work
describes various concepts related to external validity and how they
relate to these multi-database studies, discusses how Sentinel, CNODES,
and PCORnet (three such database partnerships) address these concepts,
and considers how each of their approaches reflects the structure and
purpose of that specific partnership.

\textbf{Introduction}

Legal rules and regulations keep healthcare data secure and prevent
violations of patient privacy.\textsuperscript{1} While necessary, these
precautions make it difficult to obtain permission to combine data from
multiple sources, increase the time required to conduct straightforward
analyses and make more complex analyses
impossible.\textsuperscript{2-4}As a result, data on population-level
drug safety and effectiveness generally come from a smattering of
single-database studies with limited precision and in-sample diversity
applying differing analytic approaches and statistical
analyses.\textsuperscript{5}

As healthcare data were digitized and information technology advanced,
an alternative approach was proposed: analyses using distributed data.
In 2008, the FDA launched the Sentinel Initiative to explore a system
where database custodians, called ``partners,'' maintained ownership of
their data as separate ``nodes'' of the network but transformed it into
a common data model to be analyzed in a consistent
way.\textsuperscript{6} A similar effort started in Canada with the
Canadian Network for Observational Drug Effect Studies
(CNODES),\textsuperscript{7, 8} formally funded in 2011, and the
Patient-Centered Outcomes Research Institute (PCORI) began to design its
own distributed network of partner organizations, PCORnet (the National
Patient Centered Clinical Research Network), in
2013.\textsuperscript{9}All three networks focus on generating one
``network-wide'' effect estimate in some fashion from the node-level
data. Other distributed networks include the Data Analysis and Real
World Interrogation Network (DARWIN-EU) project in
Europe;\textsuperscript{10} a network that leverages the infrastructure
built by the Observational Health Data Sciences and Informatics (OHDSI)
community;\textsuperscript{11} the Asian pharmacoepidemiology network
(AsPEN);\textsuperscript{12}the Vaccine Safety Datalink
(VSD);\textsuperscript{13} and a distributed network created for the
purposes of pregnancy research titled ConcePTION.\textsuperscript{14}

Much has already been written about the steps these and other networks
take to reduce confounding and information
bias\textsuperscript{15-1718}in analyses within the individual nodes;
after all, internal validity within nodes is necessary to generate
unbiased estimates in nonexperimental research.\textsuperscript{19}
Concepts related to external validity -- such as effect measure
modification, target populations, generalizability, and transportability
-- have received comparably less attention in methodologic work on
distributed data. Here, we describe the unique roles external validity
and related concepts play in analyses of distributed data networks,
especially those that seek to obtain a single ``network-wide'' effect
estimate. We then provide an overview of the structure of Sentinel,
CNODES, and PCORnet and describe how each network deals with these
concepts.

\textbf{Methods}

\emph{Key terms and definitions.} In the past decade, researchers have
formalized many concepts related to external validity in the context of
single-site studies.\textsuperscript{20-23} This has resulted in clearer
definitions for ``study population'', ``target population'',
``generalizability'', and ``transportability.'' The study population is
the population supplying outcome, exposure, and covariate data for the
analysis. The target population is a specific population in which
researchers hope to estimate the effect of exposure on outcome, which
may or may not be the study population. Generalizability is the extent
to which a study population can be used to estimate a given effect in a
specific target population it was sampled from (see \textbf{Figure 1A}
). Transportability is the extent to which a study population can be
used to estimate a given effect in a specific target population it
was\textbf{not} sampled from \textbf{(} see \textbf{Figure 1B} ). This
formalization has also resulted in the conceptualization of ``target
validity'', defined as how well a given study estimates a treatment
effect in a specific target population. Target validity takes into
account internal validity, for the study population, and external
validity, for the specific target population. This formalization has
also helped clarify understanding of the connection between external
validity and effect measure modifiers (i.e. variables that are
associated with different treatment effect estimates on the scale of
interest)\textsuperscript{24} as well as ways to use analytic methods to
``balance'' effect measure modifiers between study and target
populations.\textsuperscript{25, 26}

\emph{Distributed networks and external validity.} The structure of
distributed data networks (presented in \textbf{Figure 2} ) makes these
terms even more important. Unlike most epidemiologic research, there are
multiple study populations -- in fact, there are at least as many study
populations as there are nodes in the distributed data network. Because
of random error or differing degrees of confounding, effect estimates at
the various nodes may not agree with one another. Of course, different
distributions of effect measure modifiers at each node can also lead to
differing effect estimates. Each node may represent a separate
geographical region, a different insurance provider, or an entirely
different type of data source (e.g. commercial claims vs electronic
health records). It is entirely possible that an estimate in one node
may not be externally valid, or may have poor target validity, for the
population represented by another node. This is not a major concern if
researchers report only node-level estimates and make few comparisons
across nodes. The estimate of the treatment effect within each node that
makes the fewest assumptions will generally be the estimate yielded by
implementing the research study within that node.

If researchers want to combine results from the various nodes into one
``network-wide'' estimate and 95\% confidence interval, which is usually
the case, or assess whether different nodes exhibited different amounts
of confounding, however, external validity becomes an important
consideration when planning research. The target population for the
combined network-level analysis affects the analytic strategies that can
be used, both at individual sites and when combining results. Whether
the findings of a given node can generalize to the whole network becomes
an important consideration, as does the extent to which node-level
estimates can be transported to one another or to external target
populations. Finally, the network needs to consider whether their
combined estimate has target validity for the population of interest.

\emph{Specific networks and external validity.} We assessed how three
major distributed data networks that specifically focus on creating
network-wide estimates (Sentinel, CNODES, and PCORnet) address external
validity in routine aggregate data projects. Researchers from each
network provided background information on its purpose and general
structure and answered questions related to external validity, target
populations of interest, generalizability, transportability, and target
validity.

\textbf{Results}

\emph{Describing the various networks.} \textbf{Table 1} summarizes key
aspects of each network, including the number of individuals covered and
the types of data partners included, the format of the data, the main
users that can query the data partners, how analyses are generally
conducted, and how the results of the analyses are compiled and
combined.

Each network was designed to facilitate rigorous and reproducible public
health research. Sentinel and CNODES were specifically created to help
government agencies answer questions about drug safety and
effectiveness, while PCORnet was created as a resource for a wider
variety of stakeholders, including those interested in conducting
pragmatic trials (i.e. randomized controlled trials including patients
and conditions more in line with ordinary clinical practice than typical
in randomized controlled trials).\textsuperscript{27} All three include
a wide array of data partners providing claims and electronic health
record data and each network has developed or implemented its own common
data model (i.e., standardized way to store data to prepare for
analysis).\textsuperscript{28} Sentinel has pursued the most
standardized analytic approach, with almost every project using
customizable and reusable SAS code created entirely within the
operations center.\textsuperscript{29} The CNODES coordinating center
prepares a high-level protocol and analysis plan implemented with code
at each data partner for most projects, but also uses the Sentinel
common data model. PCORnet allows even more flexibility with the
potential for analyses to be performed centrally. Finally, the networks
all allow for variation in the extent to which results are aggregated,
with PCORnet once more allowing the most flexibility.

Sentinel Q\&A:

\emph{How does external validity come into play when planning and
conducting projects in the database?} A typical Sentinel analysis
includes data from multiple large national and regional commercial
health plans, a state Medicaid plan, and Medicare fee-for-service plan.
Given the demographic and geographic diversity of the source data and
the focus on internal validity, external validity is typically not an
explicit consideration when planning or conducting Sentinel projects.
That said, attention is paid to the relevance of the study population
when finalizing the analysis plan and deciding on eligibility criteria
(e.g. high risk individuals, true new users of the drug). These
considerations also shape whether specific data partners are included in
the analyses. Finally, if outlier results are observed from some data
partners, the operations center may examine how the outlier populations
differ from the other study populations.

\emph{What target population, if any, underlies most analyses?} For the
most part, the FDA is focused on studying treatment effects on
individuals ``treated'' with drugs in the United States. The major gap
with respect to this target is the lack of data on the uninsured and
individuals with Medicaid, but there are ongoing efforts to fill this
gap, such as the inclusion of more Medicaid data.

\emph{Are there ways to generalize the findings of the nodes to the
network?} Depending on the specific analysis, the operations center can
always combine the outcome and treatment data across the sites to obtain
a ``network-wide'' effect estimate. It is also possible to include a
specific portion of the network in an analysis, e.g., running the
analysis only in the Medicare fee-for-service data when examining the
treatment effects in the elderly.

\emph{How easily can node-specific estimates be transported between
nodes or to external populations?} While it is theoretically possible to
standardize node-specific estimates to one population (e.g., the entire
US population), it has not yet been implemented within Sentinel.
However, Sentinel routinely performs pre-specified subgroup analyses to
identify treatment effect heterogeneity.

\emph{Are choices ever made to maximize target validity, rather than
internal validity or precision?} Given that all the data partners are
from the US, and the general focus is on general US population treated
with the medications, requesters generally prioritize internal validity
and precision..

CNODES Q\&A:

\emph{How does external validity come into play when planning and
conducting projects in the database?} Just as with Sentinel, there is no
explicit consideration of external validity during research. Still, the
query refinement process helps establish whether limiting to specific
groups of people (e.g. true new users of drugs, patients at high risk of
the outcome) would reduce the relevance of the analysis, and whether
some specific provinces and data partners like CPRD or MarketScan differ
enough from the others in covariates and follow-up distribution to
ultimately limit their use of the research question. External validity
also comes up indirectly when combining the results, identifying
outliers, and conducting meta-analyses.

\emph{What target populations, if any, underlies most
analyses?}Typically, the main target population of interest is the
overall Canadian population. As a result of the fact that 97\% of
Canadian citizens reside in the provinces contributing to CNODES, the
population represented in the Canadian portion of CNODES analyses and
the Canadian target population are very similar; restrictions on drug
coverage can change this for some analyses, however.

\emph{Are there ways to generalize the findings of the nodes to the
network?} The main way that findings are generalized from site-specific
estimates to the broader network is typically by random-effects
meta-analysis using inverse variance weighting.\textsuperscript{30} When
using exclusively the Canadian portions of the network, this means that
provinces with more events (and likely more individuals) tend to
contribute more to the overall effect estimates. No attempt is made to
generalize the results of each site, however, and effects are generally
assumed to be constant across sites unless there is substantial
heterogeneity.

\emph{How easily can node-specific estimates be transported between
nodes or to external populations?} Differences in demographics, the
services and medications covered by each province, and the calendar time
intervals each data source contributes can make it difficult to directly
transport effect estimates between provinces. Because all these
variables are measured, however, analytic methods like inverse odds
weights or G-computation may be used to obtain more precise
within-province estimates.\textsuperscript{26} Similar approaches could
be used for researchers interested in using CNODES to estimate treatment
effects in European or US populations, provided a target population was
provided in the scientific and analytic protocols.

\emph{Are choices ever made to maximize target validity, rather than
internal validity or precision?} Internal validity is the core focus
when preparing the scientific and analytic protocols. When choosing the
provinces and data that will contribute to a given study, however,
attention is paid to populations that are may differ greatly from the
Canadian target population.

PCORnet Q\&A:

\emph{How does external validity come into play when planning and
conducting projects in the database?} PCORnet research teams take a
variety of different approaches depending on the clinical question at
hand. They often specifically discuss the degree to which in-network
study populations are representative of the overall population of
interest, which can vary substantially based on the goals of the
research. This representativeness can include demographic factors,
comorbidities, health care delivery-level factors, and broader societal
factors. If new populations are being recruited into pragmatic trials or
observational cohorts within PCORnet, the study coordinators may either
target the overall PCORnet population or more specific recruiting
targets (such as specific underserved populations). Finally, just like
Sentinel and CNODES, differences between study sites and recruitment
groups are frequently explored when different associations or effects
are observed within nodes of the network.

\emph{What target populations, if any, underlie most analyses?} Because
of the broad mission of PCORnet, specific target populations vary. That
said, most target populations are some subset of patients residing in
the United States with access to healthcare.

\emph{Are there ways to generalize the findings of the nodes to the
network?} There are some existing ways to generalize site-specific
estimates, like meta-analyzing the estimates after checking for
heterogeneity. The focus is generally on estimating a network-wide
treatment effect. How these methods are applied varies depending on the
research team conducting the study and their importance for the clinical
question at hand.

\emph{How easily can node-specific estimates be transported between
nodes or to external populations?} Currently, there are no out-of-the
box solutions for transporting estimates in this fashion. That said,
research groups can generate (if analyzing individual-level data) or
request (if the PCORnet Coordinating Center is generating aggregate
tables) cross-tabulation tables to stratify queries based on potential
effect measure modifiers.

\emph{Are choices ever made to maximize target validity, rather than
internal validity or precision?} Target validity is frequently a major
consideration when PCORnet's partner networks conduct large distributed
pragmatic trials. The diversity of the data and the direct relationships
with clinicians enable researchers to assess this target validity and
perform in-depth evaluation of study results within participating sites.

\textbf{Discussion}

External validity, target populations, generalizability,
transportability, and target validity were not explicit priorities when
designing and constructing these distributed networks. As the networks
have developed, however, differences in results between nodes of the
distributed networks have forced them to confront external validity and
related concepts, and each has chosen to adapt to this challenge in
their own way.

Sentinel uses US-based data networks that cover a large portion of the
population to respond to FDA queries about target populations treated
with a given medication in the US. As a result, it has primarily
interfaced with a lack of generalizability and transportability as
explanations for differences in effect estimates between study sites.
Its primary method for increasing representability has been adding new
data (e.g., Medicaid claims). While CNODES is similarly designed to
answer government queries, the fact that it targets a Canadian
population (despite including non-Canadian data partners) has resulted
in more consideration of how health systems can shape the
generalizability and transportability of study results earlier during
the study design process. PCORnet, in turn, was built for both the
enrollment of pragmatic trials and observational research on cohorts in
a wide array of disease states. PCORnet research questions and target
populations vary more widely than with the other two targeted networks,
making representativeness a major consideration for projects conducted
within the network. PCORnet is also uniquely situated to explore
external validity in these pragmatic trials, since confounding is very
unlikely to be the origin of any differences in estimated treatment
effects across nodes of the network. Of course, these are not the only
noteworthy distributed networks. Exploring how AsPEN, ConcePTION,
DARWIN-EU, and any other networks that may start performing research
differ in their approaches to external validity can only further
understanding of the utility of such analyses.

Despite these differences, it is reassuring to see that all three
networks consider external validity during the study design and
enrollment and interpretation of the final results. Still, none of the
three networks have established analytic tools for standardizing,
generalizing, or transporting partner- or site-specific estimates to
specific target populations (even if some ad hoc programming could
technically be used within some of the systems). This is not surprising
because these methods are still in the early stages of development and
may not yet be sufficiently formalized to be trusted by regulatory
bodies. Researchers need to adapt analytic methods for generalizing and
transporting study results to these distributed settings and establish
how they can improve the overall interpretability of findings. Such
methods are also likely to play a key role in cross-network comparisons
that contrast results from Sentinel, CNODES, PCORnet, or other
distributed networks with one another.

\emph{Conclusion}

External validity and related concepts present unique challenges to
distributed data networks due to the diverse array of populations
included in analyses. Fortunately, Sentinel, CNODES, and PCORnet
recognize these challenges and have developed their own solutions and
analytic strategies tailored to the needs and users of the network.
Still, more research needs to be done on analytic methods for addressing
external validity, target populations, generalizability, and
transportability within the networks.

\textbf{Acknowledgements:} None.\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image1/image1}

\end{center}
\end{figure}

\textbf{Figure 1:} Graphical representation showing differences between
generalizability (Panel 1A) and transportability (Panel 1B). The solid
black study population is either contained within the target population
(in generalizability) or drawn from a separate target population (in
transportability).\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image3/image3}

\end{center}
\end{figure}

\textbf{Figure 2:} A graphical representation of distributed data
networks. A central coordinating center sends analytic protocols or
programs to workers at each of several databases; these protocols yield
estimates or aggregated data, which is returned to the coordinating
center to be consolidated and presented to those who requested the
analysis.

Table 1: A summary of various aspects of the Sentinel, CNODES, and
PCORnet distributed networks.\selectlanguage{english}
\begin{longtable}[]{@{}llll@{}}
\toprule
Network & Sentinel & CNODES & PCORnet\tabularnewline
Primary goal & Act as a resource for the US government and other
interested parties to evaluate safety and expand the role of real-world
data in regulatory decisions & Use collaborative, population-based
approaches to provide rapid answers to questions about drug safety and
effectiveness & Improve US capacity to conduct health research,
particularly comparative effectiveness research (CER) by creating a
large, highly representative network for conducting clinical outcomes
research\tabularnewline
Main users of the network & The US Food and Drug Administration & The
Canadian Drug Safety and Effectiveness Network & PCORnet is a national
resource allowing a wide array of investigators and research funders to
request queries\tabularnewline
Total individuals covered and data partners & More than 228 million
individuals at 16 data partners including fee-for-service claims
databases, hospital networks, and large health plans in the US & More
than 100 million individuals from 10 data partners: 8 Canadian
provinces, 1 US fee-for-service database, and the UK CPRD system &
Integrated record or claims data for more than 70 million individuals
across 8 large Clinical Research Networks and 2 Health Plan Research
Networks in the US\tabularnewline
Data format & Data is maintained in the Sentinel Common Data Model in
SAS & Data format varies, with many partners adopting a common data
model & All PCORnet Network Partners operate under a shared PCORnet
Common Data Model\tabularnewline
Analysis & SAS code is written at the Sentinel Operations Center and
then distributed as a ``package'' to each data partner. After the code
is run, data sets with outcome and treatment are returned to SOC for
final analysis & After finalizing queries, CNODES creates a central
project team. This team creates a scientific protocol that provides
detailed analytic instructions and receives ethics board approval. This
scientific protocol is then used to write site-specific analytic
protocols and code & Study teams and analysis plans are assembled to
meet the needs of the individual study. Analysis of the distributed
query data may be performed centrally, by participating network
partners, or by individual participating sites\tabularnewline
Consolidation and reporting & Results are presented as network-wide
estimates, with the option to present partner-specific estimates;
analytic code and results are posted online & Partner-specific estimates
are returned to the central team. These results can be presented
separately or combined with meta-analytic techniques & Results of the
distributed queries are returned to the central team for a given study.
Results may be reported separately or aggregated together centrally,
depending on the protocol\tabularnewline
\bottomrule
\end{longtable}

1. Wartenberg D and Thompson WD. Privacy versus public health: the
impact of current confidentiality rules. \emph{American Journal of
Public Health} . 2010; 100: 407-12.

2. Platt RW, Henry D and Suissa S. The Canadian Network for
Observational Drug Effect Studies (CNODES): Reflections on the first
eight years, and a look to the future. 2019.

3. Platt R, Brown JS, Robb M, et al. The FDA sentinel initiative---an
evolving national resource. \emph{The New England journal of medicine} .
2018; 379: 2091-3.

4. Stang PE, Ryan PB, Racoosin JA, et al. Advancing the science for
active surveillance: rationale and design for the Observational Medical
Outcomes Partnership. \emph{Annals of internal medicine} . 2010; 153:
600-6.

5. Budrionis A and Bellika JG. The learning healthcare system: where are
we now? A systematic review. \emph{Journal of biomedical informatics} .
2016; 64: 87-92.

6. Maro JC, Platt R, Holmes JH, et al. Design of a national distributed
health data network. \emph{Annals of internal medicine} . 2009; 151:
341-4.

7. Suissa S, Henry D, Caetano P, et al. CNODES: the Canadian network for
observational drug effect studies. \emph{Open Medicine} . 2012; 6: e134.

8. Suissa S, Henry D, Caetano P, et al. CNODES: the Canadian Network for
Observational Drug Effect Studies. \emph{Open medicine : a
peer-reviewed, independent, open-access journal} . 2012; 6: e134-40.

9. Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV and Brown JS.
Launching PCORnet, a national patient-centered clinical research
network. \emph{Journal of the American Medical Informatics Association}
. 2014; 21: 578-82.

10. Agency EM. Data Analysis and Real World Interrogation Network
(DARWIN EU). 2022.

11. Hripcsak G, Duke JD, Shah NH, et al. Observational Health Data
Sciences and Informatics (OHDSI): opportunities for observational
researchers. \emph{Studies in health technology and informatics} . 2015;
216: 574.

12. Pratt N, Andersen M, Bergman U, et al. Multi-country rapid adverse
drug event assessment: the Asian Pharmacoepidemiology Network (AsPEN)
antipsychotic and acute hyperglycaemia study. \emph{Pharmacoepidemiology
and drug safety} . 2013; 22: 915-24.

13. Baggs J, Gee J, Lewis E, et al. The Vaccine Safety Datalink: a model
for monitoring immunization safety. \emph{Pediatrics} . 2011; 127:
S45-S53.

14. Thurin NH, Pajouheshnia R, Roberto G, et al. From Inception to
ConcePTION: Genesis of a Network to Support Better Monitoring and
Communication of Medication Safety During Pregnancy and
Breastfeeding.\emph{Clinical Pharmacology \& Therapeutics} . 2022; 111:
321-31.

15. Toh S, Gagne JJ, Rassen JA, Fireman BH, Kulldorff M and Brown JS.
Confounding adjustment in comparative effectiveness research conducted
within distributed research networks. \emph{Medical care} . 2013; 51:
S4-S10.

16. Toh S, Reichman ME, Houstoun M, et al. Multivariable confounding
adjustment in distributed data networks without sharing of patient-level
data. \emph{Pharmacoepidemiology and drug safety} . 2013; 22: 1171-7.

17. Secrest MH, Platt RW, Dormuth CR, et al. Extreme restriction design
as a method for reducing confounding by indication in
pharmacoepidemiologic research. \emph{Pharmacoepidemiology and drug
safety} . 2020; 29: 26-34.

18. Secrest MH, Platt RW, Reynier P, Dormuth CR, Benedetti A and Filion
KB. Multiple imputation for systematically missing confounders within a
distributed data drug safety network: A simulation study and real-world
example. \emph{Pharmacoepidemiology and drug safety} . 2020; 29 Suppl 1:
35-44.

19. Rothman KJ, Greenland S and Lash TL. \emph{Modern epidemiology} .
Wolters Kluwer Health/Lippincott Williams \& Wilkins Philadelphia, 2008.

20. Westreich D, Edwards JK, Lesko CR, Cole SR and Stuart EA. Target
Validity and the Hierarchy of Study Designs. \emph{Am J Epidemiol} .
2019; 188: 438-43.

21. Lesko CR, Ackerman B, Webster-Clark M and Edwards JK. Target
validity: bringing treatment of external validity in line with internal
validity. \emph{Current epidemiology reports} . 2020; 7: 117-24.

22. Dahabreh IJ, Robertson SE, Steingrimsson JA, Stuart EA and Hernan
MA. Extending inferences from a randomized trial to a new target
population. \emph{Statistics in medicine} . 2020; 39: 1999-2014.

23. Pearl J and Bareinboim E. External validity: From do-calculus to
transportability across populations. \emph{Statistical Science} . 2014;
29: 579-95.

24. Webster-Clark M and Breskin A. Directed acyclic graphs, effect
measure modification, and generalizability. \emph{American Journal of
Epidemiology} . 2021; 190: 322-7.

25. Westreich D, Edwards JK, Lesko CR, Stuart E and Cole SR.
Transportability of Trial Results Using Inverse Odds of Sampling
Weights. \emph{Am J Epidemiol} . 2017; 186: 1010-4.

26. Dahabreh IJ, Haneuse SJA, Robins JM, et al. Study designs for
extending causal inferences from a randomized trial to a target
population. \emph{American journal of epidemiology} . 2021; 190:
1632-42.

27. MacPherson H. Pragmatic clinical trials. \emph{Complementary
therapies in medicine} . 2004; 12: 136-40.

28. Schneeweiss S, Brown JS, Bate A, Trifir\selectlanguage{ngerman}ò G and Bartels DB. Choosing
among common data models for real\selectlanguage{english}-world data analyses fit for making
decisions about the effectiveness of medical products. \emph{Clinical
Pharmacology \& Therapeutics} . 2020; 107: 827-33.

29. Weissfeld M, Gagne JJ and Izem R. SENTINEL METHODS. 2016.

30. Marin-Martinez F and Sanchez-Meca J. Weighting by inverse variance
or by sample size in random-effects meta-analysis. \emph{Educational and
Psychological Measurement} . 2010; 70: 56-73.

\selectlanguage{english}
\FloatBarrier
\end{document}

