EMA and FDA databases of drug approvals based on non-RCT
comparisons
To provide empirical support for our theoretical framework, for this
analysis we merged the EMA and FDA database of approvals based on
non-RCT comparisons15,16, as described above. We
merged both databases because of the high concordance rate (91-98%) in
approval 26 between the two agencies. We excluded
duplicate entries between the two databases, and reviewed all regulatory
approval documents that addressed whether testing in subsequent RCTs is
required. We included a total of 134 approvals of drugs and devices that
were based on non-RCTs. For 35 of these treatments, the regulators
required further evidence from RCTs, whereas for 99 treatments they did
not. Although this is the best, contemporary dataset available15,16, it is important to note that
the agencies often failed to provide explicit comparisons of these drugs
and devices against comparators, arguing that in many cases
“efficacy has been assessed on the basis of [outcomes] in
comparison to what would be expected by expert clinical evaluation and
by comparison with previous experience in this type of patient”.15 Indeed, the evaluation of treatment effects
always depends on the comparison of experimental (direct or
counterfactual) with a control intervention if one is to estimate the
effect size. Therefore, when the agencies did not specifically provide
comparison data, we imputed the control events either based on our
interpretation of the agencies’ judgments documented in the approval
reports or the best available data available in the literature.15,16 However, in our attempts to translate the FDA
and EMA judgments into the effect sizes, we frequently imputed very low
(often equal to zero) event rates such as response rate or survival in
the control arm. As a result, we observed some empirically improbable
high effect sizes. Nevertheless, our estimates seem to reflect what the
agencies often believed – “without new treatments, most patients
would surely die ” 15 – implying that these
effects are indeed considered self-evidently large, dramatic effects and
hence confirming the role of heuristics in the decision-making process
of treatment approvals.