Overstated findings and overreaching conclusions
Very few studies provide definitive findings. Hence the cliché “more
studies are needed”. I have found that authors tend to position their
findings as definitively solving an issue or closing the book on a line
of inquiry. I suspect that is a rhetorical device aimed at conveying the
importance of publishing the manuscript. Unfortunately, from an editor
and reviewer’s point of view, that is a bad strategy. Overstated
findings or drawing conclusions that are beyond the scope of the data or
analysis give the impression of arrogance and/or ignorance. Very few
studies by themselves provide information that will (or even should)
revolutionize clinical practice or the organization and/or provision of
healthcare services (or even have an immediate impact on any aspect of
healthcare). That is the rule in science, not the exception. A recent
report by the International Association of Scientific, Technical and
Medical Publishers notes approximately 2.5 million articles published in
academic and scientific journals in 2018 (as per Scopus) [2]. A
substantial proportion of those articles are in areas related to
medicine and healthcare. The United States National Library of Medicine
notes nine hundred thousand citations added to MEDLINE in 20185. Despite these impressive numbers,
clinical practice and the organization/provision of healthcare services
seems to change little from year-to-year. One might blame poor knowledge
translation, but a more reasonable explanation is that very few of those
studies provide more than preliminary findings or something that can be
generalized across patient populations. Given that it is unlikely that a
study will revolutionize practice, it seems odd to make any claim about
how practice ought to change when stating the manuscript’s conclusion.
Such statements should be reserved for studies with highly robust
findings (however that is defined by the community).
It is convention that authors explicitly acknowledge the limitations of
their studies. Despite that acknowledgement in the manuscript, there
seems to be a fondness among some authors for drawing conclusions from
the data as if those limitations were not present! Reviewers and editors
pick up on that and often ask the authors to temper their conclusions. I
suppose such enthusiasm for the findings is an attempt to elevate the
perception of the contribution of the manuscript to advancing knowledge.
I find that it can have the opposite effect – academics are skeptical
people and are uncomfortable with hyperbole, or at least that has been
my experience.
Perhaps the reason for overstating findings and drawing conclusions that
go beyond what the data can support is the author does not have a clear
idea as to what counts as a contribution worth publishing. What matters
is that the manuscript presents information that has done something to
advance the community’s understanding of a subject or phenomenon. The
strength of the contribution is contingent on the importance of what is
being examined (i.e. the question asked, or issue raised) and how good
the study or argument is in supporting that examination (i.e. answering
that question or addressing that issue). Thus, the emphasis in the
manuscript should be on carefully articulating the importance of what is
being examined and showing how the methods used and results obtained
address the concern raised by the author (e.g. the knowledge gap),
rather than the conclusion. A contribution can be more than identifying
a causal mechanism or showing an intervention is effective. Examining a
previously described relationship or finding in a new context can count
as a contribution, although the author should communicate in the
manuscript what it is about that context that raised concern as to why
the previous finding was potentially not applicable (this is a common
criticism I see among reviewers). An interesting interpretation of the
current knowledge base or a critical analysis of an issue, body of
literature, or specific study may also be a contribution, provided it
adds something new to the community’s understanding of the subject of
inquiry.