“Expression of concern”: publication bias for positive preclinical cardioprotection studies

The field of cardioprotection is faced with a growing disappointment over the translational gap between the myriad of positive preclinical studies and the mostly neutral larger phase III clinical trials [6, 7, 9‐11]. By their very nature, preclinical studies are reductionist in that they attempt to identify and characterize mechanism(s) underlying cardioprotection. For that purpose, each laboratory uses a cardioprotective intervention that works well in its hands and can be used to search for mechanism(s). Cardioprotective interventions which do not work are then abandoned, preliminary data are not completed and most often not reported. Only recently, with the recognition of the translational gap, have preclinical studies with neutral results on cardioprotective ischemic conditioning interventions come forward and been reported [17, 18, 20]. The above personal experience led us to suspect that these neutral studies may in fact only be the tip of the iceberg. Could a publication bias for positive preclinical studies on cardioprotective interventions have contributed to unjustified expectations on clinical benefit from these interventions? then to the premature conduct of respective large phase III trials? and finally to the resulting disappointment? We realize that there were a number of so-called “proof-of-concept” clinical phase II trials before embarking on larger phase III trials, but—strictly speaking—these were not designed to provide robust proof for efficacy but for safety of the cardioprotective interventions under study [11].

To analyze the potential publication bias for positive preclinical cardioprotection studies, we went back and searched in the publicly accessible Database “Web of Science Core Collection” (Clarivate, Philadelphia, USA) for preclinical studies on cardioprotection (search criteria: “cardioprotection” or “infarct size” or “myocardial infarction”) which were published between January 2013 and December 2023 in Basic Research in Cardiology, Cardiovascular Research, and Circulation Research. In a first refinement step, we excluded all reviews, studies in humans and preclinical studies which did not compare infarct size without and with an intervention. We then used all papers that reported infarct size data without and with an intervention and stratified them as positive vs. neutral/negative according to the reported statistical significance (see Supplementary Table). If multiple data sets with a primary infarct size endpoint were available in a given paper (e.g., for different species, for different sexes), these data sets were separately used in our further analysis (see Supplementary Table: lines marked gray). In a second refinement step, for a more stringent analysis, we used only data sets which had quantified infarct size by either triphenyltetrazolium chloride (TTC) staining (normalized to area at risk, left ventricular or cardiac mass, respectively) or magnetic resonance imaging (MRI) or single photon emission tomography (SPECT) (when calculated in absolute mass or normalized to left ventricular mass). Among all data sets, we identified those with a prospective power analysis (see Supplementary Table: lines marked red) and retrospectively calculated the power of the respective results from the published data sets. In cases where numerical data were not available, we digitized the graphical data and calculated the power from those. Even in prospective power analysis-based studies the available information on the power analysis was sometimes sparse, and it was not always clear to what effect the power really related (supplemental references 102, 131, 196, 266, 267, 323, 330, 331, 337, 352, 365, 368, 370). Other studies reported a prospective power analysis with an endpoint other than infarct size (supplemental references 47, 272, 353, 361). We calculated the power for a significance level of α = 0.05 of the most reasonable significant “primary infarct size endpoint” and used only these data sets for a most stringent analysis, but also calculated the power all other available infarct size data sets for a more liberal analysis. When the n-values for a given data set were not precisely indicated but as a range, we assumed the lowest n-value for a conservative power estimate.

A total of 2155 papers on cardioprotection with key words “cardioprotection” or “infarct size” or “myocardial infarction”, were identified. Detailed data on search strategy and search results are summarized separately for each journal in Table 1. After exclusion of reviews, studies in humans and preclinical studies which did not compare infarct size data without and with an intervention, a total of 371 preclinical papers remained. A subset of 269 reported the methodologically most robust data using TTC, MRI, or SPECT for infarct size quantification. Among these papers with the methodologically most robust data, only 26 had used a prospective power analysis and only 17 specified infarct size as endpoint. Ten papers with a prospective study design were published in Basic Research in Cardiology, 4 in Cardiovascular Research and 12 in Circulation Research. For 40 prospectively designed data sets in small rodents (mice, rats, rabbits) the fraction of positive results was 83% and almost identical to the 89% in data sets without a prospective power calculation. Eleven of the prospectively designed data sets were in pigs, the most relevant species for potential translation to patient benefit [12]; in this small sample, 7 (64%) results were positive, notably less than the 39 out of 49 (80%) positive data sets in pig studies did not use a prospective design.

When calculating retrospectively the power, combined for all three journals, and considering only data sets with the most robust infarct size quantification method, only 75% of the significant positive results had a power of  ≥ 0.9 and an additional 9% had a power of  ≥ 0.8. Thus, 16% of all significant positive results did not even reach the power threshold of 0.8. Only 13% of all analyzed data sets reported neutral results. Figure 1 depicts these results separately for Basic Research in Cardiology, Cardiovascular Research and Circulation Research. With the more liberal analyses including also infarct size data determined by histologic image analysis (Suppl Fig. 1) and additionally including all infarct size comparisons in each paper (Suppl Fig. 2), the same trend was apparent. Often, a cardioprotective effect of an intervention with low power was followed by secondary antagonist effects on this cardioprotective intervention with higher power, further supporting the notion that the cardioprotective effect was not really robust.

Unfortunately, the NHLBI-sponsored CAESAR consortium [15] which had pioneered in Circulation Research 2015 the approach of a prospectively designed, multi-center trial with a core-lab centralized data analysis for experimental cardioprotection studies was not continued after the end of the funding period, but it was still followed by 17 prospective, power analysis-based study protocols in Circulation Research, and this journal in its statistical author submission guidelines continues to specifically ask for power analysis, expected effect size etc.

Why are preclinical studies mostly positive but the major clinical phase III trials neutral/negative? Our analysis clearly revealed that neutral results are underreported and that there is a lack of preclinical cardioprotection studies with a prospective, power analysis-based design, and a publication bias for positive results in those studies without a prospective design.

We realize that a power analysis is an established instrument to avoid a type II error in a specific, individual study, i.e., to prospectively determine a sample size with the aim to not miss an expected or reasonable effect size with a given probability. We here used a retrospective power analysis to quantitatively characterize a research field with a larger number of data sets with the effect of the size that was actually observed as sufficiently powered or not with an adequate sample size, if it were repeated. In our study, the neutral results can serve as a positive control for such use since they all had a power of less than 0.8, whether they were prospectively planned or not. However, the low power is no surprise since non-significant p-values always correspond to low retrospective power [13].

Thus, our use of a retrospective power analysis was not aiming to assess whether or not a given hypothesis in an individual study is indeed correct, but on the chance that positive results in a research field are repeated under the assumption of the sample size which was used and the effect size which was observed—this is in our view one essential feature of robustness [3]. Of course, there are other factors, apart from too small sample size, which undermine the robustness of preclinical cardioprotection studies, most notably lack of a priori definition of exclusion and inclusion criteria, lack of proper randomization and lack of blinding of the investigators [1, 2]. Different from power, these other factors cannot be identified retrospectively, unless they are explicitly specified in the study.

We are aware that the approach to calculating retrospective power can be criticized, since non-significant p-values always correspond to low retrospective power as mentioned above [13]. Nevertheless, we think that the approach is justified for our intention in the present study. We could not determine the positive predictive value because the baseline probability for a positive result was unknown for the analyzed studies.

Clinical trials are designed prospectively and must be pre-registered, e.g., on clinicaltrials.gov or else, and will not be published in a rigorous, high-impact journal unless they are pre-registered—so the final results can always be compared to the original study design, and neutral and negative data cannot be hidden. There are attempts to establish pre-registration also for preclinical studies [19] but pre-registration is unfortunately not really accepted and used by the cardioprotection community so far, and its use is not a prerequisite for publication in a decent journal—a measure that largely promoted the use of pre-registration for clinical trials.

For journal editors, we propose to not only explicitly encourage publication of neutral data but also not to put the burden to explain discrepant results from prior studies on the authors of the neutral study, in particular when the neutral study has a power analysis-based prospective design. Strictly speaking, only the rejection of the Null hypothesis needs a reasonable explanation. Maybe, a journal editor in a case where a neutral study did not confirm a prior positive study should solicit a comment from the authors of the prior positive study. Basic Research in Cardiology has just done that, and the results were indeed enlightening [14, 16]. Also, the request for the identical repetition of a positive prior study is futile, as that would require not only use of animals of the same breed, sex and age [15‐17], the same anesthesia, surgical approach and study protocol including route of administration, timing and dosing of the cardioprotective intervention, but also keeping such minute conditions constant as time of the year [16, 21], time of the day [4, 5], the composition of diet and tap water [15] which all impact on the study results and its robustness but are impossible to replicate. Of course, all these variables are also relevant for clinical cardioprotection trials and may differ between one trial and another one. Therefore, it is an important consideration whether or not a cardioprotection of interest relies on robust preclinical data before embarking on a clinical trial.

In conclusion, better reporting of positive and neutral data will further improve the rigor and robustness of preclinical cardioprotection studies and thus facilitate their translation to patient benefit [1, 2, 8, 15].