Towards reproducible radiomics research: introduction of a database for radiomics studies

Radiomics is an image analysis method that makes it possible to discover more than what can be seen with the naked eye, and it is a rapidly growing field of research [1]. Since its first appearance in the literature [2], thousands of articles have been published [3]. At the turn of a new decade of radiomics research, the clinical translation of radiomics models still lags far behind [3]. One of the main reasons for this chasm between research and clinical implementation is the lack of reproducibility. Shortage of clinical data- or radiomics model-sharing practices in the field further complicates the achievement of this goal [4]. This trend resembles the reproducibility crisis in the field of psychology, where independent researchers attempted to replicate results of previously published prominent articles, but succeeded only 30% of the time [5]. Open science practices [6, 7], i.e., data and code sharing, are important steps to overcome these challenges, as they enable more researchers to independently test proposed methods on an independent dataset.

With the development of checklists such as CheckList for EvaluAtion of Radiomics research (CLEAR) [8], the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) [9], or Must AI Criteria 10 Checklist (MAIC-10) [10], code and data sharing in radiomics and artificial intelligence (AI) research is increasingly encouraged. Despite the fact that these practices are being promoted, they may not be necessarily sufficient for reproducibility, i.e., authors may only share “pseudocode” that is only a representation of the main code, or there may be errors in the code even though it is fully shared [11]. This trend can also be observed in AI research in radiology: According to a recent article, data were released in only 13% of AI papers, code was shared in only 34%, and only 33% of shared code was reproducible [12].

In the absence of a universally accepted definition of reproducibility, in this paper, we accept the earlier description of “research reproducibility” [13, 14]. Namely, research is reproducible if the same results are obtained when the same data, code, and method are used, and research is replicable if the same results are obtained when the same method and code are applied on an independent dataset. Considering these definitions, reproducibility is the minimum criterion for generalizable radiomics research [14].

In this paper, we have explored the practices of sharing model, code, and data in the current radiomics research landscape. Along with presenting our findings in this paper, we also introduce a large radiomics research database to facilitate the retrieval of radiomics models, code, and data if shared. We believe the radiomics research database will help researchers disseminate scientific knowledge and pave the way for clinical implementation, and we hope that this will encourage more researchers to share their model, code, and data. We will continue to expand the radiomics research database and call on interested researchers to help us in this endeavour.

In this study, we found that most radiomics studies did not share code or data; in other words, most radiomics models were not reproducible. To our surprise, the authors tend to share their validated model with an extensive model-building methodology; however, only around 30% of the models were conceptually reproducible as they were validated on an external dataset. In this study, we also introduced a searchable radiomics research database (RadBase). To our knowledge, there has been only one such initiative within the AI research community: the Papers with Code website shares papers that contain links to code to promote code sharing [18]. With the proposed radiomics database, we aim to promote all facets of reproducible radiomics research and achieve broader acceptance within the radiomics community.

Despite the growing number of guidelines encouraging open science practices in radiomics research, our results show that the radiomics research community still lags far behind. In addition, some researchers find a way to circumvent checklists and guidelines by intentionally or unintentionally sharing broken links, empty GitHub repositories, incomplete code, or data that never allow published methods to be truly reproduced [12]. In this study, we also observed practices such as misleading statements, e.g., reporting that the model is in the supplement but it was not [19, 20], and sharing only pseudocode [21] that does not allow to reproduce proposed model. To avoid these issues, not only code or data sharing but also reproducible code sharing is encouraged by major artificial intelligence congresses such as Neural Information Processing Systems (NeuroIPS) with the Machine Learning Reproducibility checklists [22].

At the turn of a decade, we should embark on reproducible radiomics research if we do not want to spend another decade waiting for clinical implementation. We should also ensure that decades later our methodology can withstand the challenges that come with the ever-changing technologies. The results of a recent initiative to determine whether the code that researchers wrote 10 years ago still works [23] can serve as a cautionary tale. While some of the researchers were able to get 10-year-old code to work, even when the data had to be transferred from a floppy disk to the cloud, the majority of them (80%) did not work [24]. Unfortunately, our results over the last 2 years point in the same direction—most of the radiomics models were technically not reproducible. In an ever-growing field of radiomics research, as we adopt open science practices, we should also make sure that our methods persist over time.

Similar to a previous study [12], we observed that authors who were required to make a data- and code-sharing statement were more likely to share their data and code openly. Promoting reproducibility thorough journal standards, checklists, and badges could enable the widespread adoption of open science practices [25]. Moreover, this could be as simple as requesting a statement to do so [6]. Journals could adopt the proposed model of standards for reproducibility to reward these practices accordingly on a sliding scale [6, 25]. Evidently, the model and code have to be submitted with the manuscript to ensure reproducibility. However, an exception may be made if the authors can demonstrate that they wish to protect their intellectual property for a while, but if they fail to do so within a reasonable time frame, the manuscript should be withdrawn or modified. Clearly, patient privacy issues could hinder data sharing in healthcare, but this could be easily overcome by sharing data through databases with strict data use agreements.

Our study had some limitations. First, we only included a selection of articles from leading journals that were recently published. Therefore, our results might not be generalizable to the rest of the radiomics literature. Even if we were able to show that publications in some of the highest-ranked journals from Europe and the USA only rarely followed open-science practices, this result could be different in other journals from different parts of the word (e.g., the journals from the Asia–Pacific region). Secondly, we did not technically control if the shared model is reproducible or not, but we encourage researchers to join our efforts and use our database for the planning and execution of validation studies. Finally, although we did not extract data on methodological details of the entire study pipeline in this study, e.g., postprocessing methods, feature extraction techniques, feature normalization, or harmonization methods, these details could also affect the reproducibility of the study and should be properly reported. Therefore, we encourage researchers to share data not only concerning statistical, conceptual, or technical reproducibility, such as we collected in this study, but also the data on the whole pipeline in our database.

In conclusion, there is room for improvement regarding reproducibility in radiomics research. We urge researchers to adopt open science practices to ensure reproducible radiomics research now and in the future.