Key points
- Researchers from MIT and Dana-Farber Cancer Institute led a team that used large language models to review nearly 4,000 clinical prediction studies.
- Roughly one in eight of the studies said their code was available, the team reports in Nature Medicine.
- Code-sharing rates differed widely by journal, from zero to above 70 percent.
- The results feed into TRIPOD-Code, guidance under development on what researchers should share alongside their papers.
Researchers at MIT and Dana-Farber Cancer Institute led an international team that used large language models (LLMs) to audit nearly 4,000 medical studies. Roughly one in eight said they had shared the code behind their results. The open-access study in Nature Medicine also found that posting code did not reliably come with the explanations and supporting files others would need to reuse it.
The audit targeted clinical prediction studies, which build or test models that forecast how patients will fare. Since such models guide patient care, outside scientists must be able to examine how they were constructed. The team collected studies citing the TRIPOD or TRIPOD+AI reporting checklists with full texts in PubMed Central's open-access collection. An LLM sifted the papers for statements about whether code was available and pulled out links. Software then downloaded hundreds of the linked repositories, which were scored on qualities such as how complete their documentation was, what software they needed and their reuse terms.
Practices differed sharply. From one journal to the next, code-sharing rates ran from zero to above 70 percent. Several low-sharing journals had written code-sharing policies, which the authors say suggests policies by themselves may not be enough. Some researchers offered synthetic datasets so their code could be reused when real patient records had to stay private.
Two researchers each checked 500 papers and a separate set of repositories on their own. Their verdicts lined up with the machine's more closely for concrete features, like whether a repository had a license file, than for judgments about whether documentation was good enough. The results are feeding into TRIPOD-Code, a reporting guideline still in development.
Why it matters
Since clinical prediction models guide patient care, outside scientists must be able to examine how they were made and tested. The review supplies evidence for TRIPOD-Code, guidance meant to give researchers and journals clearer expectations.
What we don't know yet
Because the automated sorting can still make mistakes, the authors stress broad patterns over exact counts. Human and LLM judgments also agreed less on whether documentation was adequate.
Terms in this story
- Clinical prediction model
- A model that forecasts how a patient is likely to fare, for example the chance of getting a disease.
- Code repository
- An online collection of a project's code and related files.
- Synthetic dataset
- Artificial data made to resemble real records so that code can be tested without exposing private information.
Sources and verification
DexScience articles are produced with AI assistance and an automated fact-check against the linked sources. Spotted an error? Tell us and we will correct it openly.


