Can ChatGPT be used to predict citation counts, readership, and social media interaction? An exploration among 2222 scientific abstracts

Can ChatGPT be used to predict citation counts, readership, and social media interaction? An exploration among 2222 scientific abstracts

de Winter, J.C.F. (TU Delft Human-Robot Interaction)

2024

This study explores the potential of ChatGPT, a large language model, in scientometrics by assessing its ability to predict citation counts, Mendeley readers, and social media engagement. In this study, 2222 abstracts from PLOS ONE articles published during the initial months of 2022 were analyzed using ChatGPT-4, which used a set of 60 criteria to assess each abstract. Using a principal component analysis, three components were identified: Quality and Reliability, Accessibility and Understandability, and Novelty and Engagement. The Accessibility and Understandability of the abstracts correlated with higher Mendeley readership, while Novelty and Engagement and Accessibility and Understandability were linked to citation counts (Dimensions, Scopus, Google Scholar) and social media attention. Quality and Reliability showed minimal correlation with citation and altmetrics outcomes. Finally, it was found that the predictive correlations of ChatGPT-based assessments surpassed traditional readability metrics. The findings highlight the potential of large language models in scientometrics and possibly pave the way for AI-assisted peer review.

Citation prediction
Scientometrics
Altmetrics
ChatGPT
GPT-4
Scientific abstracts
Artificial intelligence

http://resolver.tudelft.nl/uuid:e9f814fb-1317-4820-b823-5d0a07ae3e4b

https://doi.org/10.1007/s11192-024-04939-y

0138-9130

Scientometrics: an international journal for all quantitative aspects of the science of science, communication in science and science policy

Institutional Repository

journal article

© 2024 J.C.F. de Winter