Print Email Facebook Twitter Empirical assessment of ChatGPT’s answering capabilities in natural science and engineering Title Empirical assessment of ChatGPT’s answering capabilities in natural science and engineering Author Schulze Balhorn, L. (TU Delft ChemE/Process Systems Engineering) Weber, J.M. (TU Delft Pattern Recognition and Bioinformatics) Buijsman, S.N.R. (TU Delft Ethics & Philosophy of Technology) Hildebrandt, Julian R. (Rheinisch-Westfälische Technische Hochschule) Ziefle, Martina (Rheinisch-Westfälische Technische Hochschule) Schweidtmann, A.M. (TU Delft ChemE/Process Systems Engineering) Date 2024 Abstract ChatGPT is a powerful language model from OpenAI that is arguably able to comprehend and generate text. ChatGPT is expected to greatly impact society, research, and education. An essential step to understand ChatGPT’s expected impact is to study its domain-specific answering capabilities. Here, we perform a systematic empirical assessment of its abilities to answer questions across the natural science and engineering domains. We collected 594 questions on natural science and engineering topics from 198 faculty members across five faculties at Delft University of Technology. After collecting the answers from ChatGPT, the participants assessed the quality of the answers using a systematic scheme. Our results show that the answers from ChatGPT are, on average, perceived as “mostly correct”. Two major trends are that the rating of the ChatGPT answers significantly decreases (i) as the educational level of the question increases and (ii) as we evaluate skills beyond scientific knowledge, e.g., critical attitude. Subject OA-Fund TU Delft To reference this document use: http://resolver.tudelft.nl/uuid:56ad355d-48e7-414d-b431-09e62987db79 DOI https://doi.org/10.1038/s41598-024-54936-7 ISSN 2045-2322 Source Scientific Reports, 14 (1) Part of collection Institutional Repository Document type journal article Rights © 2024 L. Schulze Balhorn, J.M. Weber, S.N.R. Buijsman, Julian R. Hildebrandt, Martina Ziefle, A.M. Schweidtmann Files PDF s41598-024-54936-7.pdf 1.41 MB Close viewer /islandora/object/uuid:56ad355d-48e7-414d-b431-09e62987db79/datastream/OBJ/view