Print Email Facebook Twitter A Cross-Lingual Evaluation of CodeGen's Performance in Code Completion Title A Cross-Lingual Evaluation of CodeGen's Performance in Code Completion Author Keeler, Miranda (TU Delft Electrical Engineering, Mathematics and Computer Science) Contributor van Deursen, A. (mentor) Nadeem, A. (graduation committee) Izadi, M. (mentor) Katzy, J.B. (mentor) Degree granting institution Delft University of Technology Programme Computer Science and Engineering Project CSE3000 Research Project Date 2023-06-28 Abstract We present an investigation into the relationship between the average depth of the first correct prediction and the performance of CodeGen. This was done on a dataset comprised of code files comprised of C++, Go, Java, Julia, Kotlin, and Python. The analysis involved investigating the model's predictions at different layers using a Tuned Lens, which enables examining the intermediate representations. Additionally, attention heads were examined to gain insights into the model's behavior. We found that there is a subset of four layers in which tokens are predicted correctly for the first time. These peaks are evident in CodeGen's performance and come after a small dip, a dip that is present in the last layer. The results shed light on the varying performance of different layers and provide valuable insights into the strengths and weaknesses of CodeGen. These findings contribute to our greater understanding of language model performance in code completion tasks and provide implications for future improvements in this domain. Subject Large Language Models (LLMs)TransformersAttentionCode Completion To reference this document use: http://resolver.tudelft.nl/uuid:3c39aae7-d982-4fd9-9c3a-350912d79acc Part of collection Student theses Document type bachelor thesis Rights © 2023 Miranda Keeler Files PDF CodeShop_Miranda.pdf 1.38 MB Close viewer /islandora/object/uuid:3c39aae7-d982-4fd9-9c3a-350912d79acc/datastream/OBJ/view