A Cross-Lingual Evaluation of CodeGen's Performance in Code Completion

A Cross-Lingual Evaluation of CodeGen's Performance in Code Completion

Keeler, Miranda (TU Delft Electrical Engineering, Mathematics and Computer Science)

van Deursen, A. (mentor)
Nadeem, A. (graduation committee)
Izadi, M. (mentor)
Katzy, J.B. (mentor)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2023-06-28

We present an investigation into the relationship between the average depth of the first correct prediction and the performance of CodeGen. This was done on a dataset comprised of code files comprised of C++, Go, Java, Julia, Kotlin, and Python. The analysis involved investigating the model's predictions at different layers using a Tuned Lens, which enables examining the intermediate representations. Additionally, attention heads were examined to gain insights into the model's behavior. We found that there is a subset of four layers in which tokens are predicted correctly for the first time. These peaks are evident in CodeGen's performance and come after a small dip, a dip that is present in the last layer. The results shed light on the varying performance of different layers and provide valuable insights into the strengths and weaknesses of CodeGen. These findings contribute to our greater understanding of language model performance in code completion tasks and provide implications for future improvements in this domain.

Large Language Models (LLMs)
Transformers
Attention
Code Completion

http://resolver.tudelft.nl/uuid:3c39aae7-d982-4fd9-9c3a-350912d79acc

Student theses

bachelor thesis

© 2023 Miranda Keeler