Evaluating Large Language Model Performance on User and Language Defined Elements in Code

Evaluating Large Language Model Performance on User and Language Defined Elements in Code

Mekkes, Erik (TU Delft Electrical Engineering, Mathematics and Computer Science)

van Deursen, A. (mentor)
Izadi, M. (mentor)
Katzy, J.B. (mentor)
Nadeem, A. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2023-06-28

Large Language Models of code have seen significant jumps in performance recently. However, these jumps tend to accompany a notable and perhaps concerning increase in scale and costs. We contribute an evaluation of prediction performance with respect to model size by assessing the layer-wise progression for language and user-defined elements in code, using a new technique of Tuned Lenses. We show that language-defined elements can be predicted more accurately in earlier layers of the PolyCoder model than user-defined elements and contribute an evaluation of the attention mechanism, which shows patterns that explain such aspects of performance and indicate areas of missed potential. These findings encourage research into the internal prediction performance for other characteristic aspects of code and could lead to the introduction of new methods that make use of these characteristics to improve performance without relying on scaling.

Large Language Models
Code Completion
Performance Evaluation
GPT
Transformers
Attention Mechanism
Tuned Lens
Token Classification

http://resolver.tudelft.nl/uuid:5d3350fb-ad8e-4975-9ce2-c541a3ec64a5

Student theses

bachelor thesis

© 2023 Erik Mekkes