Evaluating Large Language Model Performance on User and Language Defined Elements in Code

Large Language Models of code have seen significant jumps in performance recently. However, these jumps tend to accompany a notable and perhaps concerning increase in scale and costs. We contribute an evaluation of prediction performance with respect to model size by assessing the layer-wise progression for language and user-defined elements in...

A Cross-Lingual Evaluation of CodeGen's Performance in Code Completion

We present an investigation into the relationship between the average depth of the first correct prediction and the performance of CodeGen. This was done on a dataset comprised of code files comprised of C++, Go, Java, Julia, Kotlin, and Python. The analysis involved investigating the model's predictions at different layers using a Tuned Lens,...

A User Evaluation of UniXcoder Using Statement Completion in a Real-World Setting

State-of-the-art machine learning-based models provide automatic intelligent code completion based on large pre-trained language models. The theoretical accuracy of these models reaches 70%. However, the research on the practicality of these models is limited. Our paper will discuss the usefulness of UniXcoder, a machine learning-based cross...