Evaluating Large Language Model Performance on User and Language Defined Elements in Code

Large Language Models of code have seen significant jumps in performance recently. However, these jumps tend to accompany a notable and perhaps concerning increase in scale and costs. We contribute an evaluation of prediction performance with respect to model size by assessing the layer-wise progression for language and user-defined elements in...

In-IDE code generation models - a literature review

Code generation is becoming one of the most important tools in an Integrated Development Environment (IDE) for programmers, be they novices or experts. It allows them to produce code faster, avoid typos and other mistakes, be more efficient and easily turn ideas into code. As such, there are many in-IDE code generation models to choose from,...

A Cross-Lingual Evaluation of CodeGen's Performance in Code Completion

We present an investigation into the relationship between the average depth of the first correct prediction and the performance of CodeGen. This was done on a dataset comprised of code files comprised of C++, Go, Java, Julia, Kotlin, and Python. The analysis involved investigating the model's predictions at different layers using a Tuned Lens,...

Cross-lingual Performance of CodeGPT on the Code Completion Task

The development of contemporary source code auto-completion tools have significantly boosted productivity and efficiency of developers. In 2021, the GPT-2-based Transformer CodeGPT was developed to support code completion and text-to-code generation. Similarly to most code models however, CodeGPT was trained on a limited set of widely-used...

A Study on the Impact of Common Code Structures on CodeParrot’s Autocompletion Performance

In recent years, deep learning techniques, particularly transformer models, have demonstrated remarkable advancements in the accuracy and efficiency of language models. These models provide the foundation for many natural language processing tasks, including code completion. The effectiveness of code completion models has been the subject of a...

Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study

Transformer-based pre-trained models have recently achieved great results in solving many software engineering tasks including automatic code completion which is a staple in a developer’s toolkit. While many have striven to improve the code-understanding abilities of such models, the opposite – making the code easier to understand – has not been...

Utilizing Lingual Structures to Enhance Transformer Performance in Source Code Completions

We explored the effect of augmenting a standard language model’s architecture (BERT) with a structural component based on the Abstract Syntax Trees (ASTs) of the source code. We created a universal abstract syntax tree structure that can be applied to multiple languages to enable the model to work in a multilingual setting. We adapted the...

User Evaluation of InCoder Based on Statement Completion

A lot of models have been proposed to automatically complete code with promising evaluation results when tested in isolation on testing sets. This research aims to evaluate the performance of these models when used by developers when programming. Are these models still useful for actual programming and do developers even want this functionality?...

A User Evaluation of UniXcoder Using Statement Completion in a Real-World Setting

State-of-the-art machine learning-based models provide automatic intelligent code completion based on large pre-trained language models. The theoretical accuracy of these models reaches 70%. However, the research on the practicality of these models is limited. Our paper will discuss the usefulness of UniXcoder, a machine learning-based cross...

Improvement of Source Code Conversion for Code Completion

Code Completion is advancing constantly, with new research coming out all the time. One such advancement is CodeFill, which converts source files into token sequences for type prediction. To train the CodeFill model, a lot of source files are needed which take a long time to convert before training can begin. Converting the file the end-user is...

An Empirical Analysis of InCoder on the Statement Prediction Task

Automatic code completions are a widely used feature when programming code efficiently. These completions can be made by various code language models, and these can be differentiated in three categories: single token completion, statement (line) completion and block completions. These completions, and in particular statement predictions are...

An Empirical Analysis on the Performance of UniXcoder

Numerous papers have empirically studied the performance of deep learning based code completion models. However, none of these papers considered nor investigated whether good performance on statically typed languages translates to good performance on dynamically typed languages. A lack of available type information can make code completion more...

CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences

Code completion is an essential feature of IDEs, yet current auto-completers are restricted to either grammar-based or NLP-based single token completions. Both approaches have significant draw-backs: grammar-based autocompletion is restricted in dynamically-typed language environments, whereas NLP-based autocompleters struggle to understand...

Towards language-parametric semantic editor services based on declarative type system specifications

Editor services assist programmers to more effectively write and comprehend code. Implementing editor services correctly is not trivial. This paper focuses on the specification of semantic editor services, those that use the semantic model of a program. The specification of refactorings is a common subject of study, but many other semantic...

Principled Syntactic Code Completion using Placeholders

Principled syntactic code completion enables developers to change source code by inserting code templates, thus increasing developer efficiency and supporting language exploration. However, existing code completion systems are ad-hoc and neither complete nor sound. They are not complete and only provide few code templates for selected...