Large language models (LLMs) are rapidly increasing in parameter count, but this growth is not matched by an availability of high-quality data. This discrepancy raises concerns about the sustain- ability of current approaches to language model improvement, especially as forecasts suggest a potential data shortage by the end of the decade. This study investigates the impact of different tokenization strategies on the performance of small transformer (around 10 million parameters) models, evaluating three prominent subword tokenization methods: Byte-Pair Encoding (BPE), Word- Piece, and SentencePiece. Additionally, we ex- amine the trade-off between vocabulary size and embedding size and measure these factors’ effects on language understanding and model efficiency within the BabyLM pipeline with BLiMP and SuperGLUE scores. Our findings indicate that while the different tokenization strategies have minimal impact on model performance, the trade-off be- tween vocabulary size and embedding size significantly affects both language understanding and efficiency. Increasing the vocabulary size beyond a certain threshold does not seem to enhance language understanding. This research improves our understanding of how tokenization influences the language modeling process, specifically within the context of small language models.

Transformers are a type of neural network archi- tecture used in natural language processing. They excel in tasks such as translation, text generation, and language modeling by capturing long-range de- pendencies. Increasing input sequence length en- hances performance but at a high computational cost. This study investigates the effectiveness of Infini-attention, a proposed solution to mitigate these costs, and explores its integration strategies. We implemented and trained Infini-attention on the GPT-NEO platform and TinyStories dataset, evalu- ating on the BabyLM pipeline. Our findings reveal the optimal strategy for integrating Infini-attention.

The advent of transformer architectures revolutionized natural language processing, particularly with the popularity of decoder-only transformers for text generation tasks like GPT models. However, the autoregressive nature of these models challenges their inference speed, crucial for real- time applications and resource-constrained environments. Memory bandwidth is a significant bottleneck, especially in autoregressive decoding, where constant loading of large key and value tensors dominates. The Multi-Query Attention (MQA) architecture was proposed to reduce memory access by shrinking the key-value cache, enhancing infer- ence speed at the cost of generation quality. Grouped-Query Attention (GQA) was introduced to mitigate this quality decline, serving as an interpolation between Multi- Head Attention (MHA) and MQA. We explore the trade- offs between inference speed and quality in decoder-only models by experimenting with various proportions of query groups relative to attention heads during pre-training. Additionally, we investigate the impact of reducing the size of key and value vectors compared to GQA and explore a hybrid method combining GQA with shortened key-value vectors. This study aims to expand the list of possible trade-offs and help select an optimal architecture based on specific needs.