Market making, the act of providing liquidity to the market by simultaneously buying and selling, is a difficult problem to solve. The use of reinforcement learning to solve for market making is increasing, as academics and practitioners alike look for novel ways to approximate for optimal policies in increasingly complex markets. This thesis examines the use of reinforcement learning to solve the market making problem in limit order books. To this end, we provide the theoretical background on market making, modelling limit order books, and reinforcement learning. Furthermore, we implement and compare ’classical’ algorithms, such as Q-learning and value and policy iteration, and compare their policies with newer techniques involving deep learning, namely deep Q-networks and double deep Q-networks. To train and compare these models, we use two models to simulate the dynamics of an order book. Experiment results and the ultimately learned policies are presented and discussed. We propose several ideas that could be worked on in the future.