The benchmarks for the accuracy of the best performing object detectors to date are usually based on homogeneous datasets, including objects such as vehicles, people, animals and foods. This excludes a whole set of scenarios containing small, cluttered and rotated objects. This paper selects a state-of-the-art object detection model, Faster R-CNN, and investigates its performance on several custom datasets of scattered LEGO pieces. We discover that the model reaches a high F1 score on data with images containing up to 13 bricks and that data manipulation, such as cropping, can further improve this performance. Furthermore, we evaluate how this model can be optimized to perform better on a more complex dataset, showing that tweaking the Faster R-CNN RPN layer results in a higher F1 score for images containing up to 50 bricks. In conclusion, tweaking the RPN layer allows the Faster R-CNN model to reach high performance on datasets containing cluttered images of small LEGO bricks. All data is on the TU Delft server and all code is available at https://gitlab.com/legoproject-group/faster rcnn lego.git.