The spread of infectious diseases such as COVID-19 presents many challenges to healthcare systems and infrastructures across the world, exacerbating inequalities and leaving the world's most vulnerable populations at risk. Epidemiological modelling is vital to guiding evidence-informed or data-driven decision making. In forced displacement contexts, and in particular refugee and internally displaced people (IDP) settlements, it meets several challenges including data availability and quality, the applicability of existing models to those contexts, the accurate modelling of cultural differences or specificities of those operational settings, the communication of results and uncertainties, as well as the alignment of strategic goals between diverse partners in complex situations. In this paper, we systematically review the limited epidemiological modelling work applied to refugee and IDP settlements so far, and discuss challenges and identify lessons learnt from the process. With the likelihood of disease outbreaks expected to increase in the future as more people are displaced due to conflict and climate change, we call for the development of more approaches and models specifically designed to include the unique features and populations of refugee and IDP settlements. To strengthen collaboration between the modelling and the humanitarian public health communities, we propose a roadmap to encourage the development of systems and frameworks to share needs, build tools and coordinate responses in an efficient and scalable manner, both for this pandemic and for future outbreaks.