Undergraduate mathematics education is essential for building a foundation for success in various scientific disciplines. Curriculum reforms in mathematics education have emphasized the need for cultivating learning skills that depend on effective student engagement (SE). Consequently, there has been growing research on the mechanisms that facilitate SE and promote its development. To gain insights into the state of SE research in undergraduate mathematics, the current systematic review addresses the varied research by examining four key aspects: a) theoretical grounding and research aims, b) definitions, c) measurement, and d) learning outcomes. Following the PRISMA guidelines, a literature search identified 1,584 records, with 48 papers meeting the inclusion criteria. The findings reveal three primary research aims and three approaches to grounding SE research, with most studies using an instruction-focused framework to evaluate instructional methods. Nearly half of the papers provided a definition of SE, with analysis showing varied uses of elements like psychological investment and multidimensionality. Studies that used multiple elements offered more concise definitions. Measurement of SE predominantly focused on online learning through log files or course participation via self-reports, with behavioral engagement being the most commonly examined dimension. Less than half of the studies explored the relationship between SE and learning outcomes, using both variable-oriented and person-oriented approaches to examine this connection. Based on the current findings, the review offers recommendations for aligning conceptualizations, definitions, measurement, and context in future research to foster a shared understanding and guide interventions.