Object selection and manipulation are the foundation of VR interactions. With the rapid development of VR technology and the field of virtual object selection and manipulation, the literature demands a structured understanding of the core research challenges and a critical reflection of the current practices. To provide such understanding and reflections, we systematically reviewed 106 papers. We identified classic and emerging topics, categorized existing solutions, and evaluated how success was measured in these publications. Based on our analysis, we discuss future research directions and propose a framework for developing and determining appropriate solutions for different application scenarios.