From the integer quantum Hall effect to swimming at a low Reynolds number, geometric phases arise in the description of many different physical systems. In many of these systems, the temporal evolution prescribed by the geometric phase can be directly measured by an external observer. By definition, geometric phases rely on the history of the system’s internal dynamics, and so their measurement is directly related to the temporal correlations in the system. They, thus, provide a sensitive tool for studying chaotic Hamiltonian systems. In this work, we present a toy model consisting of an autonomous, low-dimensional, chaotic Hamiltonian system designed to have a simple planar internal state space and a single geometric phase. The diffusive phase dynamics in the highly chaotic regime is, thus, governed by the loop statistics of planar random walks. We show that the naïve loop statistics result in ballistic behavior of the phase and recover the diffusive behavior by considering a bounded shape space or a quadratic confining potential.