Cancer affects a countless number of lives across the world each day. Mathematical oncology develops and studies mathematical models of cancer and its treatment. This thesis focuses on spatiotemporal modeling in mathematical oncology, developing an agent-based model for prostate cancer, with the aim of gaining insights into how the different tumor cells respond to varying testosterone levels and different treatment strategies.

We began by analyzing non-spatial population models, the replicator dynamics and Lotka-Volterra dynamics, proving their equivalence under certain conditions. The study then transitioned to spatial agent-based modeling on a discrete lattice, simulating the interactions between testosterone-dependent and testosterone-independent tumor cells. Through this, we identified a possible phase transition in the testosterone level in the bloodstream, which could influence which tumor cells dominates the grid.

A continuum limit of the discrete model was derived, leading to partial differential equations that describe the tumor's spatial behavior. We applied mathematical tools like non-dimensionalization and linear stability analysis to gain deeper insights into the dynamics of the model. Additionally, we simulated three treatment strategies: (1) testosterone removal from the blood with Lupron, (2) Lupron combined with Abiraterone to stop the testosterone producing cancer cell to grow, and (3) Lupron and Abiraterone alongside high-dose testosterone injections to simulate extinction therapy.

The flexibility of our model allows for its application to other hormonal cancers, and our findings support the promising potential of hormonal manipulation in controlling tumor growth and composition, especially extinction therapy. The mathematical analysis together with simulations provide unique insights into the tumor dynamics. Future research directions include changing assumptions, expanding the model to three dimensions and integrating patient data for more accurate simulations.