Reinforcement learning (RL) is a powerful tool where the agents – or “robots” can learn from the environment based on their actions. Reinforcement learning approaches were found successful in combining predicting stock returns and portfolio allocation. Diversification is a critical element for achieving high portfolio returns with a lower level of risk. This work explores the application of reinforcement learning and multi-agent reinforcement learning (MARL) in portfolio management, emphasizing the applicability and increased portfolio performance via strategy diversification. A key contribution of this work is the development and evaluation of a MARL environment incorporating novel diversity measures, correlation, and total variation distance. The findings reveal that while fine-tuned single-agent RL models can demonstrate strong performance, roughly tuned MARL models with diverse agents reflecting a "portfolio of portfolios" paradigm show improved action diversification and portfolio performances. The work also highlights the critical role of long-term robustness testing, algorithm- and problem-specific hyperparameter optimization, and the challenges of adapting MARL methods to financial contexts. This work contributes to the RL research in portfolio management by exploring the use of MARL in portfolio management and discussing the limitations and future work directions.