This work addresses the problem of exploration and coverage using visual inputs. Exploration and coverage is a fundamental problem in mobile robotics, the goal of which is to explore an unknown environment in order to gain vital information. Some of the diverse scenarios and applications in which exploratory robots can have a significant impact include search and rescue missions, environmental monitoring, and space exploration. Specifically, we focus on the aspect of finding areas of interest, also referred to as targets, in the environment. In this thesis, we propose a deep reinforcement learning based approach relying solely on visual observations. In particular, our method builds upon the off-policy, actor-critic, Importance Weighted Actor-Learner Architectures (IMPALA) framework by including a set of novel auxiliary tasks, i.e. Pose Estimation and Local Map Prediction. These auxiliary tasks are inspired by Simultaneous Localization and Mapping (SLAM) approaches to exploratory robotics problems. The intuition is to assist internal representation learning and build locale specific knowledge by teaching the agent to predict its position and orientation, as well as transfer the visual information to information about the its local proximity. Experiments conducted in the DeepMind Lab simulation environment show improved performance over the base IMPALA agent and demonstrate the effectiveness of these auxiliary tasks. Furthermore, we investigate the performance of the agent, trained through various stages of curriculum, compared to a human controlled agent. The trained agent is shown to outperform the human in the majority of tested scenarios.