Recent works have shown that neural networks can be employed to solve partial differential equations, bringing rise to the framework of physics informed neural networks.The aim of this project is to gain a deeper understanding of these novel methods, and to use these insights to further improve them. We show that solving a partial differential equation can be formulated as a multi-objective optimization problem, and use this formulation to propose several modifications to existing methods. These modifications manifest as a scaling parameter, which can improve the accuracy by orders of magnitude for certain problems when it is chosen properly. We also propose heuristic methods to approximate the optimal scaling parameter, which can be used to eliminate the need to optimize this parameter. Our proposed methods are tested on a variety of partial differential equations and compared to existing methods. These partial differential equations include the Laplace equation, which we solve in up to four dimensions, the convection-diffucsion equation and the Helmholtz equation, all of which show that our proposed modifications lead to enhanced accuracy.