Physics-Informed Neural Networks (PINNs) are intended to solve complex problems that obey physical rules or laws but have noisy or little data. These problems are encountered in a wide range of fields including for instance bioengineering, fluid mechanics, meta-material design and high-dimensional partial differential equations (PDEs). Whilst PINNs show promising results, they often fail to converge in the presence of higher frequency components; a problem known as the spectral bias. Multiple studies have explored ways to overcome or minimize spectral bias specifically for PINNs. This paper builds on previous studies by investigating the impact of different gradient descent methods on the spectral bias.