Nanomechanical resonators with low dissipation rates are ideal tools in fundamental science applications. They have been used in the field of cavity optomechanics for example in ground-state cooling, and in sensing applications, such as atomic resolution mass sensors. Their great sensitivity is due to their high Q-factor, which is a metric that shows how fast a system loses its energy. In ultra-high Q nanomechanical resonators, energy loss is limited to intrinsic and radiation losses, the latter is due to energy dissipation from the resonator into the substrate. Experiments have shown that the Q-factor of resonators with thin substrates are limited by radiation loss. However, the precise role of the substrate remains a topic that has not received much attention, but has significant implications for how we design nanomechanical microchips. Here we show that the resonator mode can couple to nearby substrate modes, which reduces the Q-factor. We found that the strength of this mode-coupling depends on the mode-shape of the substrate, with stronger coupling at anti-nodes of the mode-shape and hardly any coupling at the nodes. Furthermore, we show that clamping down the substrate with double-sided tape reduces the Q-factor of the resonators, this is explained by a reduction in Q of substrate modes due to the tape. Lastly, we found that in thin substrates, which have a higher density of modes, the Q-factor can be limited due to mode-coupling with the substrate. Our results demonstrate that the substrate choice, as it can strongly affect the Q-factor of resonators, should become an integral part of the resonator design phase. These results can likely be used by all types of nanomechanical resonators limited by radiation loss. We can use this knowledge to design chips with resonators that have an even higher Q-factor.