Monitoring of respiratory rate (RR) is very important for patient assessment. In fact, it is considered one of the relevant vital parameters in critical care medicine. Nowadays, standard monitoring relies on obtrusive and invasive techniques, which require adhesive electrodes or sensors to be attached to the patient's body. Unfortunately, these procedures cause stress, pain, and frequently damage the vulnerable skin of preterm infants. This paper presents a 'black-box' algorithm for remote monitoring of RR in thermal videos. 'Black-box' in this context means that the algorithm does not rely on tracking of specific anatomic landmarks. Instead, it automatically distinguishes regions of interest in the video containing the respiratory signal from those containing only noise. To examine its performance and robustness during physiological (phase A) and pathological scenarios (phase B), a study on 12 healthy volunteers was carried out. After a successful validation on adults, a clinical study on eight newborn infants was conducted. A good agreement between estimated RR and ground truth was achieved. In the study involving adult volunteers, a mean root-mean-square error (RMSE) of (0.31 ± 0.09) breaths/min and (3.27 ± 0.72) breaths/min was obtained for phase A and phase B, respectively. In the study involving infants, the mean RMSE hovered around (4.15 ± 1.44) breaths/min. In brief, this paper demonstrates that infrared thermography might become a clinically relevant alternative for the currently available RR monitoring modalities in neonatal care.