Learning human motion prediction models online is key for autonomous navigation in unknown dynamic scenarios. Previous works focus solely on improving prediction network architectures, whilst training them offline. This paper introduces a self-supervised continual learning framework that continuously improves data-driven pedestrian trajectory prediction models online across various environments. We propose to use online streams of pedestrian data, normally available from detection and tracking pipelines. Examples are autonomously extracted from this data stream and aggregated in temporally bounded episodes, where the data of each episode is discarded as soon as the model has been adapted to it. Our framework overcomes the problem of catastrophic forgetting across episodes by selectively slowing down learning of important neurons and by rehearsing a small set of examples of constant length. Our approach is shown to significantly improve prediction performance in new and unseen environments compared standard gradient descent approaches. Finally, we present qualitative experimental results in simulation and in real environments.