Federated learning stands as an approach to train machine learning models on data residing at multiple clients, but where data must remain private to the client it belongs to. Despite its promise, federated learning faces significant challenges, particularly when dealing with non-IID and non-stationary data. A model trained on non-stationary data can be subject to concept shift, where the data used for training faces a sudden change of concept, leading to a large performance degradation when classifying data under the new concept. This research focuses on comparing the performance of federated and centralized models under such conditions. Our objective is to evaluate the extent to which federated models are more affected by concept shift than their centralized counterparts. Through a series of experiments involving image (CIFAR-10) and tabular data (2-dimensional, linearly separable, binary-classification), we demonstrate that while federated models can achieve performances close to centralized models, they exhibit greater sensitivity to data complexity and distribution shifts. Our findings suggest that, despite centralized models being better than federated ones, the gain in performance from gathering data in one place might not outweigh the privacy concerns. Furthermore, we also find that the accuracy under concept shift is dependent on the performance on original data.