In the field of fluid mechanics, there has been a significant shift towards the integration of machine and deep learning techniques to address challenges in reduced-order modeling, flow feature analysis, and control, especially within the realm of active flow control (AFC) for objectives such as lift optimization and drag reduction. Deep learning has taken a central role in advancing state-of-the-art AFC methods by creating data-driven models that mitigate the computational demands of conventional Computational Fluid Dynamics (CFD) simulations, enabling real-time fluid control. Despite the predominance of models trained offline and focused on simple scenarios like laminar flow around bluff bodies, the utility of sophisticated learning methods in AFC has remained largely unexplored. This research introduces a novel benchmark in fluid dynamics—a soft robotic tentacle actuator—to evaluate the effectiveness of deep learning architectures in complex flow control situations. Through the comparison of online and offline learning frameworks for predicting system behavior, the study elucidates the strengths and limitations of deep learning networks in AFC. The findings underscore the constraints faced by deep learning architectures when dealing with aperiodic motions and demonstrate the significant benefits of adopting an online learning approach over offline training methods, thereby highlighting the advantages of adaptive learning strategies in complex AFC scenarios. The online learning framework displays more stability and increased quality of forecasts at larger time horizons.