Model-based reinforcement learning can improve sample efficiency by learning a model of the environment and using it for planning. In continuous-control tasks, this allows an agent to evaluate many candidate action sequences before acting. However, planning with a learned model also introduces a failure mode: the planner optimizes a learned return estimate rather than the true environment return, and can therefore exploit errors in the dynamics model, reward model, or value function.
This thesis studies how the model predictive control objective used in TD-MPC-style agents can be made more reliable. The main contribution is EfficientTDMPC, a method that modifies the planner objective by aggregating return estimates across multiple dynamics heads and rollout depths, and by applying disagreement-based pessimism during reanalyze. These changes aim to reduce the variance and exploitability of model-based return estimates while preserving the sample efficiency benefits of latent model-based planning.
EfficientTDMPC is the new state-of-the-art on HumanoidBench-Hard and the hard DeepMind Control Suite (DMC) while matching the state-of-the-art on Easy DMC. The thesis also discusses adaptive horizon selection as a future direction, arguing that planning depth should be treated as part of the uncertainty-aware planning objective. Overall, the thesis shows that the reliability of the learned planning objective is a central design problem in sample-efficient model-based reinforcement learning.
... Read more

Ultrasonic transducers are commonly used in a variety of medical, industrial, and consumer devices, converting electrical energy into high-frequency sound waves. These devices find extensive applications in fields such as medical imaging, non-destructive testing, distance measurement, and cleaning processes. However, these transducers suffer from a narrow operating frequency range caused by a steep frequency response curve with a prominent resonance peak. Existing passive compensation methods using filters are limited due to the individual characteristics of transducers and their susceptibility to process variations, making generic compensation filters impractical. Additionally, the frequency response of transducers changes over time, input power, and environmental conditions, further complicating compensation efforts.

The objective of this project is to overcome these challenges by creating an integrated solution that can provide an ultrasonic transducer system with a consistent frequency response despite external disturbances. The proposed system will incorporate non-linear dynamics characterization and compensation, which are currently lacking in integrated solutions. By accurately characterizing the transducer’s non-linear behavior and compensating for it, the system will overcome the drawbacks associated with passive compensation.

The proposed integrated system holds promising implications for various applications, including medical imaging, material testing, and industrial processes. By mitigating the limitations associated with the narrow operating frequency range of ceramic piezoelectric transducers, this research project contributes to the advancement of ultrasonic technology and its broader impact on diverse industries. ... Read more