MK

M. Khosravi

info

Please Note

14 records found

The convergence rate of standard Q-learning is notoriously slow, demanding an immense volume of sample transitions to isolate the optimal policy in practical implementations. To mitigate this sluggish convergence, acceleration techniques utilizing second-order stochastic approximation methods have been introduced. This thesis proposes a novel preconditioned matrix gain framework built upon the foundations of Zap Q-learning. The proposed method directly addresses the high computational bottleneck associated with standard Zap Q-learning updates, reducing the per-iteration complexity to O(n2m2). Extensive empirical evaluations across diverse Markov Decision Process topologies demonstrate that the proposed framework yields significantly accelerated convergence to the optimal policy compared to existing baselines. Furthermore, a rigorous mathematical analysis ensures that the algorithm’s iterates are stable and asymptotically converge to the true optimal value function. ...
Explicit trajectory communication can be used to coordinate multiple robots, but communicating at every planning iteration can lead to congestion of the communication network, increase message delays and message loss. At the same time, collision-free trajectory planning is often formulated as a nonconvex optimization problem, which can converge to different locally optimal solutions across consecutive planning iterations.When this happens, a robots planned motion can switch between distinct high-level avoidance behaviors, such as passing an obstacle on the left versus on the right, which can lead to inefficient or unsafe behavior. Topology-based motion planners address this by explicitly computing multiple candidate motion plans that represent these different passing decisions, each associated with a distinct homotopy class. This work asks how (changes in) homotopy-class representations can trigger communication to reduce communication load while maintaining safe and efficient behavior. Building on the topology-driven trajectory optimization(T-MPC) approach of [ 1], we propose T-DMPC, a topology-aware distributed motion planner in which each robot, computes multiple guidance trajectories in distinct homotopy classes and refines them via parallel local trajectory optimization within the corresponding homotopy classes, selects a solution using a consistent decision rule that prioritizes the previously executed homotopy class, and communicates the selected trajectory using an event-triggered policy. Communication is triggered by homotopy changes and complemented by geometric-deviation and time-based triggers to bound trajectory staleness. In addition, the robots communicate a fallback trajectory during planning failures (e.g. infeasibility). We evaluate T-DMPC on antipodal swap maneuvers with 2 and3 robots in simulation and on physical robots, comparing against T-VMPC (no communication, constant-velocity predictions) and T-AMPC (always communicate). The experiments show that, T-DMPC achieves task duration and traveled distance comparable to T-AMPC and T-VMPC, while reducing communication to about 9.8% (2 robots) and 13.2% (3 robots) of planning iterations in simulation, and 17.8% (2 robots) and 13.2% (3 robots) in real-world experiments, with no observed physical collisions. Ablations however show that topology-change alone is insufficient for safety,motivating the combined trigger design. ...
Master thesis (2025) - P.A. Velchev, A.J.J. van den Boom, B. Shyrokau, M. Khosravi, A. Bertipaglia, Felipe Santafe
This paper presents a novel data-driven Reference Governor with Model Predictive Control integrating local motion replanning and path following for collision avoidance. Employing a model-free Reference Governor, the proposed solution utilises system knowledge through Bayesian Optimisation to augment predetermined evasive trajectories, minimising path-following errors and simultaneously ensuring obstacle safety margins. A single-track vehicle model in combination with non-linear tyre models is used to capture the vehicle’s dynamics. The optimised control action is the vehicle steering angle, whilst the Reference Governor optimises parameters of a sigmoid reference signal to minimise the tracking error and guarantee safety with respect to obstacles in emergency manoeuvres. The proposed approach is evaluated on a single lane change using a high-fidelity simulation environment and its performance is compared to a baseline controller integrating path following and obstacle avoidance. The results show a 14% reduction of safety critical overshoot, maximising obstacle safety distance and a four times lower controller cycle time compared to the baseline. Furthermore, through a robustness analysis, it is demonstrated that the proposed approach is more robust towards model mismatches and perception-based errors, as seen by average 30% and 40% reductions in near-miss and collision rates. ...

A Case Study, Data-Driven Model-Based Approach

This thesis explores the integration of hydrogen ( H2) into a residential hybrid energy hub and presents results based on a case study of such an energy hub located at The Green Village, an open-field lab environment at Delft university of Technology. The study focusses on analysing real operational data from the energy hub to assess the performance, behaviour, and integration of the system. The energy hub combines photovoltaic (PV) power generation, battery storage, and hydrogen-based components including an AEM electrolyser, hydrogen storage, and a PEM fuel cell.
First, a detailed data analysis of the energy hub components is performed which lays a basis for the model of the energy hub. Particular attention was paid to the ramp-up and ramp-down dynamics, the power consumption and generation capabilities, and hydrogen consumption and generation of the electrolyser and fuel cell, as these affect the overall efficiency and responsiveness of the system. In addition, insights are given into the real capacity of the energy hub, and a comparison between the intended operation of the energy hub and the real operation is given. Following insights gained from the data analysis, a first step toward integrating machine learning into the Energy Management System (EMS) was taken. One potential improvement identified was the use of a machine learning algorithm that uses weather forecasts into EMS decision-making for the electrolyser. This study explores both binary classification and regression models using outside temperature and PV inverter power as inputs. Since inverter power correlates strongly with solar irradiance, these inputs were considered sufficient for developing a preliminary machine learning model aimed at enabling smarter control of the electrolyser.
A data-driven Simulink model of the energy hub was developed to simulate various operational scenarios, sizes, and edge cases. These simulations revealed how the behaviour of the system changes under different profiles of PV generation and residential demand, even when total energy consumption remains constant.
The data results create a solid foundation for the development of the Simulink model to run experiments with. The model has successfully demonstrated that the integration of hydrogen into the energy hub facilitates the coverage of seasonal energy demands, adding flexibility, and flattening the energy demands of the grid. However, it also showed that the system performance is highly sensitive to operating conditions and system sizing. Recommendations for future energy hub upgrades are provided through control optimisation and system sizing.
...
Master thesis (2025) - S. Volk, M. Khosravi, R. Rahimi Baghbadorani, M. Guo, T.J.J. Bijl
Achieving net-zero emissions by 2050 requires a transition to renewable energy sources. With this transition comes the electrification of industrial processes, transport, and heating. However, this electrification also increases the risk of grid congestion, and grid capacity becomes scarce. In the Netherlands alone, more than 12,000 companies are currently awaiting new or expanded grid connections, which is slowing down the energy transition.

One solution to this problem is the Energy Hub (EH). This smart, decentralized system locally coordinates the generation, storage, conversion, and consumption of energy. Special juridical forms of an Energy Hub have recently been developed, such as the Group Transport Agreement (GTA) and the Capacity Limiting Contract (CLC). This thesis focuses on allocating electrical power within Energy Hubs operating under group contracts.

The core challenge lies in the collective management of this shared, limited power capacity. An Energy Management System (EMS) is required to allocate power to diverse flexible assets dynamically. These assets, including a battery, solar park, batch process, and refrigeration unit, are modeled to simulate an Energy Hub that leverages the flexibility of these assets, ensuring optimal utilization of capacity and minimizing both collective and individual costs. For this, a Model Predictive Control (MPC) scheme is employed. By proposing and evaluating an integral model for power allocation, this thesis demonstrates that the Energy Hub can simultaneously reserve significantly less overall grid capacity and reduce operating costs while strictly adhering to the limits of the aforementioned group contracts. ...
The speed and precision required in the precision motion industry is an ever-growing challenge. Linear control offers intuitive frequency-domain controller design methods that are based on a frequency response function (FRF) of the plant, which is obtained solely from measurement data. Unsurprisingly, linear controllers account for over 90% of the controllers currently used in the industry. However, as linear controllers are subject to inherent performance limitations such as Bode's gain-phase relationship and the waterbed effect, research on nonlinear control solutions to overcome these limitations is ubiquitous. Reset control first showed up in 1958 with the Clegg integrator (CI). According to a sinusoidal-input-describing-function analysis, the CI provides reduced phase lag compared to a linear integrator, suggesting Bode's gain-phase relationship can be overcome and consequently allowing for performance surpassing that of linear control systems. The drawbacks of the CI are the possible emergence of limit cycles and the excitation of high-frequent modes of the system originating from higher-order harmonics of the CI's input caused by discontinuities in the control signal. The generalised-first-order-reset-element-based integrator (GFbI) is the only reset element that can prevent the emergence of limit cycles and can reduce the generation of higher-order harmonics, whilst retaining the advantageous reduced phase lag the CI provides and allowing for closed-loop controller design based on a plant FRF while taking the effects of higher-order harmonics into account, matching design methods of linear controllers. Optimally tuning a reset controller is a complex and time-consuming task. Moreover, established tuning rules are still lacking. This work facilitates designing a reset controller containing a GFbI element by providing two contributions to utilise the potential of the GFbI element to improve upon linear control. The first is a comparative study on the effect of the controller element sequence, aimed at reducing the negative consequences of the higher-order harmonics generated by the reset element. The second is the proposal of an FRF-based optimisation algorithm utilising frequency-domain performance prediction methods to automatically tune a reset controller containing a GFbI element to adhere to the imposed constraints and maximise the benefit gained from the nonlinear element. Validation using a simulated and physical wire bonder showed the algorithm successfully tuned four different reset controllers using two different sequences. The performance of the tuned reset controllers was compared to that of equally-well-tuned linear controllers. In the first use case with the goal of suppressing a dominant vibration in the error signal, the median root-mean-square error was reduced by 16.245%. In the second use case, the goal was to improve the settling time, which was achieved by a median of 9.791%. The best results were achieved using a sequence in which the higher-order harmonics were avoided from passing through the lead filter in open loop, mitigating amplification thereof. The proposed tuning algorithm proved able to tune reset controllers containing a GFbI element such that the performance of linear control based on frequency-domain performance prediction metrics was surpassed, where two use cases confirmed the predicted performance increase through time-domain simulations and experiments on a physical setup. ...
There is an ever-increasing demand for faster and more accurate motion stages in the high-tech industry. In precision positioning systems, lightly damped higher-order resonance modes can induce undesirable vibrations that degrade system performance and accuracy. These resonances pose additional challenges in non-collocated dual-stage positioning systems, where they significantly limit control bandwidth. Although conventional notch filters are commonly used alongside tracking controllers to enhance bandwidth, they lack robustness when faced with system parameter uncertainties. Moreover, the effects of the parasitic resonance on disturbance rejection remain. Active damping control has been successfully used to mitigate issues related to the primary resonance mode, but its application to higher-order parasitic modes has not been explored. This research introduces a novel control strategy, High-Pass Positive Position Feedback (HP-PPF), designed specifically for the active damping of higher-order, non-collocated parasitic modes in positioning systems. The proposed method incorporates a second-order high-pass filter within a positive feedback loop, effectively attenuating the parasitic resonance. Integrated with a PID tracking controller in a dual-loop configuration, this method enhances disturbance rejection, noise suppression, and robustness against model uncertainties, overcoming limitations of traditional notch filter-based methods while maintaining comparable tracking performance. The proposed control architecture is validated through a proof-of-concept experimental setup that demonstrates the effectiveness of the underlying mathematical framework.
...
Master thesis (2024) - W. Tang, T. Keviczky, Y. Li, S. Walker, M. Khosravi
In light of the pressing challenges posed by global climate change and the imperative to reduce CO2 emissions, innovative approaches in energy management are critically important. This thesis presents an exploration of heat pumps integrated with Thermal Storage System (TES) systems, an area of research and application pivotal for enhancing energy efficiency and environmental sustainability. The combination of heat pumps and TES systems emerges as a key factor in reducing greenhouse gas emissions and optimizing the utilization of renewable energy. Such integration plays a crucial role in minimizing operational costs, reducing environmental negative impact, and augmenting system efficiency by enabling the storage and later use of energy from renewable sources. Moreover, this integration facilitates the effective management of demand-side energy, bolstering the capacity to incorporate fluctuating renewable generation into the energy grid. This is achieved by dynamically load shifting to balance energy supply and demand.

A central aspect of this thesis is the utilization of Model Predictive Control (MPC) for advanced energy management. The research delves into the use of MPC to optimize the operational economy of the system, aiming to maximize cost-efficiency. Additionally, an innovative MPC-based Demand-Side Management (DSM) strategy is introduced. This strategy involves two key steps: initially establishing a model to assess the system's energy flexibility, followed by harnessing this flexibility to respond to demand fluctuations. Such an approach facilitates dynamic adaptation to varying energy demands, ensuring optimal resource utilization. The predictive capability of MPC, which accounts for future disturbances including demand forecasts, electricity pricing, and weather conditions, is exploited to improve the system’s responsiveness and operational efficiency.

Experimentation was conducted both in simulations and through the implementation in real systems. These practical applications demonstrated significant savings in energy costs and energy consumption, achieving economical operation. Furthermore, the execution of the proposed two-step demand-side management strategy successfully managed energy demands. This not only underscores the practical effectiveness of the proposed system but also highlights its potential in real-world scenarios.

In summary, this research underscores how the integration of heat pumps, TES systems, and advanced control strategies like MPC can significantly improve energy efficiency, reduce operational costs, and enhance energy flexibility. It highlights the vital role of incorporating sophisticated control mechanisms into sustainable energy systems, aligning with the strategic goals of modern energy policies and advancing the field of sustainable energy management. ...
This research presents a novel data-based modal control method for actively dampening the flexible mode in a multi-input multi-output (MIMO) system. Traditional passive damping methods add significant mass to the system, making recent advances in sensor and actuator technology, such as lightweight piezoelectric materials, a more appealing solution. The key contribution of this research is a novel modal decoupling method for active damping that uses the MIMO frequency response function to circumvent the need for a parametric model. This method facilitates the design of a single-input, single-output (SISO) controller that actively dampens a flexible mode using all available sensors and actuators. This approach significantly reduces the complexity of the controller design and tuning effort compared to the conventional decentralized control architecture. Experimental validation is carried out on a cantilever beam, which shows near-perfect isolation of the mode of interest. The study's findings may offer critical insights for future mechatronics systems, enabling the creation of more efficient and powerful machines. ...
Undesired vibrations are one of the most significant sources of error in any type of mechatronic system or component. The emerging field of elastic (locally resonant) metamaterials offers a viable solution to successfully suppress these by generating bandgaps in both resonant and non-resonant regions. In this thesis, metamaterials in a sensor-actuator configuration using piezoelectric transducers are employed to generate vibration attenuation regions in beam-like structures. The main contribution of this thesis consists in studying the practical issues involved in the experimental implementation of such metamaterial architectures, often overlooked in the literature. To this aim, parasitic dynamics such as time delay and RC roll-off characteristics of piezoelectric transducers are considered, and their influence on controller choice is evaluated. The research was conducted using full model simulations in SPACAR and an experimental setup. The RC roll-off characteristic of piezoelectric transducers was found to be significant in limiting the bandgap generation capabilities of the system in non-resonant regions. The reason for this was the added phase caused by the parasitic effect, which required a reduction in controller gain for stability and ultimately reduced the bandgap performance. This was not the case for resonant bandgaps, where the phase lead was compensated by the increase in gain at the resonance. This ultimately allowed for optimal resonant bandgaps to be generated and observed. Methods to compensate for such parasitic effects are proposed and suggestions on how to implement these to attain non-resonant bandgaps are made. ...
Master thesis (2023) - J.V. Das, S. Grammatico, M. Khosravi, E. Benenati
Navigation systems in an Autonomous Vehicles (AV) can be divided into two parts: a path planning block which takes in the environmental data and rules to design a collision-free obstacle and a vehicle control and tracking block which generates actuator inputs for the AV to follow the reference path generated by the path planning block. Each task is fulfilled by a different algorithm with its own performance indices. These algorithms are not usually designed to get the best overall vehicle performance but the best performance of their respective blocks. A planned path that the vehicle cannot follow can therefore be generated and can lead to high tracking error and in some cases collision with obstacles. This can be solved by integrating path planning and vehicle control blocks with the dynamics of the AV.

The goal of this graduation project is therefore to develop an integrated planning and vehicle control algorithm for an Autonomous Vehicles (AV). This is done by integrating a novel-Artificial Potential Fields (APF) with Model Predictive Control (MPC) to solve both path planning and vehicle control using a single optimization problem. The addition of the AV and the Obstacle Vehicle (OV) dynamics to the optimization problem as prediction models along with recursive computation can determine accurate inputs to be given to the AV.

Unlike traditional APF-based path planning where the minimum potential path is generated as the result of a gradient descent method applied on the available map data and obstacle information, the APF is added as a cost to the objective function of the MPC based optimization problem to find the minimum potential path. By using a receding horizon approach for solving the final optimization problem, the potential field can be updated at each time step to avoid moving obstacles. The dynamics of the vehicle added to the optimization problem include both lateral and longitudinal dynamics and are linearized at each time step at the current state of the AV.

This however generates a path which does not travel in the centre of the lane and makes risky manoeuvres. Therefore, a Mixed-Integer Model Predictive Control (MIMPC) algorithm with logical constraints is used to generate an optimal lane to travel in. This optimal lane is used to generate a road potential which can guide the vehicle to the centre of the optimal lane. The MIMPC and the APF-MPC algorithms are run successively to generate a collision-free path.

The logical constraints, also called MLD constraints are converted into a set of linear inequalities with the introduction of logical variables. These logical variables are used to represent individual logical constraints based on the states of the system and on a combination of logical and state constraints. A novel-APF inspired by the Yukawa Potential [?] is designed to represent each obstacle. A convex representation of this non-convex obstacle potential is formulated to simplify the optimization problem. The convex representation of the obstacle APF is obtained by approximating it using a region-based APF where the region is defined by the position of the AV around the obstacle. This is further simplified by approximation using a quadratic Taylor-series expansion.

The simulation was performed on MATLAB on a two-lane road with multiple obstacles. The thesis report ends with a discussion on future work to be taken to further enhance the performance of the controller and to make it road-ready. ...
Master thesis (2023) - R. van Wilsem, A.A. Kana, A. Jarquin Laguna, M. Khosravi, Sven van den Munckhof
The Jones Act, limited US port facilities, and absence of Jones Act-compliant installation vessels pose significant challenges for offshore wind farm installations. These factors force contractors to explore new installation strategies, such as feedering. Feedering is an installation strategy where the wind turbine installation vessel remains stationed at the offshore wind farm, while a feeder vessel transports all wind turbine components from the marshaling port to the installation site. With the rise of U.S. wind farm developments on the East Coast, it becomes apparent that alternative vessel designs and strategies will play a vital role in the near future. Therefore it is crucial to gain an understanding of Jones Act complaint vessel designs and strategies.

In this thesis, a new method for optimizing feeder vessel design concurrently with a wind turbine installation strategy is introduced. The approach combines multi-agent discrete-event simulation and design space exploration to define the optimum within the design space. The proposed method facilitates the evaluation and comparison of the operational performance of design configurations using historical environmental data, operating limits, and operational characteristics. Importantly, the proposed approach accommodates for the interdependency of operations making it suitable for the design and evaluation of repetitive multi-tasked operations. This method provides an improvement over the commonly used workability percentage and thereby allows for improved and fit-for-purpose designs.

A case study is performed based on Vineyard Wind WTG installation works that shows the potential of the proposed approach and the impact of vessel size, installation strategy and equipment characteristics on operational performance. This research offers new insights into the optimization of offshore wind farm installation processes and vessel designs paving the way for more efficient and effective installations in the rapidly growing U.S. wind energy sector.
...
Master thesis (2023) - A. ATHREY, B.H.K. De Schutter, S. Shi, M. Khosravi, Othmane Mazhar
This thesis addresses the Learning-Based Control (LBC) of unknown partially observable systems in the Linear Quadratic (LQ) paradigm. In this setting of learning-based LQ control, the control action influences not only the control performance but also the rate at which the system is being learnt, causing a conflict between learning and control (exploration and exploitation), which is particularly challenging to address. This thesis aims to develop a novel LBC algorithm for unknown partially observable systems in the LQG setting that is computationally efficient and can guarantee an optimal exploration-exploitation trade-off, quantified by a metric called regret. The regret quantifies the cumulative performance gap between the LBC policy and the ideal controller having full knowledge of the true system dynamics. The contributions in this thesis involve a novel LBC algorithm deployed in a two-phase structure. The first phase involves injecting Gaussian input signals to obtain an initial system model. The subsequent second phase deploys the proposed LBC strategy in an episodic setting, where the model is updated for each episode, and the resulting updated LQG controller is applied with additive Gaussian signals for exploration. In addition, the thesis establishes strong theoretical guarantees on optimal regret growth. ...
As the share of renewable energy generation increases, the need for energy storage also increases. Therefore, there is a need for better storage representation in the current energy modelling tools. In the present day,
the longer-term energy storage systems are not fully represented since, for existing storage systems, the self-serving nature of these leads to participation in multiple energy markets. This is because participating in other markets, like the balancing markets, can lead to higher overall profits than a storage system only participating in the wholesale market.
This thesis investigates different energy storage technologies and multiple prominent storage applications for grids. Furthermore, an overview of the European energy markets will be examined, and different design options will be discussed. These markets include frequency containment reserve (FCR), frequency regulation reserves (aFRR/mFRR) and the wholesale markets. The review of storage technologies, applications, and available markets has led to the development and simulation of single-purpose energy storage models fulfilling grid applications.
By combining the specific purpose models, a complete energy market and energy storage model representation could be created. The model created is unique since the complete energy system model allows energy storage systems to optimally dispatch over multiple markets while at the same time also influencing these markets. Multiple cases were investigated using this model, such as the influence of increasing storage capacity on the wholesale and balancing market and the influence of storage systems just performing one service, so only regulation, arbitrage or peak-shaving. Based on the model results, recommendations are made on improving the current energy market designs and how to better represent storage systems in existing energy system models. ...