Rethinking Machine Learning Performance Metrics in Distributed Energy Resource Systems

Abstract

Machine learning (ML) techniques have been widely applied to distributed energy resource (DER) systems for a broad range of tasks, including load forecasting and operational control. However, evaluation practices remain inconsistent, conflating ML performance with DER system-level performance without a clear, systematic relationship, thereby limiting the interpretability and practical applicability of emerging ML applications in DER contexts. To this end, this paper introduces a hierarchical evaluation framework that systematically aligns ML performance with DER operational objectives. Specifically, this paper first discusses how ML-driven DER systems are currently evaluated, providing a comprehensive overview of performance evaluation metrics, their advantages and limitations, and practical insights. Given the challenges in evaluating ML-driven DER systems, we further organize performance evaluation metrics hierarchically, grouping them into six top-level categories that represent core DER objectives. Within each category, metrics are distinguished between ML-side and energy-side evaluation criteria. Consequently, our hierarchical DER-centric evaluation framework not only provides transparent interpretability of ML performance in the DER context but also clarifies recurring sources of confusion in current practice, particularly when ML performance is overgeneralized or implicitly assumed to reflect DER system-level performance. Finally, we highlight the challenges and identify future opportunities associated with the proposed hierarchical evaluation framework. This work serves as a valuable reference for researchers and engineers to better understand existing studies, as well as the potential, challenges, and opportunities in evaluating ML-driven DER solutions.