Calibration of Car Ownership Models for Population Group Analysis

Abstract

Predicting car ownership of households is a key step in transport modeling, traditionally performed using discrete choice models such as the multinomial logit model, while alternative machine learning algorithms have emerged in recent years. Existing evaluation of prediction models typically focuses on household-level metrics, such as log loss and probability calibration, implicitly assuming that accurate household-level predictions also lead to reliable predictions for population groups. In this paper we demonstrate that this assumption does not necessarily hold. Models that provide well-calibrated probabilities at the household level can still produce inaccurate probability distributions when predictions are aggregated over groups. Motivated by this observation, we identify subgroup-level calibration as a distinct prediction objective, and propose a model-agnostic calibration method that explicitly improves the accuracy of predicted distributions for population groups. In a case study based on a large dataset of Dutch households, we evaluate multinomial logit, ordered logit, random forest and gradient boosting models. The results show that subgroup-level prediction errors can remain even when household-level calibration metrics indicate good performance, while the proposed post-processing approach improves subgroup-level predictions across different model classes. These findings highlight the importance of evaluating prediction models at the level at which they are ultimately used, which is particularly relevant for transport policy applications involving equity and population-group analysis.