Airborne collision avoidance systems (ACASs) form a key safety barrier by providing last-moment resolution advisories (RAs) to pilots for avoiding midair collisions. Intrinsic uncertainties, such as noise in ACAS input signals and variability in pilot performance, imply that the generation of RAs and the effectuated aircraft trajectories are nondeterministic processes. Existing ACAS validation methods reflect the intrinsic uncertainties to a limited extent only. This paper develops an agent-based model, which systematically captures uncertainties in ACAS input and pilot performance for Monte Carlo (MC) simulation of encounter scenarios. The agent-based model has been integrated with industry-specific implementations of Traffic Alert and Collision Avoidance System II and ACAS Xa in a novel collision avoidance validation and evaluation tool. Through illustrative MC simulation results, it is demonstrated that the intrinsic uncertainties can have a significant effect on the variability in timing and types of RAs, and subsequently on the variability in miss distance. Even the MC simulation estimated mean miss distance can differ significantly from the deterministically simulated miss distance. Most important, the tails of miss distance probability distributions and probabilities of near-midair collisions are affected. This stipulates that addressing intrinsic uncertainties through MC simulation is essential in evaluating ACASs.

Airborne Collision Avoidance Systems (ACAS) form a key safety barrier by providing last-moment resolution advisories (RAs) to pilots for avoiding mid-air collisions. For the generation of advisories ACAS uses various ownship state estimates (e.g. pressure altitude) and othership measurements (e.g. range, bearing). Uncertainties, such as noise in ACAS input signals and variability in pilot performance imply that the generation of RAs and the effectuated aircraft trajectories are non-deterministic processes. These can be analysed effectively by Monte Carlo (MC) simulation of the various uncertainties in encounter scenarios. Existing ACAS simulation tools reflect the intrinsic uncertainties to a limited extent only. In recognition of the need of an ACAS evaluation tool that supports MC simulation of these uncertainties, this paper develops an agent-based model, which captures uncertainties in ACAS input and pilot performance for the simulation of encounter scenarios, while using ACAS algorithms (TCAS II, ACAS Xa). The novel ACAS evaluation tool is named CAVEAT (Collision Avoidance Validation and Evaluation Tool). Through illustrative MC simulation results it is demonstrated that the uncertainties can have significant effect on the variability in timing and types of RAs, and subsequently on the variability in the closest point of approach (CPA). It is shown that even mean results of MC simulation can differ significantly from results of a deterministic simulation. Most importantly, the tails of CPA probability distributions are affected. This stipulates that addressing all intrinsic uncertainties through MC simulation is essential for proper evaluation of ACAS.