Airborne Collision Avoidance Systems (ACAS) form a key safety barrier by providing last-moment resolution advisories (RAs) to pilots for avoiding mid-air collisions. For the generation of advisories ACAS uses various ownship state estimates (e.g. pressure altitude) and othership measurements (e.g. range, bearing). Uncertainties, such as noise in ACAS input signals and variability in pilot performance imply that the generation of RAs and the effectuated aircraft trajectories are non-deterministic processes. These can be analysed effectively by Monte Carlo (MC) simulation of the various uncertainties in encounter scenarios. Existing ACAS simulation tools reflect the intrinsic uncertainties to a limited extent only. In recognition of the need of an ACAS evaluation tool that supports MC simulation of these uncertainties, this paper develops an agent-based model, which captures uncertainties in ACAS input and pilot performance for the simulation of encounter scenarios, while using ACAS algorithms (TCAS II, ACAS Xa). The novel ACAS evaluation tool is named CAVEAT (Collision Avoidance Validation and Evaluation Tool). Through illustrative MC simulation results it is demonstrated that the uncertainties can have significant effect on the variability in timing and types of RAs, and subsequently on the variability in the closest point of approach (CPA). It is shown that even mean results of MC simulation can differ significantly from results of a deterministic simulation. Most importantly, the tails of CPA probability distributions are affected. This stipulates that addressing all intrinsic uncertainties through MC simulation is essential for proper evaluation of ACAS.