Surrogate Safety Measures (SSMs) are used to express road safety in terms of the safety risk in traffic conflicts. Typically, SSMs rely on assumptions regarding the future evolution of traffic participant trajectories to generate a measure of risk, restricting their applicability to scenarios where these assumptions are valid. In response to this limitation, we present the novel Probabilistic RISk Measure derivAtion (PRISMA) method. The objective of the PRISMA method is to derive SSMs that can be used to calculate in real time the probability of a specific event (e.g., a crash). The PRISMA method adopts a data-driven approach to predict the possible future traffic participant trajectories, thereby reducing the reliance on specific assumptions regarding these trajectories. Since the PRISMA is not bound to specific assumptions, the PRISMA method offers the ability to derive multiple SSMs for various scenarios. The occurrence probability of the specified event is based on simulations and combined with a regression model, this enables our derived SSMs to make real-time risk estimations. To illustrate the PRISMA method, an SSM is derived for risk evaluation during longitudinal traffic interactions. Since there is no known method to objectively estimate risk from first principles, i.e., there is no known risk ground truth, it is very difficult, if not impossible, to objectively compare the relative merits of two SSMs. Instead, we provide a method for benchmarking our derived SSM with respect to expected risk trends. The application of the benchmarking illustrates that the SSM matches the expected risk trends. Whereas the derived SSM shows the potential of the PRISMA method, future work involves applying the approach for other types of traffic conflicts, such as lateral traffic conflicts or interactions with vulnerable road users.@en

Objective: The amount of collected field data from naturalistic driving studies is quickly increasing. The data are used for, among others, developing automated driving technologies (such as crash avoidance systems), studying driver interaction with such technologies, and gaining insights into the variety of scenarios in real-world traffic. Because data collection is time consuming and requires high investments and resources, questions like “Do we have enough data?,” “How much more information can we gain when obtaining more data?,” and “How far are we from obtaining completeness?” are highly relevant. In fact, deducing safety claims based on collected data—for example, through testing scenarios based on collected data—requires knowledge about the degree of completeness of the data used. We propose a method for quantifying the completeness of the so-called activities in a data set. This enables us to partly answer the aforementioned questions. Method: In this article, the (traffic) data are interpreted as a sequence of different so-called scenarios that can be grouped into a finite set of scenario classes. The building blocks of scenarios are the activities. For every activity, there exists a parameterization that encodes all information in the data of each recorded activity. For each type of activity, we estimate a probability density function (pdf) of the associated parameters. Our proposed method quantifies the degree of completeness of a data set using the estimated pdfs. Results: To illustrate the proposed method, 2 different case studies are presented. First, a case study with an artificial data set, of which the underlying pdfs are known, is carried out to illustrate that the proposed method correctly quantifies the completeness of the activities. Next, a case study with real-world data is performed to quantify the degree of completeness of the acquired data for which the true pdfs are unknown. Conclusion: The presented case studies illustrate that the proposed method is able to quantify the degree of completeness of a small set of field data and can be used to deduce whether sufficient data have been collected for the purpose of the field study. Future work will focus on applying the proposed method to larger data sets. The proposed method will be used to evaluate the level of completeness of the data collection on Singaporean roads, aimed at defining relevant test cases for the autonomous vehicle road approval procedure that is being developed in Singapore.@en

The development of new assessment methods for the performance of automated vehicles is essential to enable the deployment of automated driving technologies, due to the complex operational domain of automated vehicles. One contributing method is scenario-based assessment in which test cases are derived from real-world road traffic scenarios obtained from driving data. Given the complexity of the reality that is being modeled in these scenarios, it is a challenge to define a structure for capturing these scenarios. An intensional definition that provides a set of characteristics that are deemed to be both necessary and sufficient to qualify as a scenario assures that the scenarios constructed are both complete and intercomparable. In this article, we develop a comprehensive and operable definition of the notion of scenario while considering existing definitions in the literature. This is achieved by proposing an object-oriented framework in which scenarios and their building blocks are defined as classes of objects having attributes, methods, and relationships with other objects. The object-oriented approach promotes clarity, modularity, reusability, and encapsulation of the objects. We provide definitions and justifications of each of the terms. Furthermore, the framework is used to translate the terms in a coding language that is publicly available.@en

The development of safety validation methods is essential for the safe deployment and operation of Automated Driving Systems (ADSs). One of the goals of safety validation is to prospectively evaluate the risk of an ADS dealing with real-world traffic. ISO 26262 and ISO/DIS 21448, the leading standards in automotive safety, provide an approach to estimate the risk where the former focuses on risks due to potential malfunctioning of components and the latter focuses on risks due to possible functional insufficiencies. The main shortcomings of the approach provided in ISO 26262 are that it depends on subjective judgments of safety experts and that only a qualitative risk estimation is performed. ISO/DIS 21448 addresses these shortcomings partially by providing statistical methods to guide the safety validation, but no complete method is provided to quantify the risk. The first objective of this article is to propose a method to estimate the risk of an ADS in a more quantitative and objective manner. A data-driven approach is used to rely less on subjective judgments of safety experts. The output of the method is the expected number of injuries in a potential collision. Thus, the method is quantitative, the result is easily interpretable, and the result can be compared with road crash statistics. The second objective is to provide a method that supports the risk assessment as stipulated by the ISO 26262 and ISO/DIS 21448 standards by decomposing the quantified risk into the 3 aspects of risk as mentioned in these standards: exposure, severity, and controllability. The proposed methods are illustrated by means of a case study in which the risk is quantified for a longitudinal controller in 3 different types of scenarios. The code of the case study is publicly available.@en

The development of assessment methods for the performance of Automated Vehicles (AVs) is essential to enable the deployment of automated driving technologies, due to the complex operational domain of AVs. One candidate is scenario-based assessment, in which test cases are derived from real-world road traffic scenarios obtained from driving data. Because of the high variety of the possible scenarios, using only observed scenarios for the assessment is not sufficient. Therefore, methods for generating additional scenarios are necessary. Our contribution is twofold. First, we propose a method to determine the parameters that describe the scenarios to a sufficient degree while relying less on strong assumptions on the parameters that characterize the scenarios. By estimating the probability density function (pdf) of these parameters, realistic parameter values can be generated. Second, we present the Scenario Representativeness (SR) metric based on the Wasserstein distance, which quantifies to what extent the scenarios with the generated parameter values are representative of real-world scenarios while covering the actual variety found in the real-world scenarios. A comparison of our proposed method with methods relying on assumptions of the scenario parameterization and pdf estimation shows that the proposed method can automatically determine the optimal scenario parameterization and pdf estimation. Furthermore, it is demonstrated that our sr metric can be used to choose the (number of) parameters that best describe a scenario. The presented method is promising, because the parameterization and pdf estimation can directly be applied to already available importance sampling strategies for accelerating the evaluation of AVs.@en

Scenario-based methods for the assessment of Automated Vehicles (AVs) are widely supported by many players in the automotive field. Scenarios captured from real-world data can be used to define the scenarios for the assessment and to estimate their relevance. Therefore, different techniques are proposed for capturing scenarios from real-world data. In this paper, we propose a new method to capture scenarios from real-world data using a two-step approach. The first step consists in automatically labeling the data with tags. Second, we mine the scenarios, represented by a combination of tags, based on the labeled tags. One of the benefits of our approach is that the tags can be used to identify characteristics of a scenario that are shared among different type of scenarios. In this way, these characteristics need to be identified only once. Furthermore, the method is not specific for one type of scenario and, therefore, it can be applied to a large variety of scenarios. We provide two examples to illustrate the method. This paper is concluded with some promising future possibilities for our approach, such as automatic generation of scenarios for the assessment of automated vehicles.@en

The safety assessment of Automated Vehicles (AVs) is an important aspect of the development cycle of AVs. A scenario-based assessment approach is accepted by many players in the field as part of the complete safety assessment. A scenario is a representation of a situation on the road to which the AV needs to respond appropriately. One way to generate the required scenario-based test descriptions is to parameterize the scenarios and to draw these parameters from a probability density function (pdf). Because the shape of the pdf is unknown beforehand, assuming a functional form of the pdf and fitting the parameters to the data may lead to inaccurate fits. As an alternative, Kernel Density Estimation (KDE) is a promising candidate for estimating the underlying pdf, because it is flexible with the underlying distribution of the parameters. Drawing random samples from a pdf estimated with KDE is possible without the need of evaluating the actual pdf, which makes it suitable for drawing random samples for, e.g., Monte Carlo methods. Sampling from a KDE while the samples satisfy a linear equality constraint, however, has not been described in the literature, as far as the authors know.In this paper, we propose a method to sample from a pdf estimated using KDE, such that the samples satisfy a linear equality constraint. We also present an algorithm of our method in pseudo-code. The method can be used to generating scenarios that have, e.g., a predetermined starting speed or to generate different types of scenarios. This paper also shows that the method for sampling scenarios can be used in case a Singular Value Decomposition (SVD) is used to reduce the dimension of the parameter vectors.@en

Automated Vehicles (AVs) have a great potential to change transport fundamentally by making it safer, by reducing travel time, and by increasing mobility and accessibility for all. The level of automation of these vehicles determines the extent to which the driver’s task is accomplished by the AV. With the increasing number of AVs entering the market, the level of automation of these vehicles is increasing. The increasing level of automation will cause a paradigm shift: traditionally, human drivers are responsible for the behavior of the vehicle, even if the vehicle is momentarily controlled by an Automated Driving System (ADS), but with increasing levels of automation, the human driver will no longer be solely responsible. So, the accountability and liability shift from the driver to the vehicle manufacturer, the operator of the vehicle (fleet), and/or the (vehicle) authorities. Due to this paradigm shift, for higher levels of automation, it can no longer be assumed that the human driver intervenes whenever the ADS does not respond appropriately. To guarantee that these ADSs respond appropriately in nearly all situations, new methods for assessing ADSs are required. Scenario-based assessment is an approach for assessing AVs that is broadly supported by the automotive field. With a scenario-based assessment, the AV under test is subjected to many different test scenarios. These test scenarios resemble situations that may be encountered in real-world traffic, to see whether the AV responds appropriately to these scenarios. One of the main challenges with scenario-based assessment of an AV with a high level of automation is to come up with a set of test scenarios that provides enough confidence that the AV responds appropriately in nearly all situations. One popular approach is to use real-world data that contain scenarios from real-world traffic as a source to automatically generate test scenarios. This dissertation describes new methods for improving this data-driven scenario-based assessment of AVs. The first contribution of this dissertation is a comprehensive and operable definition of the term scenario in the context of scenario-based assessment of AVs. We define a scenario as a quantitative description of the relevant characteristics and activities and/or goals of the ego vehicle(s), the static environment, the dynamic environment, and all events that are relevant to the ego vehicle(s) within the time interval between the first and the last relevant event. A scenario category is defined as the qualitative counterpart of a scenario and can be regarded as an abstraction of a scenario. To enable a computer to store, communicate, interact with, and interpret scenarios, an Object-Oriented Framework (OOF) is proposed in which scenarios, scenario categories, and their building blocks are defined as classes of objects having attributes, methods, and relationships. The advantage of the OOF is that it promotes clarity, modularity, and reusability of the objects that constitute a scenario. The second contribution is a novel metric for quantifying the degree of completeness of the collected data that are used for the data-driven scenario-based assessment of AVs. The data are used to estimate unknown probability density functions (pdfs) of the important parameters that are used to describe scenarios. The proposed completeness metric is based on the expected approximation error, which is the discrepancy between the real pdf and the estimated pdf: a lower approximation error indicates a higher degree of completeness. The third contribution is a novel method for capturing scenarios of a specific scenario category from a data set. For example, the provided method can capture all cut-in scenarios from a data set. One of the benefits of the method is that characteristics of a scenario that are shared among different scenario categories need to be identified only once. As a result, the provided method is easily applied to a wide range of scenario categories, such that a wide variety of scenarios can be obtained from the data. The fourth contribution is the proposal of two complementary methods for generating test scenarios for AVs. The first method automatically determines the parameters that best describe the scenarios of a specific scenario category. The underlying, unknown pdf of the parameters is estimated and scenarios are generated by sampling parameter values from the estimated pdf. The second method enables the conditional sampling of parameter values, which can be used to, e.g., generate scenarios with predefined starting conditions. The benefits of the presented methods are that the generated scenarios are representative of real-world scenarios, they cover the actual variety found in real-world traffic, and they extend the variety found in the collected data. To measure the extent to which the generated scenarios indeed represent real-world scenarios while covering the actual variety found in real-world traffic, the novel Scenario Representativeness metric is proposed. The fifth contribution is the proposal of two novel methods for quantifying the risk of an AV. Both methods calculate the risk by combining the outcome of virtual simulations of scenarios generated using the aforementioned methods and the estimated likelihood of these scenarios. The first method quantifies the risk prospectively, i.e., before the actual deployment of the AV on public roads. The quantified risk supports the risk assessment activities of ISO 26262 and ISO 21448, the leading standards in automotive safety. These standards decompose the risk into three aspects: exposure, severity, and controllability. Whereas safety experts’ opinions are traditionally used to provide qualitative, subjective ratings for each of these three aspects, our proposed method computes these aspects in a data-driven, quantitative manner. The second method is the novel data-driven Probabilistic RISk Measure derivAtion (PRISMA) method, which is used to derive Surrogate Safety Measures (SSMs) that estimate the probability of a specific event (e.g., a crash) in real time. As opposed to existing SSMs, which are only applicable in specific types of scenarios, the PRISMA method can be used to derive multiple SSMs for different types of scenarios. The work presented in this dissertation thus makes a substantial contribution to the full integration of a scenario-based assessment for the type approval of AVs. This, in turn, brings us closer to the large-scale deployment of AVs on public roads.@en

Automated vehicles (AVs) are expected to increase traffic safety and traffic efficiency, among others by enabling flexible mobility-on-demand systems. This is particularly important in Singapore, being one of the world's most densely populated countries, which is why the Singaporean authorities are currently actively facilitating the deployment of AVs. As a consequence, however, the need arises for a formal AV road approval procedure. To this end, a safety assessment framework is proposed, which combines aspects of the standardized functional safety design methodology with a traffic scenario-based approach. The latter involves using driving data to extract AV-relevant traffic scenarios. The underlying approach is based on decomposition of scenarios into elementary events, subsequent scenario parametrization, and sampling of the estimated probability density functions of the scenario parameters to create test scenarios. The resulting test scenarios are subsequently employed for virtual testing in a simulation environment and physical testing on a proving ground and in real life. As a result, the proposed assessment pipeline thus provides statistically relevant and quantitative measures for the AV performance in a relatively short time frame due to the simulation-based approach. Ultimately, the proposed methodology provides authorities with a formal road approval procedure for AVs. In particular, the proposed methodology will support the Singaporean Land Transport Authority for road approval of AVs.@en

Automated vehicles (AVs) are expected to increase traffic safety and traffic efficiency, among others by enabling flexible mobility-on-demand systems. This is particularly important in Singapore, being one of the world's most densely populated countries, which is why the Singaporean authorities are currently actively facilitating the deployment of AVs. As a consequence, however, the need arises for a formal AV road approval procedure. To this end, a safety assessment framework is proposed, which combines aspects of the standardized functional safety design methodology with a traffic scenario-based approach. The latter involves using driving data to extract AV-relevant traffic scenarios. The underlying approach is based on decomposition of scenarios into elementary events, subsequent scenario parametrization, and sampling of the estimated probability density functions of the scenario parameters to create test scenarios. The resulting test scenarios are subsequently employed for virtual testing in a simulation environment and physical testing on a proving ground and in real life. As a result, the proposed assessment pipeline thus provides statistically relevant and quantitative measures for the AV performance in a relatively short time frame due to the simulation-based approach. Ultimately, the proposed methodology provides authorities with a formal road approval procedure for AVs. In particular, the proposed methodology will support the Singaporean Land Transport Authority for road approval of AVs.@en