Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications

Abstract

Purpose

It would take billions of miles’ field road testing to demonstrate that the safety of automated vehicle is statistically significantly higher than the safety of human driving because that the accident of vehicle is rare event.

Design/methodology/approach

This paper proposes an accelerated testing method for automated vehicles safety evaluation based on improved importance sampling (IS) techniques. Taking the typical cut-in scenario as example, the proposed method extracts the critical variables of the scenario. Then, the distributions of critical variables are statistically fitted. The genetic algorithm is used to calculate the optimal IS parameters by solving an optimization problem. Considering the error of distribution fitting, the result is modified so that it can accurately reveal the safety benefits of automated vehicles in the real world.

Findings

Based on the naturalistic driving data in Shanghai, the proposed method is validated by simulation. The result shows that compared with the existing methods, the proposed method improves the test efficiency by 35 per cent, and the accuracy of accelerated test result is increased by 23 per cent.

Originality/value

This paper has three contributions. First, the genetic algorithm is used to calculate IS parameters, which improves the efficiency of test. Second, the result of test is modified by the error correction parameter, which improves the accuracy of test result. Third, typical high-risk cut-in scenarios in China are analyzed, and the proposed method is validated by simulation.

Citation

Xu, Y., Zou, Y. and Sun, J. (2018), "Accelerated testing for automated vehicles safety evaluation in cut-in scenarios based on importance sampling, genetic algorithm and simulation applications", Journal of Intelligent and Connected Vehicles, Vol. 1 No. 1, pp. 28-38. https://doi.org/10.1108/JICV-01-2018-0002

1. Introduction

According to traffic accident statistics reports, human factor is the main cause of traffic accidents. It is estimated that over 90 per cent accidents in motor crashes are because of drivers’ error (National Highway Traffic Safety Administration, 2015). Therefore, the automated vehicle (AV) is considered as the key technique to improve traffic safety. When an AV drives in real-world roads, it needs to deal with various and complicated traffic conditions. Therefore, AVs must be tested before they can be permitted to travel on the road, otherwise the AVs will be a threat to other traffic participants. However, according to the study by Kalra and Paddock (2016), it would take approximately 5 billion miles to demonstrate that the safety of AV is statistically significantly higher than the safety of human driving because of the very small probability of human driving accidents. And with a fleet of 100 AVs being field test-driven 24 h a day, 365 days a year at an average speed of 25 miles per hour, this would take about 225 years. Even by simulation, the test time required is still very long without acceleration because of the huge scenarios and long driving-testing distance. However, the existing studies pay more attention to the construction of scenario libraries (e.g. Pegasus [Federal Ministry for Economic Affairs and Energy (BMWi), 2016], enableS3 (The Enable-S3 Consortium, 2016), etc.) and test tools development (e.g. software-in-loop (Russo et al., 2007), hardware-in-loop (Gietelink et al., 2006) and vehicle in the loop (Berg et al., 2016)). The studies of accelerated testing method are neglected. Therefore, the accelerated method of automated loading of the re-sampled driving scenarios should be concerned so that the safety benefits of AVs can be effectively evaluated. This scenario acceleration method can be applied to simulation test and provide scenarios for hardware-in-loop test, driving simulator test and controlled test bed (e.g. Mcity).

This paper proposes an accelerated testing method for AVs (Level 2 to Level 5) safety evaluation based on improved importance sampling (IS) techniques. This paper uses the occurrence of high-risk event, such as conflict, crash and injury, to evaluate safety of AV. The high-risk events are identified by indirect indicators such as Time to Collision (TTC) and headway. Taking the typical cut-in scenarios as example, the proposed method extracts the critical variables of the scenario. Based on IS techniques, the scenarios with higher probability of occurrence of high-risk events are reconstructed to test the AVs. The result of test is modified by the error correction parameter which is calibrated by the empirical data, so that the safety benefits of AVs in the real world can be revealed. Finally, based on the naturalistic driving data in Shanghai, the proposed method is validated by simulation. Besides the IS accelerated testing method, the detailed contributions of this paper are as follows. First, the genetic algorithm (GA) is used to calculate the optimal IS parameters by solving an optimization problem, which improves the efficiency of accelerated testing. Second, based on the empirical data, the result of test is modified by the error correction parameter, which solves the problem in existing studies that the conflict rate in accelerated testing result is inconsistent with the conflict rate calculated by the empirical data. Third, based on the naturalistic driving data in Shanghai, typical high-risk cut-in scenarios in China are analyzed, and the proposed method is validated by simulation.

The rest of this paper is organized as follows. Section 2 presents the literature review of existing methods of AVs testing. Section 3 introduces the proposed accelerated testing method for AVs safety evaluation in detail. Section 4 verifies the proposed method by simulation based on naturalistic driving data in Shanghai. Section 5 presents the conclusion and future research needs.

2. Literature review

The test of AV requires a combination of test tools and test methods. Test tools provide facilities and test methods provide theoretical guidance for the test. This paper focuses on test methods because of the larger potential for accelerated testing and their theory significance. In existing studies, there are four main methods for AVs testing: Monte Carlo simulation, test matrix, worst-case scenario evaluation (WCSE) and accelerated evaluation. The above four methods are reviewed in this section.

Monte Carlo simulation is a stochastic method. The AVs are tested in the scenarios generated stochastically based on a certain distribution. Touran et al. (1999) evaluated the safety of autonomous intelligent cruise control model by Monte Carlo simulation. The values of some parameters in the model were obtained randomly based on a certain distribution. Althoff and Mergel (2011) evaluated the collision risk for autonomous vehicles when executing a planned maneuver by Monte Carlo simulation. The initial states in simulation were generated randomly according to a piecewise constant probability distribution. With the advent of naturalistic driving projects around the world in recent years, some studies began to build stochastic models based on naturalistic driving big data and carry out Monte Carlo simulations to evaluate AVs. Yang and Peng (2010); Lee (2004) and Woodrooffe et al. (2014) all evaluated the collision avoidance systems of vehicles by Monte Carlo simulation based on naturalistic driving big data. The advantage of Monte Carlo simulation is stochastic scenarios. However, in the simulation of rare events (such as collision), the number of tests required will be very large because of randomness.

Test matrix method is to predefine a “test matrix” consisting of a variety of typical scenarios based on past crash data and expert knowledge. Then the test matrix is used to test some properties (such as safety benefit) of AVs. The test matrix method is the basis of many test studies, such as autonomous emergency braking (AEB) protocol (Euro, N.C.A.P., 2013), CAMP (Deering, 2002), HASTE (Carsten et al., 2005), AIDE (Kussmann et al., 2004), TRACE (Karabatsou et al., 2007), APROSYS (Wohllebe et al., 2004) and ASSESS (Bühne et al., 2012). The advantage of the test matrix method is efficient, credible and repeatable. Applying test matrix method to evaluate low-level AVs is straightforward and it might continue to be the selected approach in the near future (Zhao et al., 2017a). However, test scenarios in the test matrix are predefined and fixed. The scenarios predefined based on crash data cannot test the AVs comprehensively. And the test cannot reveal the properties of AVs in real-world conditions, especially in complicated mixed traffic flows.

WCSE methodology evaluates the control system of a vehicle by generating worst scenario (Ma and Huei, 1999; Kou, 2010). WCSE can be expressed as an optimization problem that searches for the worst scenario. The scenario is quantitatively evaluated by a cost function, and the goal of WCSE is to search for the scenario with the largest cost. WCSE can identify the weakness in the control system of an AV. But WCSE does not correlate the worst scenario with real-world scenarios, and the probability of occurrence of worst-case scenario in real world cannot be identified.

Accelerated evaluation is a data (such as naturalistic driving data)-based testing method. High-risk scenarios are more efficient than normal scenarios in AV testing. However, high-risk scenarios are rare events in naturalistic driving data. Accelerated evaluation method modifies the distribution of real-world data so that the high-risk scenarios have a higher probability of occurrence. Zhao et al. (2015) extracted the car-following scenarios in naturalistic driving data and used models to fit the critical variables in the scenarios. The most frequent scenarios are deleted to increase the overall exposure rate for critical scenarios. However, this method cannot evaluate the safety benefits of AVs in real world. Considering this problem, Zhao et al. (2017a, 2017b) applied the IS technique to AVs testing and studied the cut-in and car-following scenarios. The distribution of critical variables was modified to increase the probability of occurrence of high-risk scenarios and reduce the number of required tests. Then, the IS technique was used to modify the test result so that the result can reveal the safety benefits of AVs in real world. The most important advantage of accelerated evaluation is that it can describe the real-world benefits (such as crash rate) of AVs. And the method also improves the efficiency of testing. But in existing studies, the error in critical variable distribution fitting is ignored, which leads to the final result deviating from actual. This phenomenon will be discussed in detail in Section 4.4.3.

In summary, the test matrix method and WCSE method can hardly reveal the probability of AVs being exposed to risk in real-world scenarios, and the Monte Carlo simulation is inefficient. The accelerated evaluation method can reveal the safety benefits of AVs in real world and has the advantage of high testing efficiency. However, existing studies overlooked the fitting errors of critical variables, resulting in the deviation of test result from the field operation. To solve this problem, this paper proposes an accelerated testing method for AVs safety evaluation based on improved IS techniques. Considering the fitting error of critical variables, the GA is used to calculate the optimal IS parameters to obtain a better acceleration efficiency. And the error correction parameter is used to correct the test result to make the result consistent with the result calculated by empirical data.

3. Methodology

In real-world data, high-risk events of vehicles belong to rare events, and a reliable evaluation of the probability of this event requires a large number of tests. The proposed accelerated testing method based on improved IS techniques can reduce the number of tests and obtain a reliable test result. The proposed method can be divided into four steps (Figure 1). First, based on the real-world data, the scenarios to be analyzed are extracted, and the critical variables of these scenarios are defined and obtained. Second, based on the extracted scenarios and variables, the optimal IS parameters are calculated to generate accelerated scenarios, and the IS technique and simulation are used to calculate the safety benefits of AVs. Third, the error correction parameter is calibrated by the real-world data and safety benefits. Finally, the test result is corrected by the error correction parameter, and the final safety benefits of AVs in real world are obtained. We will discuss this framework in the following sections in detail.

3.1 Probability of high-risk-event

Consider a sample space Ω with a probability measure P. Let P(ε) denote the probability of a high-risk event. Let x be a random sample in Ω. Let I_ε(x) be the indicator function of high-risk event ε. I_ε(x) is defined as:

(1) Iε(x)={1,if x⊂ε0,otherwise

Let γ denote the probability P(ε). γ can be estimated via simulation by generating independent samples (x₁, x₂,…, x_n). Let γ^_n denote an estimator of γ. γ^_n can be calculated by:

(2) γ^n=1n∑i=1nIε(xi)

According to the law of large numbers, γ^_n→γ as n → ∞, that is, when n is large enough, γ^n converges to γ.

To ensure the reliability of γ^n, the central limit theorem proves useful in developing a confidence interval (CI) for estimate and is used to determine the necessary n for accurate estimation. For a sufficiently large n, the variance of γ^n is:

(3) σ2(γ^n)=Var(1n∑i=1nIε(xi))=1n2∑i=1nVar(Iε(xi))=γ(1−γ)n

With a confidence level at 100(1 – α) per cent, the CI of γ^n is:

(4) [γ^n−zα/2σ(γ^n),γ^n+zα/2σ(γ^n)]

where, z_α/2 is defined as:

(5) zα/2=Φ−1(1−α/2)

where, Φ⁻¹ is the inverse cumulative distribution function of normal distribution N(0,1).

The half-width of CI is:

(6) lα/2=zα/2σ(γ^n)

As the value of γ^n is small, the relative half-width l_r is used to indicate the accuracy of the estimation. l_r is defined as:

(7) lr=lα/2/γ

To ensure that l_r is smaller than a constant b we need:

(8) lr=lα/2γ=zα/2σ(γ^n)γ=zα/21−γγn≤b

That is:

(9) n≥1−γγ⋅zα/22b2

The probability of high-risk events in real-world driving is very small. Therefore, the sample size n needs to be huge to ensure the reliability of the estimation. This means that a huge number of tests are required to evaluate the probability of high-risk events in AV driving if using the Crude Monte Carlo method.

3.2 Importance sampling

IS is one of the classical variance reduction techniques for increasing the efficiency of Monte Carlo algorithms (Glynn and Iglehart, 1989). The IS has been successfully used to evaluate reliability (Heidelberger, 1995) and critical events in finance (Glasserman and Li, 2005), insurance (Asmussen and Albrecher, 2010) and telecommunication networks (Chang et al., 1994). General overviews about IS can be found in Bucklew (2013) and Blanchet and Lam (2012). The basic idea of IS used in evaluation of AVs is to replace the original distribution density function f(x) by a new one f*(x) to generate event x, which leads to a higher probability of occurrence of high-risk events. And then the risk calculation function is modified to obtain the safety benefits of AVs.

Let x* be the event generated by f*(x), the estimator γ^n can be expressed as:

(10) γ^n=1n∑i=1nIε(xi*)L(xi*)

where, L(xi*) is the likelihood ratio (Radon–Nikodym derivative (Royden and Fitzpatrick, 1988)) defined as:

(11) L(xi*)=f(xi*)f*(xi*)

The relative half-width of CI constructed by IS can be expressed as:

(12) lr*=lα/2γ=zα/2σ(γ^n)γ   =zα/2Ef*(γ^n2)−Ef*2(γ^n)γn   =zα/2nEf*(Iε2(x*)L2(x*))γ2−1

To ensure lr*≤b, the necessary test number n is:

(13) n≥(Ef*(Iε2(x*)L2(x*))γ2−1) ⋅ zα/22b2

Therefore, to obtain a reliable result through a small number of tests, the density function f*(x) needs to be properly chosen to make Ef*(Iε2(x*)L2(x*)) close to γ².

3.3 High-risk scenarios: cut-in events

The proposed method can be used in a variety of scenarios such as lane changing scenarios, car-following scenarios and crossing scenarios. In the typical traffic environment, the cut-in scenario occurs more frequently and has greater risk on human driving. Therefore, this study focuses on the cut-in scenario which refers to the situation that other vehicles move into the lane where AV located from an adjacent lane in front of the AV.

The critical variables of cut-in scenario are identified as: the velocity of lane changing vehicle (LCV) v_l, the range R defined as the distance between the rear edge of the LCV and the front edge of AV, and TTC. The TTC is defined as:

(14) TTC=−RṘ

where, R is the range, Ṙ is the derivative of R.

During the driving process, the high-risk events tend to correspond to small R and TTC. Smaller R and TTC indicate that the event is rarer and less safe. Therefore, the reciprocal of R and TTC are used to put the rare events in the tail of the distribution. The Pareto distribution is used to fit R⁻¹ and the Exponential distribution is used to fit TTC⁻¹. The distribution of v_l is not fitted and the empirical distribution is directly used to generate events.

The density function of the distribution of R⁻¹ can be expressed as:

(15) fR−1(x|kR−1,σR−1,θR−1)=1σR−1(1+kR−1x−θR−1σR−1)−1−1kR−1

where, k_R⁻¹ is the shape parameter; σ_R⁻¹ is the scale parameter; θ_R⁻¹ is the threshold parameter.

The density function of the distribution of TTC⁻¹ can be expressed as:

(16) fTTC−1(x|λTTC−1)=1λTTC−1e−x/λTTC−1

where, λ_TTC⁻¹ is the rate parameter.

Therefore, every lane changing event can be expressed as:

(17) xi=(vl,R−1,TTC−1)

3.4 Accelerated Evaluation

According to IS technique, a new distribution fR−1* is used to replace f_R⁻¹, and a new distribution fTTC−1* is used to replace f_TTC⁻¹. As R⁻¹ obeys Pareto distribution, we need to construct a new exponential distribution f˜_R⁻¹ before replacing f_R⁻¹ by fR−1* to reduce the computation complexity in later steps (Zhao et al., 2017b). f˜_R⁻¹ has the smallest least square error to f_R⁻¹. f˜_R⁻ is defined as:

(18) f˜R−1(x)=1λR−1exp⁡(−xλR−1)

Apply exponential change of measure to f˜_R⁻ and f_TTC⁻¹, we get fR−1* and fTTC−1*:

(19) fR−1*(x|ϑR−1)=(1λR−1−ϑR−1)exp⁡(−xλR−1−ϑR−1)

(20) fTTC−1*(x|ϑTTC−1)=(1λTTC−1−ϑTTC−1)exp⁡(−xλTTC−1−ϑTTC−1)

Therefore, the likelihood is:

(21) L(R−1=x,TTC−1=y)=fR−1(x)fTTC−1(y)fR−1*(x)fTTC−1*(y)

The probability of high-risk events is:

(22) P(ε)=Ef(Iε(x))=Ef*(Iε(x)L(x))

Therefore, proper fR−1* and fTTC−1* need to be constructed to calculate P(ε) efficiently. In other words, optimal IS parameters ϑ_R⁻¹ and ϑ_TTC⁻¹ need to be obtained to get the reliable test result through the minimum number of tests.

3.5 Searching for optimal importance sampling parameters with genetic algorithm

The efficiency of accelerated evaluation is closely related to the IS parameters. To obtain the optimal IS parameters, consider the following optimization problem:

(23) min  n=(Ef*(Iε2(x*)L2(x*))γ2−1)⋅zα/22b2

(24) s.t γ=Ef(Iε(x))

(25) x=[vl,R−1,TTC−1]

(26) fR−1(x)=1σR−1(1+kR−1x−θR−1σR−1)−1−1kR−1

(27) fTTC−1(x)=1λTTC−1e−x/λTTC−1

(28) x*=[vl,R−1*,TTC−1*]

(29) fR−1*(x)=(1λR−1−ϑR−1)exp⁡(−xλR−1−ϑR−1)

(30) fTTC−1*(x)=(1λTTC−1−ϑTTC−1)exp⁡(−xλTTC−1−ϑTTC−1)

(31) L(R−1=μ,TTC−1=φ)=fR−1(μ)fTTC−1(φ)fR−1*(μ)fTTC−1*(φ)

In this optimization problem, the objective function is number of tests n, and the decision variables are IS parameters ϑ_R⁻¹ and ^ϑTTC⁻¹. In the constrains, I_ε is the indicator function of high-risk event ε; L(x*) is the likelihood; γ is the probability of high-risk events; z_α/2 is given by equation (5); b is the threshold of relative half-width; x is a matrix of a series of samples x₁, x₂,…, x_i, and x_i is given by equation (17); f_R−1(x) is the density function of distribution of R⁻¹; fTTC⁻¹(x) is the density function of distribution of TTC⁻¹; x* is a matrix of a series of samples x1*,x2*,…,xi* and xi*=(vl,R−1*,TTC−1*); fR−1*(x) is the density function of distribution of R^−1*; fTTC−1*(x) is the density function of distribution of TTC^−1s*.

GA is one of the effective methods to solve the optimization problem (Goldberg, 1989; Michalewicz, 2013). GA is an adaptive global search algorithm, which has the advantages of short calculation time and high robustness. GA is widely used in multi-objective optimization, industrial engineering, management science, artificial intelligence and so on (Gen and Cheng, 2000). The optimization problem can be solved by GA to calculate the optimal IS parameter. With the assistant of GA tool in MATLAB, the optimization problem can be easily solved.

3.6 Error correction

The error in distribution fitting of R⁻¹ and TTC⁻¹ is inevitable during the accelerated testing. Through the comparison with the empirical data, it is obvious that these fitting errors will lead to the final result γ^ deviating from the actual probability of risk γ. However, the existing studies only compare the accelerated testing result with Monte Carlo simulation result. The data used in these two methods strictly follow the well-fitted distributions, and the error is overlooked. Fortunately, this study found this issue by comparing the result with the actual data and corrected the error. The error is corrected by error correction parameter τ, let:

(32) γ=γ^⋅τ

where, γ is the probability of risk calculated by empirical data; γ^ is the estimator obtained by accelerated evaluation. The fitting error can be reflected by the difference between mean of the fitted distributions and empirical distributions. Therefore, τ is defined as:

(33) τ=k1(μR1−μR0)+k2(μTTC1−μTTC0)

where, μ_R1 is the mean of fitted R⁻¹ distribution; μ_R0 is the mean of empirical R⁻¹ distribution; μ_TTC1 is the mean of fitted TTC⁻¹ distribution; μ_TTC0 is the mean of empirical TTC⁻¹ distribution; k₁ and k₂ parameters need to be calibrated.

The significance of error correction parameter is to modify the test result to make it closer to empirical result rather than the result of Monte Carlo simulation. The revised result can better reveal the safety benefits of AVs in real world, and it is more persuasive.

3.7 Algorithm of the proposed method

The execution algorithm of the proposed method is as follows:

Execution Algorithm of the Proposed MethodStep 1 Critical Variables Extraction      Step 1.1: Input the real-world data Ω. Extract scenario Ψ tobe analyzed. Take cut-in scenario as example.      Step 1.2: Input the scenarios Ψ. Define the critical variablesξ₁, ξ₂,…,ξ_i. Extract ξ₁,ξ₂,…,ξ_i. from Ψ. The criticalvariables of cut-in scenario are v_l, R and TTC.Step 2 Accelerated Distribution Generating      Step 2.1: Input the critical variables R and TTC. Fit thedistribution of R⁻¹ based on equation (15). Fit thedistribution of TTC⁻¹ based on equation (16).      Step 2.2: Input the distribution of R⁻¹. Generate f^^R−1 basedon equation (18).      Step 2.3: Input the distribution f^^R−1 and f*TTC⁻¹. Generate theoptimal IS parameters ϑ_R−1 and TTC⁻¹ based on equations(23)-(31).      Step 2.4: Input the parameters ϑ_R−1 and ϑ_TTC⁻¹. Generateaccelerated distribution fR−1* and fTTC−1* based on equations(19)-(20).     Step 3 Error Correction Parameter Calibration      Input the real-world data Ω₁. Where Ω₁ is a subset of Ω.Input the accelerated distribution fR−1* and fTTC−1*. Generateparameter k₁ and k₂ based on equations (32) and (33).     Step 4 Test of AVs      Step 4.1: Input the accelerated distribution fR−1* and fTTC−1*,and the empirical distribution of v_l. Generate the acceleratedscenarios xi based on equation (17).      Step 4.2: Input accelerated scenarios xi, and the parameter k₁and k₂. Calculate the test result based on equations (21) (22)(32) (33).

4. Simulation

4.1 Data

The data used in this research are from Shanghai Naturalistic Driving Research project. The project is the first naturalistic driving data collection project using real vehicles and high-precision equipment in China. The project aims to collect real-world traffic data and study the behavioral characteristics of Chinese drivers. It recorded over 500,000 km naturalistic driving data from December 2012 to December 2015. The driving trajectories of vehicles basically covered the main roads in Shanghai (Figure 2). The vehicles are equipped with Mobileye vehicle active safety system and SHARP2 NextGen data acquisition system. The data collected include information such as vehicle position, velocity and distance to surrounding vehicles (Figure 3).

The simulation takes the cut-in scenario as an example to validate the proposed method. Therefore, the cut-in scenarios are extracted from the naturalistic driving data. The cut-in scenario refers to the situation that other vehicles move into the lane where AV located from an adjacent lane in front of the AV. The moment when the LCV crosses the lane line is defined as lane changing moment. Then, the variables of cut-in scenario of lane changing moment are extracted. The variables include v_l, v and R (Figure 4). Where v_l is the velocity of LCV, v is the velocity of evaluated vehicle and R is the range defined as the distance between the rear edge of the LCV and the front edge of evaluated vehicle. In Figure 4, the evaluated vehicle is AV, and the LCV can be AV or manual-driving vehicle.

For comparison analysis, the same criteria as Zhao et al. (2017b) were applied: v ∈ (2m/s, 40m/s); v_c ∈ (2m/s, 40m/s); and R ∈ (0.1m, 75m). Finally, 32,104 cut-in scenarios were extracted from the data.

4.2 Evaluated models

For comparison analysis, the same AV model and parameters as that of Zhao et al. (2014b) were used in the simulation. The AV is equipped with adaptive cruise control (ACC) and AEB system. When TTC ≥ TTC_AEB, the AV is controlled by ACC; and when TTC < TTC_AEB, the AV is controlled by AEB. TTC_AEB is a function of vehicle velocity (Figure 5).

The ACC is approximated by a discrete proportional-integral controller to achieve a desired time headway THWdACC (Ulsoy et al., 2012). The input of the controller is time headway error tHWErr, and the output is the command acceleration of the next time step which can be expressed as:

(34) ad(k+1)=ad(k)+KpACC(tHWErr(k)−tHWErr(k−1))   +KiACC(tHWErr(k)−tHWErr(k−1))Ts/2

(35) tHWErr=tHW−THWdACC

(36) tHW=R/v

(37) |ad|≤aACCmax⁡

where, a_d is the command acceleration; tHWErr is the time headway error; KpACC and KiACC are constant gains; T_s is the sampling time; THWdACC is the desired time headway; t_HW is the time headway; R is the range; v is the velocity of AV; and aACCmax⁡ is the maximum acceleration.

Once triggered, AEB aims to achieve an acceleration a_AEB. Let the triggered moment be 0, the AEB model can be expressed as:

(38) a(t)={0,       if t≤TarAEB⋅(t−Ta),if t>Ta and a(t)≤aAEBaAEB,        else         

where, a(t) is the acceleration of vehicle at t moment; r_AEB is the derivative of acceleration; T_a is the action time; and a_AEB is the acceleration of emergency braking.

A time τ_AV is needed to model the transfer function from the commanded acceleration to the actual acceleration. For simplicity, let τ_AV be a constant.

The parameters of AV model in simulation are listed in Table I.

4.3 Analyzed event

There are three kinds of high-risk events: conflict, crash and injury. Crash and injury events are included in conflict events. Besides, the simulation analysis methods and conclusions of three kinds of events are both similar. Therefore, for simplicity, simulation analysis in this paper focuses on the conflict events.

A conflict event happens when an AV enters the proximity zone of the LCV during time t to t + T. The proximity zone is defined as the area in the adjacent lane from 1.2 m in front of the bumper of LCV to 9 m behind the rear bumper of LCV (Lee et al., 2004). The definition of proximity zone is shown in Figure 6.

4.4 Simulation result

Simulation can be used to evaluate the real-world safety benefits of AVs if the models are strictly calibrated by real-world data. In this paper, the distributions of critical variables are fitted based on empirical data. The IS parameter is calculated based on a subset of empirical data. The error correction parameter is calibrated by a subset of empirical data. The AEB system was extracted from a 2011 Volvo V60, based on a test conducted by ADAC (Allgemeiner Deutscher Automobil-Club e.V.) (Gorman, 2013). Therefore, the results of simulation can reveal the real-world safety benefits of tested AV.

4.4.1 Distribution of critical variables

As is mentioned in Section 3.3, the critical variables of cut-in scenario are the velocity of LCV v_l, the range R and time to collision TTC. The distribution of v_l is not fitted and the empirical distribution is shown in Figure 7.

The Pareto distribution is used to fit the distribution of R⁻¹. The fitting result is shown in Figure 8 and Table II.

The exponential distribution is used to fit the distribution of TTC⁻¹. Fitting result is shown in Figure 9. The estimate of parameter is λTTC⁻¹=0.0647(Std. Err.=0.0004).

4.4.2 Result of accelerated evaluation

The optimal IS parameters ϑ_R−1 and ^ϑTTC⁻¹ are calculated by solving the optimization problem in equations (22)-(30). The GA tool of MATLAB is used to solve this problem. The value of fitness function in solution searching process is shown in Figure 10. The result is ϑ_R−1=−0.3419, and ^ϑTTC⁻¹=−0.0120. Traversing calculation may determine the approximate region of the optimal solution. Therefore, to verify the validity of the GA optimal solution, different combinations of ϑ_R−1 and ϑTTC⁻¹ values are traversed [Figure 11]. The result shows that the GA optimal solution is reliable.

Then, the conflict rate is calculated using the optimal ϑϑ_R−1 and ϑTTC⁻¹. The convergence is reached when the relative half-width l_r < 0.2 with 80 per cent confidence. And the test number is calculated when the conflict rate reaches convergence.

As the simulation is stochastic, the required test number may fluctuate within a certain range. To ensure the credibility of the result, certain times of simulations are required. Therefore, the simulation was done ten times using the optimal ϑϑ_R−1 and ϑTTC⁻¹. The average test number of ten simulations was compared with the result of non-accelerated simulation based on empirical data. The result shows that the average required test number of proposed method is 286. And the required test number of non-accelerated simulation is 10,391. The result of one of the ten simulations is shown in Figures 12 and 13.

Based on the same data, the method proposed in Zhao et al. (2017b) was also used in simulation. The simulation was done ten times too, and the average test number was compared with the result obtained by the proposed method. The result shows that the average test number of method in Zhao et al. (2017b) is 435. Therefore, the proposed method is more efficient. Compared with the method in Zhao et al. (2017b), the proposed method improves the test efficiency by 35 per cent.

4.5 Error correction

The error in distribution fitting of R⁻¹ and TTC⁻¹ is inevitable. However, the data used in Monte Carlo simulation and existing accelerated evaluation method strictly follow the well-fitted distributions, and the error is overlooked, which leads to the final conflict rate deviating from the actual. Therefore, the error correction is necessary. As shown in Figure 14, the conflict rate calculated by the proposed method converges to the similar value of actual conflict rate. However, the conflict rate calculated by Monte Carlo method and the method in Zhao et al. (2017b) converges to another value. By error correction, the accuracy of accelerated test result was increased by 23 per cent.

As mentioned in Section 3, the main advantage of accelerated evaluation is that the method can reveal the safety benefits of real world. However, if the error is not corrected, the accelerated testing cannot accurately reveal the conflict rate in real world and cannot provide the correct reference to the safety of AVs. Therefore, error correction is of great importance to accelerated testing, which is an indispensable step.

5. Conclusion

The rapid development of automated driving technology constantly puts forward new requirements for the testing technique of AVs. This paper proposes an accelerated testing method for AVs safety evaluation based on improved IS techniques. Based on the real-world data, the critical variables of research scenarios are extracted, and the distributions of these variables are fitted. Then the optimal IS parameters are calculated by solving an optimization problem with GAs to generate accelerated scenarios, and the AVs are tested in these scenarios to obtain the safety benefits. Finally, the testing result is modified by the error correction parameter which is calibrated by the real-world data and safety benefits, and the final result of accelerated testing is obtained. Focusing on the cut-in scenario, the proposed method is validated by simulation based on the Shanghai naturalistic driving data. The result shows that compared with the existing IS technique, the proposed method improves the test efficiency by 35 per cent and increases the accuracy of accelerated test result by 23 per cent.

Compared with the existing IS technique, the proposed improved importance technique method has the following contributions: First, GA is used to calculate the optimal IS parameters by solving an optimization problem, which improves efficiency of test. Second, based on the empirical data, the result of test is modified by the error correction parameter, which solves the problem in existing studies that the conflict rate in accelerated testing result is inconsistent with the conflict rate calculated by the empirical data. Third, based on the naturalistic driving data in Shanghai, typical high-risk cut-in scenarios in China are analyzed, and the proposed method is validated by simulation.

As a new method for AVs testing, the accelerated testing method has broad application prospects. Further research needs to focus on the following two aspects. One is the definition and testing method of the more complex high-risk behaviors, especially the multi-object interactive behavior in mixed traffic flow. The other aspect is the further integration of testing tools and accelerated testing methods.

Figures

Figure 1

Framework of proposed accelerated testing method

Figure 2

The trajectories of naturalistic driving vehicles

Figure 3

Data collected by data acquisition system

Figure 4

Extracted variables of cut-in scenario

Figure 5

TTC_AEB as a function of vehicle velocity

Figure 6

Definition of proximity zone

Figure 7

The empirical distribution of v_l

Figure 8

Fitting result of R⁻¹ using Pareto distribution

Figure 9

Fitting result of TTC⁻¹ using exponential distribution

Figure 10

Fitness value in solution searching process

Figure 11

Result of traversing calculation

Figure 12

Estimation of the conflict rate

Figure 13

Relative half-width of conflict rate

Figure 14

Conflict rate calculated by different methods

Table I

Parameters in evaluated models

Parameter	Value	Unit	Equation
KpACC	−38.6	–	(34)
KiACC	−1.35	–	(34)
THWdACC	2	s	(35)
Ts	0.1	s	(34)
aACCmax⁡	5	m/s2	(37)
aAEB	−10	m/s2	(38)
rAEB	−16	m/s2	(38)
Ta	0.5	s	(38)
τAV	0.0796	s	–

Table II

Parameters in distribution of R⁻¹

Parameter	kR−1	σR−1	θR−1
Estimate	0.1987	0.0180	0.0133
Std. Err.	0.0066	0.0002	0