DETERMINATION AND RANKING OF TRAJECTORY ACCURACY FACTORS

Sergio Torres, Ph.D., Lockheed Martin Information Systems and Global Services Civil, Rockville, MD 20850

Abstract
Trajectory accuracy improvements provide the opportunity to reduce fuel consumption and emissions by increasing the predictability of the National Airspace System (NAS). In addition to the environmental benefits, the ability to improve trajectory prediction accuracy enables trajectory based operations (TBO). Because of the foundational reliance on accurate gate-to-gate four dimensional trajectory (4DT) predictions in TBO, trajectory predictors (TP) will have to meet stringent accuracy performance requirements. There have been significant advances in understanding the accuracy performance and limitations of different TP algorithms. However, implementation of TBO requires identification of the specific aspects of trajectory prediction that will need improvement in order to deliver the necessary accuracy. The challenge is not to select a single trajectory model approach but rather to define which modeling approaches can deliver better performance under different circumstances and to understand the limitations of each. A clear understanding of accuracy factors and the way they affect different modeling approaches allows development of hybrid algorithms that combine the strengths of different approaches. It is known, for instance, that kinetic models are affected by the uncertainty in aircraft mass, while parametric models are affected by the variance in the velocities encoded in lookup tables, but what is the relative contribution of these errors to the overall accuracy of predictions? The paper presents an accuracy factors analysis methodology to rank the sources of error according to their respective impact. This analysis facilitates the identification of the modeling issues that have the largest impact in prediction accuracy and the systematic evaluation of the potential improvements that could be expected from the use of aircraft intent information that will be available when air-ground data link is deployed. The sources of errors in trajectory prediction have been amply studied; however, a clear understanding of the relative contribution of these errors to accuracy 978-1-4244-6618-4/10/$26.00 2010 IEEE

performance using realistic scenarios is necessary in order to be able to improve the models. The accuracy factors ranking methodology presented here relies on the computation of the Past Maximum Deviation (PMD) metric for each error measurement and using the PMD to attribute the error to one of four possible error source categories: lateral, vertical, velocity and heading. The PMD metric keeps track of the largest track-trajectory deviation prior to the measurement along each of the four categories. PMD distributions serve as a diagnostic that reveal the performance of a trajectory predictor (TP) along the four dimensions discussed and can be used as an effective tool to compare two TPs or two variants of the same TP. Results of accuracy factor analysis based on the En Route Automation Modernization ERAM parametric algorithm using a live recorded scenario are presented.

I. Introduction
Given the key role played by trajectories within the Trajectory Based Operations (TBO) concept there has been renewed interest in evaluating new or enhanced 4D trajectory (4DT) modeling approaches that could potentially deliver better performance. Since trajectories are the primary input to the decision support tools (DSTs) used by ATM operators, the accuracy of trajectory algorithms is of paramount importance, however it is recognized that due to the large disparity in trajectory predictor (TP) algorithms and accuracy evaluation approaches it is difficult to discern a TP solution path capable of supporting TBO needs. The FAA/Eurocontrol Action Plan 16 has put forward a convenient conceptual infrastructure to guide TP development and evaluation and to define common terminology that enables comparisons between different TP approaches [1]. A TP is an algorithm that computes the expected position (latitude, longitude and altitude) of the aircraft, as a function of time, based on flight plan information, controller clearances, weather data, and aircraft state. A trajectory, the main output of a TP, is a representation of the computed

1.C.1-1

prediction usually encapsulated in a data structure consisting of a list of waypoints with altitude, corresponding estimated times of arrival (ETA) and possibly other attributes. A TP is not a monolithic algorithm but rather a collection of algorithms each specialized to solve a particular modeling problem to build a trajectory. Thus, it is envisioned that as existing TPs evolve towards meeting the requirements to support TBO, there is need to evaluate the impact of individual factors to trajectory accuracy to be able to focus algorithmic improvement efforts where they have more relevance. Trajectory predictors in operational air and ground systems currently implement two different algorithmic approaches: kinetic and parametric (aka kinematic). In the kinetic approach, detailed modeling of aircraft operations performance (aerodynamics, engine thrust, fuel consumption, and aircraft envelope) is used to solve the point-mass equations of motion. In the parametric approach, speeds and vertical rates are obtained from lookup tables that have been prepared from empirical data and/or from pilot performance handbooks. The adoption and level of maturity of these two approaches has been driven by system requirements, CPU demands and by the availability of the input data required by the algorithms. Note that the difference between kinetic and parametric approaches is mainly relevant for vertical modeling. Cruise level segments can be modeled using the cleared cruise speed, the observed track speed and the component of the wind along the planned horizontal path (no need to integrate equations of motion). The TP implemented in the FAAs En Route Automation Modernization (ERAM) system Release 1 and 2 (R1, R2) uses the parametric trajectory modeling software ported over and enhanced from the User Request Evaluation Tool (URET) developed by the MITRE Corporation [2]. A parametric trajectory modeling was also used in the FAAs Enhanced Traffic Management System (ETMS), developed by the Volpe Transportation System Laboratory [3]. Kinetic models are the natural choice for the TPs used in Flight Management Systems (FMS) because they can take advantage of the specific aircraft performance and aircraft intent information that is available on board the aircraft. The FAAs Traffic Management Advisor (TMA) uses the kinetic model developed by NASA for the Center TRACON Automation System (CTAS) [4].

Another kinetic model commonly used Eurocontrols Base of Aircraft Data (BADA) [5].

is

Trajectory accuracy measurements are strongly dependent on the scenario used. It is not the same to measure prediction accuracy using a flight untouched by the controller than using a flight subject to repeated tactical intervention by controllers. For this reason, to achieve a meaningful accuracy performance assessment it is imperative to use realistic scenarios that reflect actual flight routings and operational details that affect changes of the flight throughout its lifetime. Exposing the trajectory models to realistic inputs is an effective way to diagnose performance problems and to compare algorithm alternatives; however, given the complexities of real life operations it is difficult to make a one to one comparison between two TPs in these circumstances. There are numerous reports in the technical literature that show trajectory accuracy measurements that cover a range of two orders of magnitude! Presenting accuracy results in terms of an aggregate metric is not enough to reveal the multidimensional nature of accuracy performance. The problem is compounded by the lack of uniformity in scenario data and measurement approaches. Reference [1], while highlighting the need for a comprehensive sensitivity analysis of trajectory prediction factors, has pointed out that most prior investigation of TP accuracy has focused on specific DST applications with results not easily cross-comparable to other applications. This paper presents a method to assess the relative contribution of different factors to the accuracy performance of a given TP. The accuracy factors analysis described here serves as a tool to identify the TP elements that have the lowest performance and by quantifying accuracy performance along multiple dimensions it allows meaningful and systematic comparison between alternative TP algorithms. The paper is organized as follows: an introductory section presents basic definitions and results from prior research, Section II deals with some important considerations that need to be taken into account in trajectory accuracy work, Section III describes the most common sources of errors in trajectory prediction, Section IV presents the accuracy factors analysis approach, and the following section illustrates the approach when applied to the ERAM-R1 TP.

1.C.1-2

II. Trajectory Accuracy

A straightforward approach to quantify trajectory accuracy is to compute the deviations of the predicted lateral and vertical position and time along the trajectory relative to a reference as flown 4D path [6]. Agreement in accuracy metrics, however, is not enough in order to make comparable assessments of different TPs. The literature is full of trajectory accuracy reports that quote a large range of performance numbers, but because they are not normalized these numbers are misleading at best. A study of URET and CTAS accuracy performance reports an average absolute horizontal error that depends on look-ahead time and ranges between 1.2 nmi to 10.2 nmi for URET and 0.3 nmi to 10.9 nmi for CTAS, both within a look-ahead range of 0 to 30 min [7]. The authors of the study point out that the results cannot be used to compare CTAS versus URET because the scenario data is different for each DSTs measurement. An evaluation of the performance of kinetic versus parametric models in the context of traffic flow management (TFM) applications indicates that kinetic models exhibit better accuracy performance [8]. The study reports ETA errors ranging from 1.2 to 3.4 min for ETMS (parametric model) and from 0.2 to 2.4 min for CTAS (kinetic) within a look-ahead range of 0 to 60 min. It is not clear, however, whether the poor performance of the parametric model reported in that study is due to issues with the speed look-up tables or if it is really a problem with the algorithm. Reference [9] reports a horizontal error (distance from radar reported position and time-coincident trajectory predicted position) for ERAM that grows near linearly from 0.8 nmi to 6 nmi for look-ahead times of 0 to 20 min. Using an extensive database of recorded field data in the USA, Europe and Australia researchers show that lateral intent errors (in the USA sample mostly) are the dominant source of trajectory errors [10]. They report a standard deviation of measured lateral track-trajectory deviations of 20.8 nmi. The same report quotes lateral deviation statistics based on Australian traffic for flights virtually untouched by ATC, with a dramatic difference of a factor 800 smaller than those found in USA traffic. Using a flight simulator to provide ground truth, ETA prediction errors of an FMS-TP (kinetic model) were measured resulting in errors as high as 37 sec [11] while an experiment with actual aircraft reports ETA errors of less than 5 sec based on

FMS trajectory predictions [12]. A sensitivity analysis of trajectory accuracy factors performed by simulations where each accuracy factor is randomly varied around a baseline realization indicates that (setting aside intent errors) speed and wind errors are the major source of trajectory accuracy errors [13]. The latter is an interesting study aimed at assessing the impact of separate accuracy factors but it relies on knowledge of prior error distributions and does not incorporate (in a direct way) the natural weight of factors that is present in real life scenarios. In contrast, the accuracy factors analysis presented here is based on realized traffic that captures the underlying relative weight of the many factors affecting accuracy. The broad overview of trajectory accuracy studies reported above clearly manifests the state of confusion and limited applicability of results due to the lack of standardization in conducting and presenting trajectory accuracy results. There are four normalization factors that have to be homogenized before any accuracy comparison can take place: (a) definition of the reference 4D path; (b) data sampling approach; (c) accuracy metric and statistic; and (d) scenario data.

A. Reference 4D Path
The reference 4D path for accuracy measurements (aka ground truth) is usually taken to be the smoothed position reports produced by the tracker based on surveillance sensor inputs, which is a good approximation to the as flown path. The raw sensor data, in the case of radar targets alone, is too noisy and would pollute the trajectory accuracy measurements; however as the adoption of the more accurate ADS-B surveillance gains wider use it will be possible to use the sensor data directly as a very good representation of the as flown path. The use of tracker outputs on the other hand could bias the error measurements because of the inherent lag to maneuvers in the underlying smoothing function in the tracker (an IMM Kalman filter in the case of ERAM and a - tracker in the Host Computer System in En Route centers). Tracker outputs also have residual noise that could leak into trajectory accuracy measurements. An alternative to generate the as flown flight is to use an algorithm, such as the Path Extraction (PE) algorithm described in

1.C.1-3

Reference [14], that extracts the noise-free as flown path from noisy sensor data

B. Data Sampling Approach

Even though the mathematical representation of a trajectory consists of discrete points along the predicted 4D path, a trajectory should be thought as a prediction continuum with time interpolation providing the predicted position at any arbitrary time. For accuracy measurements a selection of points along the trajectory has to be made. The data sampling approach determines the frequency and the times when measurements are computed. The sampling approach in Reference [15] defines a starting point along the trajectory, call it T0; makes a measurement of the track to trajectory deviations at T0 and at future times at T0 + 1, T0 + 2, T0 + 3, T0 + 4, etc., with {1, 2, 3, 4,} being fixed lookahead times (i.e. 1 min, 5 min, 10 min, 15 min, etc.); sampling continues by successive iterations with T0 incremented by a sampling step (i.e. 2 min) and measurements are done again at T0 and the T0+i times. This approach implements a controller viewpoint whereby at any given time (hence the 2 min sampling) the controller can look at the screen and at that time it is pertinent to ask what is the estimated prediction errors for different look-ahead times (i) relative to T0. Note however that this sampling approach blends measurements with different future times relative to the time of the prediction, or trajectory build time (Tb): suppose that at time Tb the system builds a trajectory; at this time (T0 = Tb) accuracy measurements are made selecting points along the trajectory at times T0 + 1 min, T0 + 5 min, T0 + 10 min, and so on; a second round of accuracy measurements for the same trajectory continue with T0 updated to T0 T0 + 2 min; in this second round the measurement done at T0 + 5 min contributes to the 5 min look-ahead bin even though the effective future time relative to Tb is 7 min (5 min + 2 min) not 5 min, and so on. For some applications, it is more relevant to measure trajectory accuracy in specific future time bins that do not blend measurements at different future times. This sampling approach [16] represents the system view and it is useful for prediction error analysis, where the relevant question is: given that I have a trajectory prediction produced by the system at time Tb (the time of the forecast), what are the expected errors (lateral, horizontal, vertical, and ETA) x minutes in

the future at time = Tb + x min? Analytical studies of error propagation work with specific future times, not with look-ahead times that blend different future times. For example, the longitudinal error (s) estimated at some future time Tf (relative to the time of the forecast Tb) due to velocity uncertainty (v) can be computed analytically, with s = v Tf: s = (s/v)v = Tfv (1) The parameter that enters the error propagation formula is Tf, namely a specific future time that cannot include a blend of different future times. The results presented in this paper use an implementation of system view sampling as follows: for each trajectory define a build time (Tb) equal to the time when the system made the prediction; for each trajectory select all of the available track position reports between time Tb and the time when that flight received a new clearance; for each of the selected track position reports compute the track trajectory deviations and define the future time as Tf = Tt Tb, where Tt is the time stamp associated with the track position report; this measurement contributes to the future time bin where Tf falls. Figure 1 illustrates the sampling strategies discussed above.

Figure 1. Sampling Approaches

C. Error Statistic
Although it may seem borderline nitpicking it is important to specify the statistic used when reporting accuracy measurements. Researchers use root-meansquared (RMS), standard deviation (), average absolute deviation (AD), and the standard deviation of the AD to quote their measurement results. It turns out that trajectory prediction errors, unlike for instance the random jitter noise that characterizes sensor reports, do not benefit from the central limit

1.C.1-4

theorem [17] because, although stochastic, flight deviations from its plan often times result from human actions that do not necessarily follow a tendency towards gaussianity and result in very large tails in the error distributions. The ETA error distribution obtained from the 5 hour scenario described in Section V-A exhibits highly nonGaussian (kurtosis = 5.7) but clear exponential distribution with much longer tails than a normal distribution (the 2/DOF is 34065/97 for a fit to a Gaussian and 975/96 for a fit to a double exponential). With these types of long tailed distributions there is a wide discrepancy between the various statistics. In the example mentioned above the standard deviation differs from the AD by a factor of 2. Whereas for a normal distribution the relation = sqrt(/2)*AD holds, for a long tailed distribution the AD statistic tends to sub-estimate the width of the distribution. With long tailed distributions the difference between AD and could be significant; however one frequently sees experiment results that freely quote errors using different statistics, a practice that could yield misleading comparisons. It is also common to see accuracy results reported using AD and its standard deviation. The latter statistic is particularly problematic and misleading. Whereas the standard deviation of signed errors has a straightforward probabilistic interpretation (1- around the mean contains 68.27% of the data, if normally distributed) the interpretation of the standard deviation of the AD is not clear. The ADs are distributed near exponentially, the distribution is one-sided, always occupying the positive side; the standard deviation of this distribution represents some width around the average, but the distribution is highly asymmetric, thus thinking that 1 standard deviation around the average is some sort of symmetric width is totally misleading. Furthermore, since for an exponential distribution the mean is equal to the standard deviation, this metric does not provide additional information. [10] makes the interesting suggestion to use the inter-quartile range (IQR) as an estimator of the width of the error distribution. The IQR measures the width occupied by 50% of the data around the center and as such is less sensitive to outliers. The authors show that the IQR of the distribution applied to lateral prediction errors can be used to extract the error component not affected by controller actions. The standard deviation (which is equivalent to the

RMS when there are no systematic errors) is the preferred statistic to report accuracy measurements. The advantages of the standard deviation are its sensitivity to large deviations, the fact that it is a well established and standard estimator of the width of a distribution, and its analyticity (see for instance Equation 1; the AD is not amenable to analytic manipulation because it is defined in terms of the absolute value operator). Since large deviations do occur in real life situations a statistic that is not sensitive to large deviations is not desirable. Measurements of trajectory prediction errors are not only use for TP performance but also can be used as inputs to computations and simulation exercises, therefore care must be taken to characterize the actual error distributions appropriately (that is including the effect of long tails). Trajectory accuracy measurements are tied to a specific look-ahead, however, often times researchers quote an error number without specifying the look-ahead time that applies for the quoted result. A useful and more informative trajectory error metric is the slope of the error versus look-ahead time curve.

D. Scenario Data
A measurement of TP accuracy is also a measurement of features of the scenario used. The same TP could yield significantly different accuracy results for two different input data sets. This is a trivial statement, but it helps explaining the wide range of accuracy results that have been reported and highlights the risk of attributing metrics results solely to TP performance. Ideally one would use the same inputs when in need to compare two TPs or one TP with two variants of the algorithms. Assessing the accuracy performance of a TP entails the use of a realistic scenario so as to exercise all the components of the trajectory modeling algorithms that have an impact on real life operations. In this environment however there are complications that produce results more difficult to interpret. Two related effects that arise in scenarios derived from live recorded data are selection bias and short look-ahead segments. Selection bias is explained in the next subsection, short look-ahead segments refers to the fact that in real life operations flights undergo frequent changes [10], [18] (tactical intervention by the controller, strategic planning, weather related reroutes, operator preferences, open ended clearances, etc.) Trajectory accuracy measurements

1.C.1-5

can be done for a trajectory for a prediction time window that goes from the time the trajectory is built (which is the present time) to any future time before a clearance changes the plan. As indicated, that time window could be severely shortened under heavy traffic load (could be few minutes on average) and limits the sample size for longer look-ahead times.

world input data. This task is made difficult by the statistical nature of accuracy measurements that require sufficiently large sample sizes. The next section describes how the accuracy factors analysis facilitates this task. Potential sources of error in trajectory prediction are listed below and reflect the relevant accuracy factors identified by a team of experts [13]. Route intent: situations when the controller issues an altitude clearance or change in route (hold, vector, direct-to) that is not entered into the ground automation. Open ended clearances can also be classified as route intent issues. Aircraft intent: inputs known to the FMS system but not available to the ground TP such as cost index, turn parameters, thrust de-rating, target level-off altitude, navigation mode engaged (LNAV, VNAV), whether the aircraft is flying to meet an RTA, knowledge of the top of descent, etc. ATC factors: special restrictions associated with inter-center coordination procedures that are not coded into automation; unknown runway assignment; metering and other traffic constraints usually result in modifications of the flight not known to the trajectory modeler. Wind velocity: inaccuracies in weather forecast, specially wind speed and direction are a major source of prediction errors. Aircraft performance modeling: for parametric algorithms aircraft performance tables with speeds and vertical rates are built from empirical data using a representative sample, however due to the differences in aircraft behaviors depending on airports, regions, weather, etc. the aircraft tables are subject to large statistical variances. Kinetic algorithms on the other hand are subject to errors in thrust and drag coefficients, as well as uncertainties in aircraft mass. Algorithm implementation: neglecting vertical wind gradient; approximating constant wind along a segment; fidelity of turn modeling; approximations in the equations of motion; algorithm parameters that determine segment size, trajectory build triggers and route re-join logic; earth model (WGS84, conformal sphere, flat plane, etc.)

E. Selection Bias
An immediate consequence of the effect described above where most of the trajectory accuracy measurements tend to come from short segments (hence short look-ahead times) is that measurements for long look-ahead times tend to be biased towards smaller deviations. This counterintuitive result is explained as follows: flights that have a long straight and level cruise phase are more likely to contribute to long look-ahead bins, but precisely those long straight and level segments are where one would expect small track-trajectory deviations (hence its name selection bias). Conversely, flights rapidly undergoing changes are more likely to show high track-trajectory deviations and are contributing preferentially to short lookahead times. Due to this effect (but strongly depending on the scenario) it is possible to obtain measurements that show trajectory errors decreasing for longer look-ahead times instead of increasing as one would expect theoretically (Equation 1). Note however that, because of the smaller number of long straight and level segments, the sample size for long look-ahead bins tends to be much smaller than that of short look-ahead bins and as a result the accuracy measurement in these bins is less robust. This fact highlights the need for researchers to quote the number of entries per bin (N) along with the RMS or on each look-ahead bin (N is an important piece of information needed to assess the statistical significance of the measurement because the error of the RMS or tend to decrease as ~1/sqrt(2N)).

III. Trajectory Accuracy Factors

To assess the performance of a TP it is necessary to understand the impact of all possible sources of errors. The impact of each accuracy factor can be studied individually by analytic or semi-analytic means [19], however due to the complex situations present in realistic scenarios it is necessary to evaluate the relative impact of each factor given real

1.C.1-6

Departure time: modeling a trajectory for a flight in pre-departure phase is subject to large uncertainties in departure time.

IV. Accuracy Assessment Approach

A. Error Source Categories
Before going into the details of prediction error analysis it is important to make a clear distinction between trajectory errors due to intent issues and errors due to the physics in the model. In generic terms intent issues could be understood as large track-trajectory deviations due to the flight not following the cleared route (lateral intent) or not complying with altitude restrictions (vertical intent). Lateral intent errors are typically observed when the controller verbally communicates a route amendment to the pilot (hold, path stretch, direct-to, etc.) and the information is not entered into ground automation. Lateral intent issues have been identified as a significant source of trajectory prediction errors [10,8]. The type of errors referred to as physics modeling errors is due to limitations or approximations in the prediction algorithms or the inputs that the algorithms use. Examples of modeling errors are the use of nominal aircraft mass instead of its actual value in kinetic models; statistical variance in the aircraft performance parameters used in parametric models; wind forecast errors; approximating turns by instantaneous changes in course, etc. Whereas physics model errors can in principle be reduced by increasing the fidelity of the algorithms and the accuracy of the input data, barring Artificial Intelligence (not a practical option in the ATM context) intent errors cannot be resolved algorithmically. For the reasons stated above it is important that in evaluating TP accuracy performance, researchers indicate to what extent the results are due to intent issues: the study in Reference [10] quotes a dramatic difference (up to a factor of 800!) in accuracy results when lateral intent errors are not present. In other words, a trajectory accuracy experiment polluted by intent issues is not measuring the accuracy of the TP core physics algorithms but something else instead. Although TFM applications are affected by intent errors (they use the available trajectory at any given time) to focus on evaluating and improving the core physics algorithms one must separate out intent issues. It is

foreseen that as ground automation systems evolve in support of TBO, controllers will have the tools to grant re-route clearances through the automation system, hence removing or substantially reducing intent issues. The aim of this paper is to study algorithm accuracy factors. For the purpose of analyzing the sources of prediction errors it is helpful to group the accuracy factors described above into four categories according to the dimension where the effect operates, namely: lateral, vertical, velocity and heading (other relevant categories could be added). A simplistic notional example (Figure 2) consisting of a small lateral deviation (red broken line) shows how an ETA error can be attributed to a lateral effect.

Figure 2. Lateral Deviation Example In the example the TP models the trajectory in the lateral dimension to go from A to B to C. Assume that the flight is level at the same altitude throughout those two segments, the velocity is constant and the change in course angle between AB and BC is very small (just a few degrees so that turn modeling details are not important). An ETA error is measured at point C and beyond. Analysis of the history of track position reports in the past of C is sufficient to find that the measured ETA error is due to the lateral deviation (d) that shortened the path and hence allowed the aircraft to cut few seconds of its planned trajectory. Define the past maximum deviation (PMD) metric as the largest track-trajectory deviation in the past of the measurement in any of the dimensions of interest (lateral, altitude, velocity, accumulated heading). In the example the lateral PMD is d, whereas the velocity, altitude and heading PMD are null. The heading PMD is somewhat different than in the other dimensions: it keeps track of the total change in course angle accumulated in the past of the measurements. The idea is to determine to what extent an ETA error is due to turn modeling effects, lateral deviation, velocity deviations or vertical modeling effects. The approach consists of attributing

1.C.1-7

observed errors to one of the four possible categories; at the end of the exercise one obtains a distribution of the frequency with which a given category is responsible for the observed errors. Such distributions provide a signature of the accuracy performance of a given TP and allows for meaningful comparison with another TP or a variant of the same TP while identifying and quantifying the specific components of the TP that show weak performance. Further insight into the root cause of a prediction error can be gained by correlation analysis: once an error has been attributed to, for instance, a lateral effect, then looking at correlations (or lack thereof) with large PMDs in the other dimensions provides information that in some cases points to a specific physical cause. To continue with the example in Figure 2, one could imagine a variant where the change in course angle between the segments AB and BC is very large (for instance 60 degrees), in which case there could be a large velocity PMD and definitely a large heading PMD indicating that in this case the ETA error is most likely due to a turn modeling error. Yet another variant of the original example could be a small lateral PMD (say d < 0.3 nmi) accompanied by a large velocity PMD, in which case the ETA error is most likely due to a velocity modeling effect. Additional detail on error sources and distributions is obtained by performing the analysis for data sub-sets selected according to the phase of flight (or other flight attributes of interest).

measurement, a record is kept with the PMD in lateral (cross-track), altitude and velocity. Also, the accumulated course angle change is stored. The PMDs observed prior to a detected ETA error exceeding the 10 sec threshold are used to determine the factor most likely responsible for the ETA error. In other words, here one has a measurement of a significant ETA error the question is, which of the four factors is this error associated with? To find an answer one can look at the PMDs and see if there is a large past deviation in one of the dimensions that can explain the ETA error. Potentially one could see deviations in all four dimensions or various combinations, in which case it is necessary to compare the PMDs and to choose the one with the largest relative strength. To make the PMD comparison in different dimensions the PMDs are normalized to a suitable reference deviation that has comparable probability on each one of the dimensions. With the data used in the experiment described in the next section it is observed that the aggregate track-trajectory deviations of the sub-set of trajectories that adhere to the plan at all times yield 95-percentiles of 3.5 nm (cross-track), 40 kt (velocity) and 800 ft (altitude). The PMD normalization factors are set equal to these 95percentiles. To attribute the ETA error to one of the four categories the normalized PMDs are computed and the category with the largest normalized PMD is chosen as the source of the error. With an ETA error (> 10 sec) following a lateral PMD of 7 nm, a speed PMD of 10 kt, and altitude PMD of zero (the flight is level throughout all this time) the normalized PMDs are: lateral PMD = 2 (7 nm/3.5 nm), speed PMD = 0.25 (10 kt/40 kt), altitude PMD = 0 (0/800 ft), therefore the cause of the ETA error is attributed to a lateral effect. This example also helps illustrate the presence of correlations. A large lateral PMD could be accompanied by a large speed PMD: the aircraft slowing down to take a sharp turn. By looking at the PMD combinations that exhibit large values in the event of a large ETA error it is possible to go deeper in the root cause analysis responsible for the error.

B. Accuracy Factors Analysis

To show how the accuracy factors analysis method works we will focus on ETA errors. ETA error is defined as the difference between the predicted crossing time at a point in the future along the trajectory and the actual time when the aircraft crosses that same point. The process starts by defining a suitable error threshold beyond which a measured error is considered a significant deviation. For example, for the analysis presented below this threshold was set to 10 sec for ETA error. Each time that a measurement of ETA error exceeds 10 sec the analysis proceeds to attribute the error to one of the four source categories (lateral, vertical, velocity or heading). For each track position report a tracktrajectory deviation measurement is performed (per system view sampling discussed in Section II-B). As the trajectory is traversed with each subsequent

V. Experiment
The accuracy factor analysis described in Section IV was applied to the ERAM-R1 TP. The subsections below describe the data used, the analysis and the results of the experiment.

1.C.1-8

A. Scenario and Data Processing

The primary data used for the experiment was provided by the FAA and consists of a 5 hour 22 minutes recording of flight and surveillance data collected in March of 2005 at the United States Washington Air Route Traffic Control Center (ARTCC). The scenario has 2259 flights and was fed to the ERAM system. Upon completion of the ERAM run, track and trajectory data was extracted from the system analysis recording (SAR) and made available for offline analysis. The EZR tool, used for various analysis tasks in ERAM [20], was enhanced with the accuracy factor analysis algorithms. The tool computes track-trajectory deviations, PMDs, and other metrics useful for analysis. Before the analysis, measurements for tracks with reported altitude below 15,000 ft and measurements performed while the aircraft is out of adherence were discarded. Also, measurements are flagged according to the phase of flight (climb, descent, cruise) using the vertical rate reported by the tracker. As stated in Section IV-A the focus of this study is modeling errors, therefore it is important to discard measurements affected by intent issues and predeparture uncertainties. Only trajectories for active flights were used. A trajectory that is built before the flight has departed is subject to large departure time uncertainties resulting in unbounded ETA and longitudinal prediction errors that cannot be solved algorithmically within the TP. To understand how lateral and vertical intent issues were handled it is important to look into the re-adherence logic in the ERAM TP. For each track report received by the TP the algorithm checks whether the position and altitude is within adapted thresholds of the Aircraft Trajectory (ERAM maintains two trajectories: the Aircraft Trajectory which is used by functions such as conflict probe that required lateral re-adherence, and the Flight Plan trajectory, used by functions such as boundary crossing determination that require a more stable prediction; the Aircraft Trajectory is used in this study). When the track deviation exceeds any of the adapted thresholds the trajectory (Aircraft Trajectory) is remodeled using heuristics to rejoin the path to the cleared route and altitude. Due to the readherence logic the TP could rebuild the trajectory multiple times during the lifetime of a flight,

however, at any given time the available trajectory at that time is used for accuracy measurements including future times (up to such time when a new clearance is received) during which out of adherence could be triggering new trajectory builds. To remove lateral and vertical intent issues affecting the experiment, a filter was applied to discard measurements with a lateral PMD greater than 7 nmi and a vertical PMD greater than 2500 ft. After all of the data selection filters were applied there remains a total of 65,227 measurements left for analysis.

B. Results
Figure 3 shows the distribution of the fraction of ETA errors categorized by cause for the cruise phase of the flight and in three look-ahead bins (< 5 min, between 5 10 min and > 10 min). Similar distributions for climb and descent phases are shown in Figure 4 and Figure 5.

Figure 3. Attribution of ETA Errors (Cruise)

Figure 4. Attribution of ETA Errors (Climb)

1.C.1-9

Figure 5. Attribution of ETA Errors (Descent) The error attribution distributions indicate that velocity effects are the dominant source of errors during the cruise portion of the flight and vertical modeling effects during climb and descent. The plots also show that turn modeling effects are negligible. Even though these results were expected based on prior extensive analysis of ERAM TP performance, the accuracy factors analysis provides a systematic way to quantify the sources of error and their relative impact, therefore allowing for a more robust method to compare the accuracy performance of different TPs or variants of the same TP. The aim of these initial measurements is not to obtain an absolute measurement of TP accuracy but rather to establish a baseline measurement and to guide the planned efforts towards improving trajectory algorithms. The dominance of velocity errors during the cruise phase is consistent with previous studies based on flight tests [21] and by sensitivity analysis based on simulations [13]. The fact that velocity effects, not lateral effects, are the main cause of ETA errors could be explained by the multiplicative effect in error propagation (see Equation 1). Whereas a lateral effect, for instance the few seconds gained when cutting a corner, translates into an ETA error equal to the time gained or loss during the maneuver, a velocity effect grows with time throughout the duration of the flight. Velocity modeling errors could be due to wind velocity inaccuracies or to speed deviations relative to the cleared speed. Pilots deviate from the cleared speed in order to follow a Cost

Index (CI) or RTA but the actual speed is usually not reported to the ground system (re-adherence logic in ERAM provides for adjusting the speed used in modeling according to the observed speed based on track data). A study of speed variance due to CI indicates that speed variations of 10% can be expected while in cruise and up to 30% during climb and descent [22]. Similarly, wind speed errors could be a major source of velocity modeling errors. Measurements of the accuracy of the 40-km Rapid Update Cycle 2 (RUC-2) wind data based on actual measurements on board a Boeing 737 report frequent wind speed errors in excess of 20 kt and occasionally reaching up to 58 kt [23], which translate to ETA errors in the range 1 3 min and longitudinal errors in the range 7 20 nmi for a look-ahead time of 20 min. Note that speed errors during cruise equally affect kinetic and parametric models (both are basically using the cleared cruise speed), a fact that stresses the potential benefits of using aircraft intent (i.e. speed schedule) downloaded from the aircraft in trajectory modeling. Table 1 shows a detailed breakdown of all possible combinations or accuracy factors for the cruise data. The table is constructed by counting, separately for each error source category, the number of instances when the corresponding PMD exceeds its detection threshold (explained in Section IV-A). An entry of 1 in a cell means that the corresponding category was detected as a source. The column labels stand for lateral (LAT), altitude (ALT), velocity (VEL) and heading (HDG). For example, the fifth raw with entries 0,1,01 means that 0.14% of the measurements had altitude and heading categories flagged as error sources. It is observed that there are correlations between factors but they are small; most of the errors are attributed to a single factor with velocity being the major contributor. The results clearly indicate the priority where modeling improvements need to be made: for the cruise phase, velocity accuracy is a source of errors with an impact a factor of seven higher than lateral effects or vertical modeling.

1.C.1-10

Table 1. Breakdown of ETA Error Cause Analysis LAT 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 ALT 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 VEL 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1 HDG 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 Contribution (%) 0.78 69.27 0.18 11.03 0.14 3.00 0.07 12.93 0.41 1.00 0.03 0.58 0.12 0.44 0.01

(ETMS) Functional Description, Version 5.0, Report DTS56-TMS-002, U. S. Department of Transportation, Cambridge MA. [4] Erzberger, H., W. Nedell, June 1989, Design of Automated System for Management of Arrival Traffic, NASA TM-102201, http://www.aviationsystemsdivision.arc.nasa.gov/pub lications/more/ctas/erzberger_06_89.pdf [5] Eurocontrol, BADA (Base of Aircraft Data), Eurocontrol Validation Infrastructure Center of Expertise, Bretigny-sur-Orge, France, http://www.eurocontrol.int/eec/public/standard_page/ proj_BADA.html [6] Ryan, H.F., M. M. Paglione, S.M. Green, 2004, Review of Trajectory Accuracy Methodology and Comparison of Error Measurement Metrics, AIAA2004-4787,AIAA GN&C Conference, Providence, RI [7] Paglione, M.M., H.F. Ryan, R.D. Oaks, J.S. Summerill, M.L. Cale, May 1999, Trajectory prediction accuracy report: User Request Evaluation Tool (URET) Center TRACON Automation System (CTAS), U.S. Department of Transportation, DOT/FAA/CT-TN99/10, http://acy.tc.faa.gov/cpat/docs/trjrpt_f2.pdf [8] Reisman, R., J. Murphy, P. Krolak, R. Wright, Sep 2007, Modeling tactical trajectory accuracy effects on traffic flow management operations, 7th AIAA Aviation Technology, Integration and Operations Conference (ATIO), Belfast, Northern Ireland. [9] Paglione, M., Santiago, C., Crowell, A., Oaks, R.D., Analysis of the Aircraft to Aircraft Conflict Properties in the National Airspace System, 18 21 August, 2008, AIAA Guidance, Navigation and Control Conference and Exhibit, Honolulu, Hawaii, USA. [10] Paglione, M., I. Bayraktutar, G. McDonald, J. Bronsvoort, Jun 29 July 2 2009, Lateral Intent Errors Impact on Aircraft Prediction, 8th USA/Europe Air Traffic Management Research and Development Seminar (ATM2009), Napa, California, USA. [11] Stell, L., October 25-29 2009, Flight management system prediction and execution of idlethurst descents, 28th Digital Avionics Systems Conference, Orlando, Florida, USA.

VI. Conclusions
Due to the complexity of situations that occur in realistic scenarios, comparing TP accuracy results based on an aggregate metric alone could be misleading. It is necessary to look into various dimensions in order to check how different accuracy factors affect the accuracy performance measurements. It was demonstrated that the accuracy factor analysis described in this paper is an effective methodology to perform meaningful and systematic comparisons of diverse trajectory algorithms. The approach was demonstrated with measurements performed with the ERAM-R1 TP and, by quantifying the sources of error, showed that it provides an effective instrument to guide the planned efforts to improve trajectory algorithms.

References
[1] Mondoloni, S., S. Swierstra, Oct 30, 2005, Commonality in disparate trajectory predictors for air traffic management applications, 24th Digital Avionics Systems Conference, Washington, D.C., USA. [2] Walker, M.G., 2002, A Benefits Assessment of the User Request Evaluation Tool (URET), Air Traffic Control Quarterly, Vol. 10, No. 4 [3] Volpe National Transportation Systems Center, June 1995, Enhanced Traffic Management System

1.C.1-11

[12] Klooster, J.K., A. Del Amo, P. Manzi, Jun 29 July 2 2009, Controlled Time-of-Arrival Flight Trials, 8th USA/Europe Air Traffic Management Research and Development Seminar (ATM2009), Napa, California, USA. [13] Mondoloni, S., I. Bayraktutar, June 2005, Impact of factors, conditions and metrics on trajectory prediction accuracy, 6th USA/Europe Air Traffic Management Research and Development Seminar, Baltimore, MD, USA. [14] Torres, S., Nov. 2-5, 2008, Aircraft Path Extraction from Noisy Target Data, ATCA 53rd Annual Conference and Exposition, Washington. [15] Lee Cale, M., S. Liu, R.D. Oaks, M. Paglione, H.F. Ryan, S. Summerill, Dec 3 7, 2001, "A Generic Sampling Technique for Measuring Aircraft Trajectory Prediction Accuracy", 4th USA/Europe Air Traffic Management Research and Development Seminar, Santa Fe, NM, USA, http://acy.tc.faa.gov/cpat/docs/tjs_atm2001.pdf [16] Rozen, N.E., 2001, Functional performance assessment of the User Request Evaluation Tool (URET), MITRE/CAASD, McLean, VA, USA, http://www.mitre.org/work/tech_papers/tech_papers_ 01/rozen_functional/rozen_functional.pdf [17] Weisstein, Eric W., "Central Limit Theorem", MathWorld A Wolfram Web Resource, http://mathworld.wolfram.com/CentralLimitTheorem .html [18] Torres, S., E.G. McKay, 24-25 June 2008, Intent Issues Noted in Current U.S. En Route Trajectory Modeling, Synchronising Business Trajectories in the Future ATM System, Seville, Spain.

[19] Mondoloni, S., July 2006, Aircraft trajectory prediction errors: including a summary of error sources and data, FAA/Eurocontrol Action Plan 16 Common Trajectory Prediction Capabilities, CSSI, Inc. [20] Torres, S., E.G. McKay, 25-29 October 2009, Assessing Tactical Alert Function Accuracy Performance, 28th Digital Avionics Systems Conference, Orlando, Florida, USA. [21] Williams, D.H., S.M. Green, 1998, Flight evaluation of the center/TRACON automation system trajectory prediction process, NASA TP-1988208439. [22] Rumler, W., Gnther, T., Fricke, H., Wei haar, U, June 1-4, 2010, Flight profile variations due to spreading practice of cost index based flight planning, 4th International Conference on Research in Air Transportation, Budapest. [23] Cole, R.E., S. Green, M. Jardin, Jun 13 16, 2000, Wind prediction accuracy for air traffic management decision support tools, 3th USA/Europe Air Traffic Management Research and Development Seminar (ATM2000), Napoli, Italy.

Acknowledgements
The author wishes to thank Mike Paglione of the FAAs Simulation and Analysis Team AJP661 for facilitating the scenario data used in this study.

29th Digital Avionics Systems Conference October 3-7, 2010

1.C.1-12