05655521

8/3/2019 05655521

http://slidepdf.com/reader/full/05655521 1/12

978-1-4244-6618-4/10/$26.00 ©2010 IEEE1.C.1-1

DETERMINATION AND RANKING OF TRAJECTORY ACCURACY

FACTORS

Sergio Torres, Ph.D., Lockheed Martin Information Systems and Global Services – Civil,

Rockville, MD 20850

AbstractTrajectory accuracy improvements provide the

opportunity to reduce fuel consumption andemissions by increasing the predictability of the National Airspace System (NAS). In addition to theenvironmental benefits, the ability to improvetrajectory prediction accuracy enables trajectory based operations (TBO). Because of the foundationalreliance on accurate gate-to-gate four dimensionaltrajectory (4DT) predictions in TBO, trajectory predictors (TP) will have to meet stringent accuracy performance requirements. There have beensignificant advances in understanding the accuracy performance and limitations of different TPalgorithms. However, implementation of TBOrequires identification of the specific aspects of trajectory prediction that will need improvement inorder to deliver the necessary accuracy. Thechallenge is not to select a single trajectory modelapproach but rather to define which modelingapproaches can deliver better performance under different circumstances and to understand thelimitations of each. A clear understanding of accuracy factors and the way they affect different

modeling approaches allows development of hybridalgorithms that combine the strengths of differentapproaches. It is known, for instance, that kineticmodels are affected by the uncertainty in aircraftmass, while parametric models are affected by thevariance in the velocities encoded in lookup tables, but what is the relative contribution of these errors tothe overall accuracy of predictions? The paper presents an accuracy factors analysis methodology torank the sources of error according to their respectiveimpact. This analysis facilitates the identification of the modeling issues that have the largest impact in

prediction accuracy and the systematic evaluation of the potential improvements that could be expectedfrom the use of aircraft intent information that will beavailable when air-ground data link is deployed. Thesources of errors in trajectory prediction have beenamply studied; however, a clear understanding of therelative contribution of these errors to accuracy

performance using realistic scenarios is necessary inorder to be able to improve the models. The accuracyfactors ranking methodology presented here relies onthe computation of the Past Maximum Deviation(PMD) metric for each error measurement and usingthe PMD to attribute the error to one of four possibleerror source categories: lateral, vertical, velocity andheading. The PMD metric keeps track of the largesttrack-trajectory deviation prior to the measurementalong each of the four categories. PMD distributionsserve as a diagnostic that reveal the performance of atrajectory predictor (TP) along the four dimensions

discussed and can be used as an effective tool tocompare two TPs or two variants of the same TP.Results of accuracy factor analysis based on the EnRoute Automation Modernization ERAM parametricalgorithm using a live recorded scenario are presented.

I. Introduction

Given the key role played by trajectories withinthe Trajectory Based Operations (TBO) concept therehas been renewed interest in evaluating new or

enhanced 4D trajectory (4DT) modeling approachesthat could potentially deliver better performance.Since trajectories are the primary input to thedecision support tools (DSTs) used by ATMoperators, the accuracy of trajectory algorithms is of paramount importance, however it is recognized thatdue to the large disparity in trajectory predictor (TP)algorithms and accuracy evaluation approaches it isdifficult to discern a TP solution path capable of supporting TBO needs. The FAA/Eurocontrol ActionPlan 16 has put forward a convenient conceptualinfrastructure to guide TP development and

evaluation and to define common terminology thatenables comparisons between different TPapproaches [1]. A TP is an algorithm that computesthe expected position (latitude, longitude andaltitude) of the aircraft, as a function of time, basedon flight plan information, controller clearances,weather data, and aircraft state. A trajectory, the mainoutput of a TP, is a representation of the computed

8/3/2019 05655521


1.C.1-2

prediction usually encapsulated in a data structureconsisting of a list of waypoints with altitude,corresponding estimated times of arrival (ETA) and possibly other attributes. A TP is not a monolithicalgorithm but rather a collection of algorithms eachspecialized to solve a particular modeling problem to build a trajectory. Thus, it is envisioned that as

existing TPs evolve towards meeting therequirements to support TBO, there is need toevaluate the impact of individual factors to trajectoryaccuracy to be able to focus algorithmic improvementefforts where they have more relevance.

Trajectory predictors in operational air andground systems currently implement two differentalgorithmic approaches: kinetic and parametric (akakinematic). In the kinetic approach, detailedmodeling of aircraft operations performance(aerodynamics, engine thrust, fuel consumption, andaircraft envelope) is used to solve the point-massequations of motion. In the parametric approach,speeds and vertical rates are obtained from lookuptables that have been prepared from empirical dataand/or from pilot performance handbooks. Theadoption and level of maturity of these twoapproaches has been driven by system requirements,CPU demands and by the availability of the inputdata required by the algorithms. Note that thedifference between kinetic and parametric approachesis mainly relevant for vertical modeling. Cruise levelsegments can be modeled using the cleared cruisespeed, the observed track speed and the component of

the wind along the planned horizontal path (no needto integrate equations of motion).

The TP implemented in the FAA’s En RouteAutomation Modernization (ERAM) system Release1 and 2 (R1, R2) uses the parametric trajectorymodeling software ported over and enhanced fromthe User Request Evaluation Tool (URET) developed by the MITRE Corporation [2]. A parametrictrajectory modeling was also used in the FAA’sEnhanced Traffic Management System (ETMS),developed by the Volpe Transportation System

Laboratory [3]. Kinetic models are the natural choicefor the TPs used in Flight Management Systems(FMS) because they can take advantage of thespecific aircraft performance and aircraft intentinformation that is available on board the aircraft.The FAA’s Traffic Management Advisor (TMA)uses the kinetic model developed by NASA for theCenter TRACON Automation System (CTAS) [4].

Another kinetic model commonly used isEurocontrol’s Base of Aircraft Data (BADA) [5].

Trajectory accuracy measurements are stronglydependent on the scenario used. It is not the same tomeasure prediction accuracy using a flight“untouched” by the controller than using a flightsubject to repeated tactical intervention bycontrollers. For this reason, to achieve a meaningfulaccuracy performance assessment it is imperative touse realistic scenarios that reflect actual flightroutings and operational details that affect changes of the flight throughout its lifetime. Exposing thetrajectory models to realistic inputs is an effectiveway to diagnose performance problems and tocompare algorithm alternatives; however, given thecomplexities of real life operations it is difficult tomake a one to one comparison between two TPs inthese circumstances. There are numerous reports inthe technical literature that show trajectory accuracymeasurements that cover a range of two orders of magnitude! Presenting accuracy results in terms of anaggregate metric is not enough to reveal themultidimensional nature of accuracy performance.The problem is compounded by the lack of uniformity in scenario data and measurementapproaches. Reference [1], while highlighting theneed for a comprehensive sensitivity analysis of trajectory prediction factors, has pointed out thatmost prior investigation of TP accuracy has focusedon specific DST applications with results not easilycross-comparable to other applications. This paper

presents a method to assess the relative contributionof different factors to the accuracy performance of agiven TP. The accuracy factors analysis describedhere serves as a tool to identify the TP elements thathave the lowest performance and by quantifyingaccuracy performance along multiple dimensions itallows meaningful and systematic comparison between alternative TP algorithms. The paper isorganized as follows: an introductory section presents basic definitions and results from prior research,Section II deals with some important considerationsthat need to be taken into account in trajectory

accuracy work, Section III describes the mostcommon sources of errors in trajectory prediction,Section IV presents the accuracy factors analysisapproach, and the following section illustrates theapproach when applied to the ERAM-R1 TP.

8/3/2019 05655521


1.C.1-3

II. Trajectory Accuracy

A straightforward approach to quantifytrajectory accuracy is to compute the deviations of the predicted lateral and vertical position and timealong the trajectory relative to a reference “as flown”4D path [6]. Agreement in accuracy metrics,however, is not enough in order to make comparableassessments of different TPs. The literature is full of trajectory accuracy reports that quote a large range of performance numbers, but because they are not‘normalized’ these numbers are misleading at best. Astudy of URET and CTAS accuracy performancereports an average absolute horizontal error thatdepends on look-ahead time and ranges between 1.2nmi to 10.2 nmi for URET and 0.3 nmi to 10.9 nmifor CTAS, both within a look-ahead range of 0 to 30min [7]. The authors of the study point out that theresults cannot be used to compare CTAS versusURET because the scenario data is different for eachDST’s measurement. An evaluation of the performance of kinetic versus parametric models inthe context of traffic flow management (TFM)applications indicates that kinetic models exhibit better accuracy performance [8]. The study reportsETA errors ranging from 1.2 to 3.4 min for ETMS(parametric model) and from 0.2 to 2.4 min for CTAS (kinetic) within a look-ahead range of 0 to 60min. It is not clear, however, whether the poor performance of the parametric model reported in thatstudy is due to issues with the speed look-up tables or if it is really a problem with the algorithm.

Reference [9] reports a horizontal error (distancefrom radar reported position and time-coincidenttrajectory predicted position) for ERAM that growsnear linearly from 0.8 nmi to 6 nmi for look-aheadtimes of 0 to 20 min. Using an extensive database of recorded field data in the USA, Europe and Australiaresearchers show that lateral intent errors (in the USAsample mostly) are the dominant source of trajectoryerrors [10]. They report a standard deviation of measured lateral track-trajectory deviations of 20.8nmi. The same report quotes lateral deviationstatistics based on Australian traffic for flights

virtually “untouched” by ATC, with a dramaticdifference of a factor 800 smaller than those found inUSA traffic. Using a flight simulator to provide“ground truth”, ETA prediction errors of an FMS-TP(kinetic model) were measured resulting in errors ashigh as 37 sec [11] while an experiment with actualaircraft reports ETA errors of less than 5 sec based on

FMS trajectory predictions [12]. A sensitivityanalysis of trajectory accuracy factors performed bysimulations where each accuracy factor is randomlyvaried around a baseline realization indicates that(setting aside intent errors) speed and wind errors arethe major source of trajectory accuracy errors [13].The latter is an interesting study aimed at assessing

the impact of separate accuracy factors but it relies onknowledge of prior error distributions and does notincorporate (in a direct way) the natural weight of factors that is present in real life scenarios. Incontrast, the accuracy factors analysis presented hereis based on realized traffic that captures theunderlying relative weight of the many factorsaffecting accuracy.

The broad overview of trajectory accuracystudies reported above clearly manifests the state of confusion and limited applicability of results due tothe lack of standardization in conducting and presenting trajectory accuracy results. There are four normalization factors that have to be homogenized before any accuracy comparison can take place: (a)definition of the reference 4D path; (b) data samplingapproach; (c) accuracy metric and statistic; and (d)scenario data.

A. Reference 4D Path

The reference 4D path for accuracymeasurements (aka “ground truth”) is usually takento be the smoothed position reports produced by the

tracker based on surveillance sensor inputs, which isa good approximation to the “as flown” path. Theraw sensor data, in the case of radar targets alone, istoo noisy and would pollute the trajectory accuracymeasurements; however as the adoption of the moreaccurate ADS-B surveillance gains wider use it will be possible to use the sensor data directly as a verygood representation of the “as flown” path. The useof tracker outputs on the other hand could bias theerror measurements because of the inherent lag tomaneuvers in the underlying smoothing function inthe tracker (an IMM Kalman filter in the case of

ERAM and a α-β tracker in the Host Computer System in En Route centers). Tracker outputs alsohave residual noise that could leak into trajectoryaccuracy measurements. An alternative to generatethe “as flown” flight is to use an algorithm, such asthe Path Extraction (PE) algorithm described in

8/3/2019 05655521


1.C.1-4

Reference [14], that extracts the noise-free “asflown” path from noisy sensor data

B. Data Sampling Approach

Even though the mathematical representation of a trajectory consists of discrete points along the

predicted 4D path, a trajectory should be thought as a prediction continuum with time interpolation providing the predicted position at any arbitrary time.For accuracy measurements a selection of pointsalong the trajectory has to be made. The datasampling approach determines the frequency and thetimes when measurements are computed. Thesampling approach in Reference [15] defines astarting point along the trajectory, call it T0; makes ameasurement of the track to trajectory deviations atT0 and at future times at T0 + Δ1, T0 + Δ2, T0 + Δ3, T0

+ Δ4, etc., with {Δ1, Δ2, Δ3, Δ4,…} being fixed look-

ahead times (i.e. 1 min, 5 min, 10 min, 15 min, etc.);sampling continues by successive iterations with T0 incremented by a sampling step (i.e. 2 min) andmeasurements are done again at T0 and the T0+Δi times. This approach implements a “controller viewpoint” whereby at any given time (hence the 2min sampling) the controller can look at the screenand at that time it is pertinent to ask what is theestimated prediction errors for different look-aheadtimes (Δi) relative to T0. Note however that thissampling approach blends measurements withdifferent future times relative to the time of the

prediction, or trajectory build time (T b): suppose thatat time T b the system builds a trajectory; at this time(T0 = T b) accuracy measurements are made selecting points along the trajectory at times T0 + 1 min, T0 + 5min, T0 + 10 min, and so on; a second round of accuracy measurements for the same trajectorycontinue with T0 updated to T0 ← T0 + 2 min; in thissecond round the measurement done at T0 + 5 mincontributes to the “5 min” look-ahead bin eventhough the effective future time relative to T b is 7 min(5 min + 2 min) not 5 min, and so on. For someapplications, it is more relevant to measure trajectoryaccuracy in specific future time bins that do not blendmeasurements at different future times. Thissampling approach [16] represents the “system view”and it is useful for prediction error analysis, wherethe relevant question is: given that I have a trajectory prediction produced by the system at time T b (thetime of the forecast), what are the expected errors(lateral, horizontal, vertical, and ETA) x minutes in

the future at time = T b + x min? Analytical studies of error propagation work with specific future times, notwith look-ahead times that blend different futuretimes. For example, the longitudinal error (σs)estimated at some future time ΔTf (relative to thetime of the forecast T b) due to velocity uncertainty(σv) can be computed analytically, with s = v ΔTf :

σs = (∂s/∂v)σv = ΔTf σv (1)

The parameter that enters the error propagationformula is ΔTf , namely a specific future time thatcannot include a blend of different future times. Theresults presented in this paper use an implementationof “system view” sampling as follows: for eachtrajectory define a build time (T b) equal to the timewhen the system made the prediction; for eachtrajectory select all of the available track positionreports between time T b and the time when that flightreceived a new clearance; for each of the selected

track position reports compute the track trajectorydeviations and define the future time as ΔTf = Tt – T b,where Tt is the time stamp associated with the track position report; this measurement contributes to thefuture time bin where ΔTf falls. Figure 1 illustratesthe sampling strategies discussed above.

Figure 1. Sampling Approaches

C. Error Statistic

Although it may seem borderline nitpicking it isimportant to specify the statistic used when reporting

accuracy measurements. Researchers use root-mean-squared (RMS), standard deviation (σ), averageabsolute deviation (AD), and the standard deviationof the AD to quote their measurement results. It turnsout that trajectory prediction errors, unlike for instance the random jitter noise that characterizessensor reports, do not benefit from the central limit

8/3/2019 05655521


1.C.1-5

theorem [17] because, although stochastic, flightdeviations from its plan often times result fromhuman actions that do not necessarily follow atendency towards gaussianity and result in very largetails in the error distributions. The ETA error distribution obtained from the 5 hour scenariodescribed in Section V-A exhibits highly non-

Gaussian (kurtosis = 5.7) but clear exponentialdistribution with much longer tails than a normaldistribution (the χ 2/DOF is 34065/97 for a fit to aGaussian and 975/96 for a fit to a doubleexponential). With these types of long taileddistributions there is a wide discrepancy between thevarious statistics. In the example mentioned abovethe standard deviation differs from the AD by a factor of 2. Whereas for a normal distribution the relation σ= sqrt(π/2)*AD holds, for a long tailed distributionthe AD statistic tends to sub-estimate the width of thedistribution. With long tailed distributions the

difference between AD and σ could be significant;however one frequently sees experiment results thatfreely quote errors using different statistics, a practicethat could yield misleading comparisons. It is alsocommon to see accuracy results reported using ADand its standard deviation. The latter statistic is particularly problematic and misleading. Whereas thestandard deviation of signed errors has astraightforward probabilistic interpretation (±1-σaround the mean contains 68.27% of the data, if normally distributed) the interpretation of thestandard deviation of the AD is not clear. The ADs

are distributed near exponentially, the distribution isone-sided, always occupying the positive side; thestandard deviation of this distribution representssome width around the average, but the distribution ishighly asymmetric, thus thinking that ±1 standarddeviation around the average is some sort of symmetric width is totally misleading. Furthermore,since for an exponential distribution the mean isequal to the standard deviation, this metric does not provide additional information.

[10] makes the interesting suggestion to use theinter-quartile range (IQR) as an estimator of the

width of the error distribution. The IQR measures thewidth occupied by 50% of the data around the center and as such is less sensitive to outliers. The authorsshow that the IQR of the distribution applied tolateral prediction errors can be used to extract theerror component not affected by controller actions.The standard deviation (which is equivalent to the

RMS when there are no systematic errors) is the preferred statistic to report accuracy measurements.The advantages of the standard deviation are itssensitivity to large deviations, the fact that it is a wellestablished and standard estimator of the width of adistribution, and its analyticity (see for instanceEquation 1; the AD is not amenable to analytic

manipulation because it is defined in terms of theabsolute value operator). Since large deviations dooccur in real life situations a statistic that is notsensitive to large deviations is not desirable.Measurements of trajectory prediction errors are notonly use for TP performance but also can be used asinputs to computations and simulation exercises,therefore care must be taken to characterize the actualerror distributions appropriately (that is including theeffect of long tails). Trajectory accuracymeasurements are tied to a specific look-ahead,however, often times researchers quote an error

number without specifying the look-ahead time thatapplies for the quoted result. A useful and moreinformative trajectory error metric is the slope of theerror versus look-ahead time curve.

D. Scenario Data

A measurement of TP accuracy is also ameasurement of features of the scenario used. Thesame TP could yield significantly different accuracyresults for two different input data sets. This is atrivial statement, but it helps explaining the wide

range of accuracy results that have been reported andhighlights the risk of attributing metrics results solelyto TP performance. Ideally one would use the sameinputs when in need to compare two TPs or one TPwith two variants of the algorithms. Assessing theaccuracy performance of a TP entails the use of arealistic scenario so as to exercise all the componentsof the trajectory modeling algorithms that have animpact on real life operations. In this environmenthowever there are complications that produce resultsmore difficult to interpret. Two related effects thatarise in scenarios derived from live recorded data are‘selection bias’ and short look-ahead segments.Selection bias is explained in the next subsection,short look-ahead segments refers to the fact that inreal life operations flights undergo frequentchanges [10], [18] (tactical intervention by thecontroller, strategic planning, weather relatedreroutes, operator preferences, open endedclearances, etc.) Trajectory accuracy measurements

8/3/2019 05655521


1.C.1-6

can be done for a trajectory for a prediction timewindow that goes from the time the trajectory is built(which is the present time) to any future time before aclearance changes the plan. As indicated, that timewindow could be severely shortened under heavytraffic load (could be few minutes on average) andlimits the sample size for longer look-ahead times.

E. Selection Bias

An immediate consequence of the effectdescribed above where most of the trajectoryaccuracy measurements tend to come from shortsegments (hence short look-ahead times) is thatmeasurements for long look-ahead times tend to be biased towards smaller deviations. Thiscounterintuitive result is explained as follows: flightsthat have a long straight and level cruise phase aremore likely to contribute to long look-ahead bins, but

precisely those long straight and level segments arewhere one would expect small track-trajectorydeviations (hence its name ‘selection bias’).Conversely, flights rapidly undergoing changes aremore likely to show high track-trajectory deviationsand are contributing preferentially to short look-ahead times. Due to this effect (but stronglydepending on the scenario) it is possible to obtainmeasurements that show trajectory errors decreasingfor longer look-ahead times instead of increasing asone would expect theoretically (Equation 1). Notehowever that, because of the smaller number of long

straight and level segments, the sample size for longlook-ahead bins tends to be much smaller than that of short look-ahead bins and as a result the accuracymeasurement in these bins is less robust. This facthighlights the need for researchers to quote thenumber of entries per bin (N) along with the RMS or σ on each look -ahead bin (N is an important piece of information needed to assess the statisticalsignificance of the measurement because the error of the RMS or σ tend to decrease as ~1/sqrt(2 N)).

III. Trajectory Accuracy Factors

To assess the performance of a TP it is necessaryto understand the impact of all possible sources of errors. The impact of each accuracy factor can bestudied individually by analytic or semi-analyticmeans [19], however due to the complex situations present in realistic scenarios it is necessary toevaluate the relative impact of each factor given real

world input data. This task is made difficult by thestatistical nature of accuracy measurements thatrequire sufficiently large sample sizes. The nextsection describes how the accuracy factors analysisfacilitates this task. Potential sources of error intrajectory prediction are listed below and reflect therelevant accuracy factors identified by a team of

experts [13].

Route intent: situations when the controller issues an altitude clearance or change in route (hold,vector, direct-to) that is not entered into the groundautomation. Open ended clearances can also beclassified as route intent issues.

Aircraft intent: inputs known to the FMS system but not available to the ground TP such as cost index,turn parameters, thrust de-rating, target level-off altitude, navigation mode engaged (LNAV, VNAV),whether the aircraft is flying to meet an RTA,

knowledge of the top of descent, etc.ATC factors: special restrictions associated with

inter-center coordination procedures that are notcoded into automation; unknown runway assignment;metering and other traffic constraints usually result inmodifications of the flight not known to the trajectorymodeler.

Wind velocity: inaccuracies in weather forecast,specially wind speed and direction are a major sourceof prediction errors.

Aircraft performance modeling: for parametric

algorithms aircraft performance tables with speedsand vertical rates are built from empirical data usinga representative sample, however due to thedifferences in aircraft behaviors depending onairports, regions, weather, etc. the aircraft tables aresubject to large statistical variances. Kineticalgorithms on the other hand are subject to errors inthrust and drag coefficients, as well as uncertaintiesin aircraft mass.

Algorithm implementation: neglecting verticalwind gradient; approximating constant wind along a

segment; fidelity of turn modeling; approximations inthe equations of motion; algorithm parameters thatdetermine segment size, trajectory build triggers androute re-join logic; earth model (WGS84, conformalsphere, flat plane, etc.)

8/3/2019 05655521


1.C.1-7

Departure time: modeling a trajectory for a flightin pre-departure phase is subject to large uncertaintiesin departure time.

IV. Accuracy Assessment Approach

A. Error Source CategoriesBefore going into the details of prediction error

analysis it is important to make a clear distinction between trajectory errors due to intent issues anderrors due to the ‘physics’ in the model. In genericterms intent issues could be understood as largetrack-trajectory deviations due to the flight notfollowing the cleared route (lateral intent) or notcomplying with altitude restrictions (vertical intent).Lateral intent errors are typically observed when thecontroller verbally communicates a route amendmentto the pilot (hold, path stretch, direct-to, etc.) and the

information is not entered into ground automation.Lateral intent issues have been identified as asignificant source of trajectory predictionerrors [10,8]. The type of errors referred to as‘physics’ modeling errors is due to limitations or approximations in the prediction algorithms or theinputs that the algorithms use. Examples of modelingerrors are the use of nominal aircraft mass instead of its actual value in kinetic models; statistical variancein the aircraft performance parameters used in parametric models; wind forecast errors;approximating turns by instantaneous changes in

course, etc. Whereas physics model errors can in principle be reduced by increasing the fidelity of thealgorithms and the accuracy of the input data, barringArtificial Intelligence (not a practical option in theATM context) intent errors cannot be resolvedalgorithmically. For the reasons stated above it isimportant that in evaluating TP accuracy performance, researchers indicate to what extent theresults are due to intent issues: the study inReference [10] quotes a dramatic difference (up to afactor of 800!) in accuracy results when lateral intenterrors are not present. In other words, a trajectory

accuracy experiment polluted by intent issues is notmeasuring the accuracy of the TP core physicsalgorithms but something else instead. AlthoughTFM applications are affected by intent errors (theyuse the available trajectory at any given time) tofocus on evaluating and improving the core physicsalgorithms one must separate out intent issues. It is

foreseen that as ground automation systems evolve insupport of TBO, controllers will have the tools togrant re-route clearances through the automationsystem, hence removing or substantially reducingintent issues. The aim of this paper is to studyalgorithm accuracy factors.

For the purpose of analyzing the sources of prediction errors it is helpful to group the accuracyfactors described above into four categoriesaccording to the dimension where the effect operates,namely: lateral, vertical, velocity and heading (other relevant categories could be added). A simplisticnotional example (Figure 2) consisting of a smalllateral deviation (red broken line) shows how an ETAerror can be attributed to a lateral effect.

Figure 2. Lateral Deviation Example

In the example the TP models the trajectory inthe lateral dimension to go from A to B to C. Assumethat the flight is level at the same altitude throughoutthose two segments, the velocity is constant and thechange in course angle between AB and BC is verysmall (just a few degrees so that turn modeling detailsare not important). An ETA error is measured at point C and beyond. Analysis of the history of track position reports in the past of C is sufficient to findthat the measured ETA error is due to the lateraldeviation (d) that shortened the path and henceallowed the aircraft to cut few seconds of its plannedtrajectory.

Define the past maximum deviation (PMD)metric as the largest track-trajectory deviation in the past of the measurement in any of the dimensions of interest (lateral, altitude, velocity, accumulatedheading). In the example the lateral PMD is d,whereas the velocity, altitude and heading PMD are

null. The heading PMD is somewhat different than inthe other dimensions: it keeps track of the totalchange in course angle accumulated in the past of themeasurements. The idea is to determine to whatextent an ETA error is due to turn modeling effects,lateral deviation, velocity deviations or verticalmodeling effects. The approach consists of attributing

8/3/2019 05655521


1.C.1-8

observed errors to one of the four possible categories;at the end of the exercise one obtains a distribution of the frequency with which a given category isresponsible for the observed errors. Suchdistributions provide a signature of the accuracy performance of a given TP and allows for meaningfulcomparison with another TP or a variant of the same

TP while identifying and quantifying the specificcomponents of the TP that show weak performance.

Further insight into the root cause of a predictionerror can be gained by correlation analysis: once anerror has been attributed to, for instance, a lateraleffect, then looking at correlations (or lack thereof)with large PMDs in the other dimensions providesinformation that in some cases points to a specific physical cause. To continue with the example inFigure 2, one could imagine a variant where thechange in course angle between the segments AB andBC is very large (for instance 60 degrees), in whichcase there could be a large velocity PMD anddefinitely a large heading PMD indicating that in thiscase the ETA error is most likely due to a turnmodeling error. Yet another variant of the originalexample could be a small lateral PMD (say d < 0.3nmi) accompanied by a large velocity PMD, in whichcase the ETA error is most likely due to a velocitymodeling effect. Additional detail on error sourcesand distributions is obtained by performing theanalysis for data sub-sets selected according to the phase of flight (or other flight attributes of interest).

B. Accuracy Factors Analysis

To show how the accuracy factors analysismethod works we will focus on ETA errors. ETAerror is defined as the difference between the predicted crossing time at a point in the future alongthe trajectory and the actual time when the aircraftcrosses that same point. The process starts bydefining a suitable error threshold beyond which ameasured error is considered a significant deviation.For example, for the analysis presented below thisthreshold was set to 10 sec for ETA error. Each time

that a measurement of ETA error exceeds 10 sec theanalysis proceeds to attribute the error to one of thefour source categories (lateral, vertical, velocity or heading). For each track position report a track-trajectory deviation measurement is performed (per “system view” sampling discussed in Section II-B).As the trajectory is traversed with each subsequent

measurement, a record is kept with the PMD inlateral (cross-track), altitude and velocity. Also, theaccumulated course angle change is stored. ThePMDs observed prior to a detected ETA error exceeding the 10 sec threshold are used to determinethe factor most likely responsible for the ETA error.In other words, here one has a measurement of a

significant ETA error the question is, which of thefour factors is this error associated with? To find ananswer one can look at the PMDs and see if there is alarge past deviation in one of the dimensions that canexplain the ETA error.

Potentially one could see deviations in all four dimensions or various combinations, in which case itis necessary to compare the PMDs and to choose theone with the largest relative strength. To make thePMD comparison in different dimensions the PMDsare normalized to a suitable reference deviation thathas comparable probability on each one of thedimensions. With the data used in the experimentdescribed in the next section it is observed that theaggregate track-trajectory deviations of the sub-set of trajectories that adhere to the plan at all times yield95-percentiles of 3.5 nm (cross-track), 40 kt(velocity) and 800 ft (altitude). The PMDnormalization factors are set equal to these 95- percentiles. To attribute the ETA error to one of thefour categories the normalized PMDs are computedand the category with the largest normalized PMD ischosen as the source of the error. With an ETA error (> 10 sec) following a lateral PMD of 7 nm, a speed

PMD of 10 kt, and altitude PMD of zero (the flight islevel throughout all this time) the normalized PMDsare: lateral PMD = 2 (7 nm/3.5 nm), speed PMD =0.25 (10 kt/40 kt), altitude PMD = 0 (0/800 ft),therefore the cause of the ETA error is attributed to alateral effect. This example also helps illustrate the presence of correlations. A large lateral PMD could be accompanied by a large speed PMD: the aircraftslowing down to take a sharp turn. By looking at thePMD combinations that exhibit large values in theevent of a large ETA error it is possible to go deeper in the root cause analysis responsible for the error.

V. Experiment

The accuracy factor analysis described inSection IV was applied to the ERAM-R1 TP. Thesubsections below describe the data used, the analysisand the results of the experiment.

8/3/2019 05655521


1.C.1-9

A. Scenario and Data Processing

The primary data used for the experiment was provided by the FAA and consists of a 5 hour 22minutes recording of flight and surveillance datacollected in March of 2005 at the United States’Washington Air Route Traffic Control Center (ARTCC). The scenario has 2259 flights and was fed

to the ERAM system. Upon completion of the ERAMrun, track and trajectory data was extracted from thesystem analysis recording (SAR) and made availablefor offline analysis. The EZR tool, used for variousanalysis tasks in ERAM [20], was enhanced with theaccuracy factor analysis algorithms. The toolcomputes track-trajectory deviations, PMDs, andother metrics useful for analysis. Before the analysis,measurements for tracks with reported altitude below15,000 ft and measurements performed while theaircraft is out of adherence were discarded. Also,measurements are flagged according to the phase of

flight (climb, descent, cruise) using the vertical ratereported by the tracker.

As stated in Section IV-A the focus of this studyis modeling errors, therefore it is important to discardmeasurements affected by intent issues and pre-departure uncertainties. Only trajectories for activeflights were used. A trajectory that is built before theflight has departed is subject to large departure timeuncertainties resulting in unbounded ETA andlongitudinal prediction errors that cannot be solvedalgorithmically within the TP. To understand how

lateral and vertical intent issues were handled it isimportant to look into the re-adherence logic in theERAM TP. For each track report received by the TPthe algorithm checks whether the position andaltitude is within adapted thresholds of the AircraftTrajectory (ERAM maintains two trajectories: theAircraft Trajectory which is used by functions suchas conflict probe that required lateral re-adherence,and the Flight Plan trajectory, used by functions suchas boundary crossing determination that require amore stable prediction; the Aircraft Trajectory is usedin this study). When the track deviation exceeds anyof the adapted thresholds the trajectory (AircraftTrajectory) is remodeled using heuristics to rejoin the path to the cleared route and altitude. Due to the re-adherence logic the TP could rebuild the trajectorymultiple times during the lifetime of a flight,

however, at any given time the available trajectory atthat time is used for accuracy measurementsincluding future times (up to such time when a newclearance is received) during which out of adherencecould be triggering new trajectory builds. To removelateral and vertical intent issues affecting theexperiment, a filter was applied to discard

measurements with a lateral PMD greater than 7 nmiand a vertical PMD greater than 2500 ft. After all of the data selection filters were applied there remains atotal of 65,227 measurements left for analysis.

B. Results

Figure 3 shows the distribution of the fraction of ETA errors categorized by cause for the cruise phaseof the flight and in three look-ahead bins (< 5 min, between 5 – 10 min and > 10 min). Similar distributions for climb and descent phases are shown

in Figure 4 and Figure 5.

Figure 3. Attribution of ETA Errors (Cruise)

Figure 4. Attribution of ETA Errors (Climb)

8/3/2019 05655521


1.C.1-10

Figure 5. Attribution of ETA Errors (Descent)

The error attribution distributions indicate thatvelocity effects are the dominant source of errorsduring the cruise portion of the flight and verticalmodeling effects during climb and descent. The plotsalso show that turn modeling effects are negligible.Even though these results were expected based on prior extensive analysis of ERAM TP performance,the accuracy factors analysis provides a systematicway to quantify the sources of error and their relativeimpact, therefore allowing for a more robust methodto compare the accuracy performance of differentTPs or variants of the same TP. The aim of theseinitial measurements is not to obtain an absolutemeasurement of TP accuracy but rather to establish a baseline measurement and to guide the plannedefforts towards improving trajectory algorithms.

The dominance of velocity errors during thecruise phase is consistent with previous studies based

on flight tests [21] and by sensitivity analysis basedon simulations [13]. The fact that velocity effects, notlateral effects, are the main cause of ETA errorscould be explained by the multiplicative effect inerror propagation (see Equation 1). Whereas a lateraleffect, for instance the few seconds gained whencutting a corner, translates into an ETA error equal tothe time gained or loss during the maneuver, avelocity effect grows with time throughout theduration of the flight. Velocity modeling errors could be due to wind velocity inaccuracies or to speeddeviations relative to the cleared speed. Pilots deviate

from the cleared speed in order to follow a Cost

Index (CI) or RTA but the actual speed is usually notreported to the ground system (re-adherence logic inERAM provides for adjusting the speed used inmodeling according to the observed speed based ontrack data). A study of speed variance due to CIindicates that speed variations of 10% can beexpected while in cruise and up to 30% during climb

and descent [22]. Similarly, wind speed errors could be a major source of velocity modeling errors.Measurements of the accuracy of the 40-km RapidUpdate Cycle 2 (RUC-2) wind data based on actualmeasurements on board a Boeing 737 report frequentwind speed errors in excess of 20 kt and occasionallyreaching up to 58 kt [23], which translate to ETAerrors in the range 1 – 3 min and longitudinal errorsin the range 7 – 20 nmi for a look-ahead time of 20min. Note that speed errors during cruise equallyaffect kinetic and parametric models (both are basically using the cleared cruise speed), a fact that

stresses the potential benefits of using aircraft intent(i.e. speed schedule) downloaded from the aircraft intrajectory modeling.

Table 1 shows a detailed breakdown of all possible combinations or accuracy factors for thecruise data. The table is constructed by counting,separately for each error source category, the number of instances when the corresponding PMD exceedsits ‘detection threshold’ (explained in Section IV-A).An entry of “1” in a cell means that thecorresponding category was detected as a source.The column labels stand for lateral (LAT), altitude

(ALT), velocity (VEL) and heading (HDG). For example, the fifth raw with entries “0,1,01” meansthat 0.14% of the measurements had altitude andheading categories flagged as error sources. It isobserved that there are correlations between factors but they are small; most of the errors are attributed toa single factor with velocity being the major contributor. The results clearly indicate the prioritywhere modeling improvements need to be made: for the cruise phase, velocity accuracy is a source of errors with an impact a factor of seven higher thanlateral effects or vertical modeling.

8/3/2019 05655521


1.C.1-11

Table 1. Breakdown of ETA Error Cause Analysis

LAT ALT VEL HDG Contribution

(%)

0 0 0 1 0.78

0 0 1 0 69.27

0 0 1 1 0.18

0 1 0 0 11.030 1 0 1 0.14

0 1 1 0 3.00

0 1 1 1 0.07

1 0 0 0 12.93

1 0 0 1 0.41

1 0 1 0 1.00

1 0 1 1 0.03

1 1 0 0 0.58

1 1 0 1 0.12

1 1 1 0 0.44

1 1 1 1 0.01

VI. Conclusions

Due to the complexity of situations that occur inrealistic scenarios, comparing TP accuracy results based on an aggregate metric alone could bemisleading. It is necessary to look into variousdimensions in order to check how different accuracyfactors affect the accuracy performancemeasurements. It was demonstrated that the accuracyfactor analysis described in this paper is an effectivemethodology to perform meaningful and systematic

comparisons of diverse trajectory algorithms. Theapproach was demonstrated with measurements performed with the ERAM-R1 TP and, byquantifying the sources of error, showed that it provides an effective instrument to guide the plannedefforts to improve trajectory algorithms.

References

[1] Mondoloni, S., S. Swierstra, Oct 30, 2005,Commonality in disparate trajectory predictors for air traffic management applications, 24

thDigital

Avionics Systems Conference, Washington, D.C.,USA.

[2] Walker, M.G., 2002, A Benefits Assessment of the User Request Evaluation Tool (URET), Air Traffic Control Quarterly, Vol. 10, No. 4

[3] Volpe National Transportation Systems Center,June 1995, Enhanced Traffic Management System

(ETMS) Functional Description, Version 5.0, ReportDTS56-TMS-002, U. S. Department of Transportation, Cambridge MA.

[4] Erzberger, H., W. Nedell, June 1989, Design of Automated System for Management of ArrivalTraffic, NASA TM-102201,http://www.aviationsystemsdivision.arc.nasa.gov/publications/more/ctas/erzberger_06_89.pdf

[5] Eurocontrol, BADA (Base of Aircraft Data),Eurocontrol Validation Infrastructure Center of Expertise, Bretigny-sur-Orge, France,http://www.eurocontrol.int/eec/public/standard_page/ proj_BADA.html

[6] Ryan, H.F., M. M. Paglione, S.M. Green, 2004,Review of Trajectory Accuracy Methodology andComparison of Error Measurement Metrics, AIAA-2004-4787,AIAA GN&C Conference, Providence, RI

[7] Paglione, M.M., H.F. Ryan, R.D. Oaks, J.S.Summerill, M.L. Cale, May 1999, Trajectory prediction accuracy report: User Request EvaluationTool (URET) – Center TRACON AutomationSystem (CTAS), U.S. Department of Transportation,DOT/FAA/CT-TN99/10,http://acy.tc.faa.gov/cpat/docs/trjrpt_f2.pdf

[8] Reisman, R., J. Murphy, P. Krolak, R. Wright,Sep 2007, Modeling tactical trajectory accuracyeffects on traffic flow management operations, 7th AIAA Aviation Technology, Integration andOperations Conference (ATIO), Belfast, Northern

Ireland.

[9] Paglione, M., Santiago, C., Crowell, A., Oaks,R.D., Analysis of the Aircraft to Aircraft ConflictProperties in the National Airspace System, 18 – 21August, 2008, AIAA Guidance, Navigation and Control Conference and Exhibit , Honolulu, Hawaii,USA.

[10] Paglione, M., I. Bayraktutar, G. McDonald, J.Bronsvoort, Jun 29 – July 2 2009, Lateral IntentError’s Impact on Aircraft Prediction, 8th

USA/Europe Air Traffic Management Research and

Development Seminar (ATM2009), Napa, California,USA.

[11] Stell, L., October 25-29 2009, Flightmanagement system prediction and execution of idle-thurst descents, 28th Digital Avionics Systems

Conference, Orlando, Florida, USA.

8/3/2019 05655521


1.C.1-12

[12] Klooster, J.K., A. Del Amo, P. Manzi, Jun 29 – July 2 2009, Controlled Time-of-Arrival FlightTrials, 8th USA/Europe Air Traffic Management Research and Development Seminar (ATM2009), Napa, California, USA.

[13] Mondoloni, S., I. Bayraktutar, June 2005, Impactof factors, conditions and metrics on trajectory prediction accuracy, 6 th USA/Europe Air Traffic

Management Research and Development Seminar ,Baltimore, MD, USA.

[14] Torres, S., Nov. 2-5, 2008, Aircraft PathExtraction from Noisy Target Data, ATCA 53rd Annual Conference and Exposition, Washington.

[15] Lee Cale, M., S. Liu, R.D. Oaks, M. Paglione,H.F. Ryan, S. Summerill, Dec 3 – 7, 2001, "AGeneric Sampling Technique for Measuring AircraftTrajectory Prediction Accuracy", 4th USA/Europe Air

Traffic Management Research and Development

Seminar , Santa Fe, NM, USA,http://acy.tc.faa.gov/cpat/docs/tjs_atm2001.pdf

[16] Rozen, N.E., 2001, Functional performanceassessment of the User Request Evaluation Tool(URET), MITRE/CAASD, McLean, VA, USA,http://www.mitre.org/work/tech_papers/tech_papers_ 01/rozen_functional/rozen_functional.pdf

[17] Weisstein, Eric W., "Central Limit Theorem",MathWorld – A Wolfram Web Resource,http://mathworld.wolfram.com/CentralLimitTheorem.html

[18] Torres, S., E.G. McKay, 24-25 June 2008, IntentIssues Noted in Current U.S. En Route TrajectoryModeling, Synchronising Business Trajectories in the Future ATM System, Seville, Spain.

[19] Mondoloni, S., July 2006, Aircraft trajectory prediction errors: including a summary of error sources and data, FAA/Eurocontrol Action Plan 16Common Trajectory Prediction Capabilities, CSSI,Inc.

[20] Torres, S., E.G. McKay, 25-29 October 2009,Assessing Tactical Alert Function AccuracyPerformance, 28th Digital Avionics Systems

Conference, Orlando, Florida, USA.

[21] Williams, D.H., S.M. Green, 1998, Flightevaluation of the center/TRACON automation systemtrajectory prediction process, NASA TP-1988-208439.

[22] Rumler, W., Günther, T., Fricke, H., Weiβhaar,U, June 1-4, 2010, Flight profile variations due tospreading practice of cost index based flight planning, 4th International Conference on Research

in Air Transportation, Budapest.

[23] Cole, R.E., S. Green, M. Jardin, Jun 13 – 16,2000, Wind prediction accuracy for air trafficmanagement decision support tools, 3th USA/Europe Air Traffic Management Research and Development Seminar (ATM2000), Napoli, Italy.

Acknowledgements

The author wishes to thank Mike Paglione of theFAA’s Simulation and Analysis Team AJP‐661 for facilitating the scenario data used in this study.

29th Digital Avionics Systems Conference

October 3-7, 2010

Documents

05655521