Full Content is available to subscribers

Subscribe/Learn More  >

Metrics for Evaluating the Accuracy of Diagnostic Fault Detection Systems

[+] Author Affiliations
Hans DePold, Jason Siegel, Jon Hull

United Technologies, P&W, East Hartford, CT

Paper No. GT2004-54144, pp. 835-841; 7 pages
  • ASME Turbo Expo 2004: Power for Land, Sea, and Air
  • Volume 2: Turbo Expo 2004
  • Vienna, Austria, June 14–17, 2004
  • Conference Sponsors: International Gas Turbine Institute
  • ISBN: 0-7918-4167-7 | eISBN: 0-7918-3739-4
  • Copyright © 2004 by ASME


This paper presents a method for providing metrics to evaluate the accuracy and cost effectiveness of diagnostic decision support systems. One intention of engine health management (EHM) fault detection systems is to have engines identified for removal and refurbishment as soon as there is evidence of an adverse gas generator trend shift. The benefits of EHM diagnostics and prognostics tests are derived from the resulting improved safety, the reduced operating costs, and most importantly, the good will and trust of the customer. The method presented in this paper is a generalized way of evaluating the performance of some of the tests that are used to make inspection, removal, and maintenance decisions [Ref 1,2]. The detection of faults from shifts in classification data is the first step in EHM systems that use diagnostics and prognostics [Ref 3,4,5]. The minimum parameter shift required to trigger a fault indication is called the threshold. Typically, it is a predetermined multiple of the standard deviation of the parameter measurements. Root cause isolation is usually invoked following these detection tests for the gas path parameter shifts. This paper shows how the achievable accuracy of diagnostic and prognostic system tests can be determined from the signal to noise ratio (SNR), and the system’s design (sensitivity and specificity). From these tests we extract two features, true positives (TP) and false positives (FP) that can be used to compare the accuracy of any simple or complex decision support system. This method is conducive to efficiently handling large amounts of data from multiple sensor tests because it avoids explicit correlation among individual diagnostic tests, and focuses instead on the net results. Each piece of classification information is used to reduce ambiguity. In this approach, the individual diagnostic tests and any data fusion weighting factors can be parametrically varied to optimize the accuracy of the decisions. The resulting plot of TP versus FP is then directly compared to the results of simple idealized classifier systems having known SNRs. This paper applies the receiver operating characteristics (ROC) process to evaluate the potential accuracy of EHM decisions. The paper also shows that the actual accuracy depends on how thresholds are set, and on the local shape of the ROC in the regions where it is used. The method presented can be applied to test the relative accuracy of each phase of the EHM decision-making process. The effects of test accuracies, event probabilities, and consequential event costs on the value of the decision support system are also presented.

Copyright © 2004 by ASME
Topics: Flaw detection



Interactive Graphics


Country-Specific Mortality and Growth Failure in Infancy and Yound Children and Association With Material Stature

Use interactive graphics and maps to view and sort country-specific infant and early dhildhood mortality and growth failure data and their association with maternal

Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging and repositioning the boxes below.

Related eBook Content
Topic Collections

Sorry! You do not have access to this content. For assistance or to subscribe, please contact us:

  • TELEPHONE: 1-800-843-2763 (Toll-free in the USA)
  • EMAIL: asmedigitalcollection@asme.org
Sign In