American Statistical Association
New York City
Metropolitan Area Chapter

Memorial Sloan-Kettering Cancer Center
Biostatistics Seminar


Andrew Vickers, Ph.D.
Associate Attending Research Methodologist
Memorial Sloan-Kettering Cancer Center

FAILURE OF TRADITIONAL PERFORMANCE METRICS
FOR THE EVALUATION OF DIAGNOSTIC TESTS,
MOLECULAR MARKERS AND PREDICTION MODELS

Mike Kattan, a well-known proponent of prediction modelling, posed the following conundrum: A doctor approaches a statistician, saying "Here are two statistical models (or diagnostic tests): which one should I use? What performance metric should the statistician use to answer the doctors question?

Kattan himself argued in favor of the concordance index. Here I examine a broad range of performance metrics, including sensitivity, specificity, positive and negative predictive value, Brier score, concordance index, net reclassification index. I give case studies applying these metrics to pairs of tests or statistical models, including molecular markers. Different metrics gave inconsistent answers as to the preferable model or test. No metric gave answers that were routinely consistent with the optimal model or test as determined by decision theory. I conclude that traditional performance metrics may be of value to statisticians, but have little value for helping clinicians make choices over appropriate tools to use in the clinic.


Date: Wednesday, October 15, 2008
Time: 4:00 P.M. - 5:00 P.M.
Location: Memorial Sloan-Kettering Cancer Center
Department of Epidemiology and Biostatistics
307 East 63rd Street
(between First and Second Avenues)
3rd Floor Conference Room
New York, New York
Note: To gain access to the building, please follow the directions by the telephone in the foyer.

RESERVATIONS ARE NOT REQUIRED


Home Page | Chapter News | Chapter Officers | Chapter Events
Other Metro Area Events | ASA National Home Page | Links To Other Websites
NYC ASA Chapter Constitution | NYC ASA Chapter By-Laws

Copyright © 2008 by New York City Metropolitan Area Chapter of the ASA
Designed and maintained by Cynthia Scherer
Send questions or comments to nycasa@mindspring.com