American Statistical Association
Mike Kattan, a well-known proponent of prediction modelling, posed the following conundrum: A doctor approaches a statistician, saying "Here are two statistical models (or diagnostic tests): which one should I use?” What performance metric should the statistician use to answer the doctors question?
Kattan himself argued in favor of the concordance index. Here I examine a broad range of performance metrics, including sensitivity, specificity, positive and negative predictive value, Brier score, concordance index, net reclassification index. I give case studies applying these metrics to pairs of tests or statistical models, including molecular markers. Different metrics gave inconsistent answers as to the preferable model or test. No metric gave answers that were routinely consistent with the optimal model or test as determined by decision theory. I conclude that traditional performance metrics may be of value to statisticians, but have little value for helping clinicians make choices over appropriate tools to use in the clinic.
|Date:||Wednesday, October 15, 2008|
|Time:||4:00 P.M. - 5:00 P.M.|
Memorial Sloan-Kettering Cancer Center
Department of Epidemiology and Biostatistics
307 East 63rd Street
(between First and Second Avenues)
3rd Floor Conference Room
New York, New York
Note: To gain access to the building, please follow the directions by the telephone in the foyer.