NEWS BRIEF: University of Maryland Study Sites Statistical Issues with FMCSA’s SMS BASIC Scores, Concluding Scores are Not Valid Predictors of Crash Frequency

 NEWS BRIEF

University of Maryland Study Sites Statistical Issues with FMCSA’s SMS BASIC Scores, Concluding Scores are Not Valid Predictors of Crash Frequency

Alliance for Safe, Efficient, and Competitive Truck Transportation Sponsors Study of FMCSA Data; Findings Prove Published SMS BASIC Scores Lack Correlation to Crashes and Suffer from Numerous Statistical Flaws

The Alliance for Safe, Efficient, and Competitive Truck Transportation (ASECTT) commissioned a study by a Dr. James Gimpel professor at the University of Maryland with a Ph.D. from the University of Chicago. Dr. Gimpel served as the editor of American Politics Research from 2003 to 2011 and is the author of numerous books and peer-reviewed articles. Dr. Gimpel performed a detailed study of the statistical soundness of the SMS methodology and the correlation between individual carrier percentile rankings and crash frequency based upon data obtained from the Federal Motor Carrier Safety Administration.


See the attached report Statistical Issues in the Safety Measurement and Inspection of Motor Carriers.

The study challenges the methodology as being “predisposed by design toward recordkeeping only on problems and violations” which is “tantamount to the naïve research error of selecting on the dependent variable”. The  study points out numerous geographic deficiencies of the methodology that penalize carriers for where they operate, not how safely they operate. The biases injected at the implementation stage prevent the BASIC indicators from assessing what they are intended to evaluate.

The study also challenges the deficiencies of SMS with regard to large versus small carriers. The data vastly over-represent the firms with very large fleets, while vastly under-representing the large number of small carriers. Of the 326,000 carriers with at least one inspection in the last two years, fully 200,000 or 61% have 5 or fewer inspections. Only 43,555 carriers (19%) have had 20 or more inspections. Data with fewer than 20 observations are not considered to meet a sufficient level of accuracy and most studies would not even publish such sparse data. Rates based on a small number of observations are highly variable and for that reason unreliable as measures. The law of large numbers indicates that only as the number of samples increases does the average of the sample represent the mean of the entire population. The vast majority of small carriers, which are critical to the efficient and effective flow of goods, will never have enough inspections in a two-year period to yield a valid mathematical score.

Finally the study is consistent with the two Wells Fargo studies and the ASECTT-Iyoob study in proving that with respect to individual carriers, percentile rankings of carriers both above and below the arbitrary “monitoring thresholds” indicated with the are not valid predictors of crash frequency. The Unsafe Driving BASIC accounts for only 2% of the variation in crash frequency for over 5 thousand fleets with 9-19 inspections. “Using the unsafe driving scores as a predictor of crash risk for these small carriers is little better than guessing, which is surprising given what these scores are supposed to indicate and how the data are generated with a bias toward violations.” Even for the 3,351 larger carriers with more than 150 inspection records, only 7% of the variance in crash frequency can be explained using Unsafe Driving BASIC scores. “As Wells-Fargo indicated, because it is intuitive that this relationship should be positive and clear-cut, there is either something wrong with the SMS measurement of unsafe driving, or something wrong with the sample of carriers in the MCMIS data.” The Unsafe Driving BASIC accounts for less than 1% of the variation in crash frequency for over 9 thousand fleets with 9-19 inspections. For the largest fleets, the study points out that the Vehicle Maintenance BASIC actually has a negative (but not statistically significant) correlation to crash frequency. The higher the BASIC the lower the accident frequency.

The study concludes that “violations are not reflective of the actual performance and safety of firms, but are an artifact of the application of the measuring instrument.“ Accidents are very poorly predicted by the BASIC scores in the MCMIS data and this is especially astounding given that the data generation process selects specifically on carriers supposedly at risk for accidents, not even including carriers until they have a violation. It is important to ask why the relationships are so weak. Certainly it is intuitively plausible that unsafe driving, poor vehicle maintenance and driver fatigue would be positively related to crash risk. There are a litany of systematic biases that are contaminating the SMS methodology, from the irregular data collection practices across geographic areas and agencies, to inappropriate definitions of the measures themselves. More inspections is not the answer because increasing the number of biased observations only amplifies the magnitude of the bias.

For more information about ASECTT and its position on CSA 2010/SMS methodology go to ASECTT.blogspot.com or contact ASECTT@gmail.com.