SMS BASIC Scores are Not Valid Predictors of Crash Frequency
Inam Iyoob, PhD
Director of Engineering, Transplace
I am the Director of Engineering for Transplace and am a data analyst and mathematical expert with a PhD in Engineering from the
In advising shippers and brokers to use SMS methodology, the Agency concludes: “Internal, external, and independent (University of Michigan’s Transportation Research Institute) evaluations have all shown that, of the six BASICs based on regulatory compliance (the Crash Indicator BASIC is based on actual crashes), the Unsafe Driving BASIC and the Fatigued Driving (HOS) BASIC have the strongest relationships to future crash risk.”
In a separate study by Wells Fargo, the 200 largest carriers, for which there is actually sufficient data, were measured. No perceptible correlation between safety and SMS percentiles was noted in Unsafe Driving or in Fatigued Driving, the two BASICs the Agency proclaims as most definitive. The Wells Fargo Study concluded, “Quite simply, we found very little relationship (i.e., not statistically significant) between Unsafe Driver or Fatigued Driver scores and actual Accidents per Power Unit.”
Months after release of the Wells Fargo study, the Agency attempted to re-substantiate the
At the request of ASECTT, I have reviewed the FMCSA’s defense of SMS methodology as a valid predictor of carrier safety.
In refutation of the Wells Fargo conclusion, the Agency has submitted the two graphs shown below (Figures 1 and 2) arguing that the older 2009 Volpe National Transportation Study is more accurate than the Wells Fargo’s study because it effectively measures 29 and 43 thousand carriers, not just the largest 200.
Figure 1: FMCSA Regression of Averages – Unsafe Driving
Figure 2: FMCSA Regression of Averages – Fatigued Driving
An examination of the study demonstrates that FMCSA’s data cannot be used to predict the crash performance of individual carriers, even though the FMCSA claims SMS scores are correlated to the average crash frequency of hundreds of carriers at each percentile integral. Consumers of freight transportation do not select “average” carriers, they select individual carriers and the Agency study offers no proof that SMS methodology is a predictor of individual carrier safety performance at any percentile level.
Based upon data obtained from the FMCSA’s own data bank, I was asked to perform a detailed study of individual carrier percentile rankings and crash frequency correlations.
That study resulted in the graphs shown in Figures 3 and 4. The study clearly shows that with respect to individual carriers, percentile rankings of carriers both above and below the arbitrary “monitoring thresholds” indicated with theare not valid predictors of crash frequency. Regression analysis shows that SMS percentile scores account for less than one percent of the variation in crash frequency for each of these BASICs.
Figure 3: Unsafe Driving – Plot of 26,435 Carriers
Figure 4: Fatigued Driving – Plot of 35,933 Carriers
I can’t see any useful purpose in averaging the crash data of hundreds of carriers in each of 100 different percentiles and then calculating a regression of the average values. The purpose of regression analysis is to explain variation. Averaging hundreds of carriers at each percentile eliminates most of the variation in the data. It is not statistically accurate to say the SMS methodology and BASIC percentile scores are an accurate predictor of carrier safety predicated upon the crash data the Agency uses to justify its conclusions.
Logically, unsafe driving and driver fatigue do impact crashes. However, the way the SMS BASICs Unsafe Driving and Fatigued Driving are captured, calculated and interpreted by FMCSA does not show any correlation to crashes. Hence usage of SMS data for carrier selection will unduly favor some and penalize others, and thus should be avoided.