Monday, June 25, 2012

"SMS BASIC Scores are Not Valid Predictors of Crash Frequency" by Inam Iyoob, PhD Director of Engineering, Transplace

SMS BASIC Scores are Not Valid Predictors of Crash Frequency
Inam Iyoob, PhD
Director of Engineering, Transplace

I am the Director of Engineering for Transplace and am a data analyst and mathematical expert with a PhD in Engineering from the University of Arkansas and a Masters in Engineering degree from Oklahoma State University. I have 12 years of work experience with Transplace. 

In advising shippers and brokers to use SMS methodology, the Agency concludes:  “Internal, external, and independent (University of Michigan’s Transportation Research Institute) evaluations have all shown that, of the six BASICs based on regulatory compliance (the Crash Indicator BASIC is based on actual crashes), the Unsafe Driving BASIC and the Fatigued Driving (HOS) BASIC have the strongest relationships to future crash risk.”

In a separate study by Wells Fargo, the 200 largest carriers, for which there is actually sufficient data, were measured.  No perceptible correlation between safety and SMS percentiles was noted in Unsafe Driving or in Fatigued Driving, the two BASICs the Agency proclaims as most definitive.  The Wells Fargo Study concluded, “Quite simply, we found very little relationship (i.e., not statistically significant) between Unsafe Driver or Fatigued Driver scores and actual Accidents per Power Unit.” 

Months after release of the Wells Fargo study, the Agency attempted to re-substantiate the University of Michigan and Volpe National Transportation Systems Center studies in a paper devoted largely to touting the benefits of progressive intervention entitled “Review of Wells Fargo Equity Research Report on Compliance, Safety, Accountability” published March 16, 2012.

At the request of ASECTT, I have reviewed the FMCSA’s defense of SMS methodology as a valid predictor of carrier safety.

In refutation of the Wells Fargo conclusion, the Agency has submitted the two graphs shown below (Figures 1 and 2) arguing that the older 2009 Volpe National Transportation Study is more accurate than the Wells Fargo’s study because it effectively measures 29 and 43 thousand carriers, not just the largest 200.

Figure 1: FMCSA Regression of Averages – Unsafe Driving

Figure 2: FMCSA Regression of Averages – Fatigued Driving

An examination of the study demonstrates that FMCSA’s data cannot be used to predict the crash performance of individual carriers, even though the FMCSA claims SMS scores are correlated to the average crash frequency of hundreds of carriers at each percentile integral. Consumers of freight transportation do not select “average” carriers, they select individual carriers and the Agency study offers no proof that SMS methodology is a predictor of individual carrier safety performance at any percentile level.

Based upon data obtained from the FMCSA’s own data bank, I was asked to perform a detailed study of individual carrier percentile rankings and crash frequency correlations.

That study resulted in the graphs shown in Figures 3 and 4.  The study clearly shows that with respect to individual carriers, percentile rankings of carriers both above and below the arbitrary “monitoring thresholds” indicated with theare not valid predictors of crash frequency. Regression analysis shows that SMS percentile scores account for less than one percent of the variation in crash frequency for each of these BASICs.

Figure 3: Unsafe Driving – Plot of 26,435 Carriers

Figure 4: Fatigued Driving – Plot of 35,933 Carriers

I can’t see any useful purpose in averaging the crash data of hundreds of carriers in each of 100 different percentiles and then calculating a regression of the average values. The purpose of regression analysis is to explain variation. Averaging hundreds of carriers at each percentile eliminates most of the variation in the data. It is not statistically accurate to say the SMS methodology and BASIC percentile scores are an accurate predictor of carrier safety predicated upon the crash data the Agency uses to justify its conclusions.

Logically, unsafe driving and driver fatigue do impact crashes. However, the way the SMS BASICs Unsafe Driving and Fatigued Driving are captured, calculated and interpreted by FMCSA does not show any correlation to crashes. Hence usage of SMS data for carrier selection will unduly favor some and penalize others, and thus should be avoided.

1 comment:

  1. As a small Trucking co. owner, I feel the SAFER System is setup like a shelf life; it’s only a matter of time before you hit the % and have high numbers. This is due to SAFER’s focus to find problems and not also true solutions.
    Our drivers go through a class 1 inspection, get the CVSA stickers to put in the window and on the trailer, feeling really good they passed the big one, just to get a "warning" about a strap isn't tight enough or something. When the inspection hits SAFER it becomes a violation not a warning. There is no balance for passing an inspection only punishment for any small thing. (IE. strap not tight enough, chain 3” out of spacing, drivers signature isn’t legible (fatigued driver) and so on).
    In order to balance the system, we should get points removed for a good inspection. Where is the balance? I don’t see it? Out of a Class 1 DOT inspection, 37 inspection points are evaluated. You pass 36 clean, you get a warning on the 37th and for this we get 7 bad points in SAFER and no recognition for the other 36. Where is the balance?
    I was told by a FMCSA inspector “just getting good inspections, our numbers will go down”.
    How can good inspections happen when it is their job to uncover the dust and find something wrong and when they don’t find something they just say “You’re free to go!” and you don’t receive a report.
    DOT officers have said” I can find something and fail a Brand new truck.”
    We need to be heard on this SAFER system, it doesn’t work. I have looked at some of the Large O/O companies and their numbers are getting high. Not because of real accidents but because of bad inspections.


Please email: