What is the IRT?

What is the IRT?

Henry E. Seaton

IRT stands for Item Response Theory. It is the FMCSA’s acronym for a new numbers-crunching proposal which would allow the Agency to develop a new system to process and publish new algorithm driven scores for motor carriers.

An excellent article on the IRT model written by Nell Sedransk, PhD will be published soon in the Journal of Transportation Management. Critics of CSA SMS methodology, of which I am one, point out that the NAS in recommending the IRT method noted that it did not address data sufficiency or accuracy issues which have plagued SMS methodology and led to its removal.

Under the Daubert standard used for admission of studies and analyses in court, data accuracy issues must be addressed. Also, OMB requires the FMCSA to follow the Data Quality Act which places restraints on Agency publication of findings which cannot be supported as true or accurate.

Although the NAS suggests inclusion of a number of invasive and irrelevant data issues like driver pay, driver turnover, and method of pay, the IRT is left with basically the same roadside data which has been roundly criticized.

The IRT, when using SMS data, is faced with the same junk-in-junk-out issue which led the Agency to present and then withdraw the safety fitness determination rulemaking that would have identified based on data a mere 252 carriers for an unsat, the majority of whom had no relevant crashes based on the Agency’s own data.

One must ask, how can the Agency propose to use the same flawed and insufficient data and expect a different result?

Moreover, there is a new significant and looming issue with the misuse of algorithms and mathematical profiling which the FMCSA’s pivot to the IRT presents. In an article entitled, “Our Software is Biased Too” it is noted that “Data scientists and civil rights groups are raising the alarm about algorithms that determine everything from who goes to jail to how much your insurance will cost.” See Wall Street Journal (March 23,-24, 2019) at B4.

The author concludes, “No matter how much we know about algorithms making them ‘fair’ may be impossible.” Clearly, the publication of algorithm-generated carrier scores to determine which carriers go to “safety jail” and which ones are subject to crippling insurance costs presents a systemic fairness issue because of the data sufficiency issues for small carriers.