Yet another smackdown of fMRI lie-detection

I haven't been following this all that closely, but Mind Hacks today linked to this excellent Wired Science article summarizing the reasons that a judge in Tennessee decided to disallow "brain scan lie detection" evidence. The 1993 case Daubert v. Merrell Dow Pharmaceuticals set the current standard for scientific testimony in courtrooms:

Judge Pham, who presided over this evidentiary
, summarized his reading of Daubert: Reasonable tests to
apply and ideas to consider include “(1) whether the theory or technique
can be tested and has been tested; (2) whether the theory or technique
has been subjected to peer review and publication; (3) the known or
potential rate of error of the method used and the existence and
maintenance of standards controlling the technique’s operation; and (4)
whether the theory or method has been generally accepted by the
scientific community.”

The verdict in the Tennessee hearing was that this use of fMRI does not meet the standard in multiple ways. Two of the problems in this particular implementation of imaging-as-evidence are particularly startling, in the sense of "What were you thinking?!"

First up is

. . . the scientific methodology employed by Cephos, the company who conducted
the lie-detection test. After Semrau failed one of the two tests he’d
agreed to take, Cephos CEO Steven Laken retested him a third time,
claiming his client had been tired.

. . .

“It seems almost laughable that Cephos could parade this as a great
method when, in this very case, they tried it three times and got one
result twice and the other one once,” Greely wrote in an e-mail to “In the only ‘real world’ test we’ve got evidence about,
their accuracy rate was either 66.7 percent or 33.3 percent.”

That's right, they ran two tests, and when one of them looked bad for the client, they ran it again to get a "better" result. Gah! This is the sort of thing that every research methods and research ethics and stats class IN THE WORLD tells you not to do, because it COMPLETELY INVALIDATES your data and analysis.


Furthermore, and the judge quoted extensively from the prosecution’s
cross-examination on this point, Cephos only claims to be able to offer a
general impression of whether someone is being deceptive. While they
ask dozens of individual questions, Laken admitted that his company’s
method could not be used to tell whether someone was lying or telling
the truth on any of specific facts.

That is to say, Laken refused to say that Semrau was telling the
truth to a question like, “Did you enter into a scheme to defraud the
government by billing for AIMS tests conducted by psychiatrists under
CPT Code 99301?” but was willing to say that Semrau was “more overall”
telling the truth.

"More overall" telling the truth? So he could be a generally truthful guy.. and still lying through his teeth about the issue of interest to the court, and this test would support only the general truthfulness? Like I said, I hadn't realized just how bad the presentation of this was, and I'm kind of horrified. I hope it stays far far away from the judicial system unless/until we figure out a way to do it right.