Randomness and Fraud

A discussion last night about the Madoff fraud reminded me of this Nate Silver post from 2009 questioning the legitimacy of Strategic Vision's polls.  I found the information fascinating, and wanted to repost it.  This introduction contains a lot of interesting stuff:

One of the things I learned while exploring the statistical proprieties of the Iranian election, the results of which were probably forged, is that human beings are really bad at randomization. Tell a human to come up with a set of random numbers, and they will be surprisingly inept at trying to do so. Most humans, for instance, when asked to flip an imaginary coin and record the results, will succumb to the Gambler's Fallacy and be more likely to record a toss of 'tails' if the last couple of tosses had been heads, or vice versa. This feels right to most of us -- but it isn't. We're actually introducing patterns into what is supposed to be random noise.

Sometimes, as is the case with certain applications of Benford's Law, this characteristic can be used as a fraud-detection mechanism. If, for example, one of your less-trustworthy employees is submitting a series of receipts, and an unusually high number end with the trailing digit '7' ($27, $107, $297, etc.), there is a decent chance that he is falsifying his expenses. The IRS uses techniques like this to detect tax fraud.

Yesterday, I posed several pointed questions to David E. Johnson, the founder of Strategic Vision, LLC, an Atlanta-based PR firm which also occasionally releases political polls. One of the questions, in light of Strategic Vision LLC's repeated failure to disclose even basic details about its polling methodology, is whether the firm is in fact conducting polling at all, or rather, is creating fake but plausible-looking results in order to increase traffic and attention to its core business as a PR and literary firm.

I posed that question largely as a hypothetical yesterday. But today, I pose it much more literally. Certain statistical properties of the results reported by Strategic Vision, LLC suggest, perhaps strongly, the possibility of fraud, although they certainly do not prove it and further investigation will be required.

There are several other good posts from that time frame here.  This one is especially entertaining:

The poll in question comes from the Oklahoma Council of Public Affairs (OCPA), a conservative-leaning thinktank that recently commissioned Strategic Vision, LLC to conduct a poll of 1,000 Oklahoma high school students. (A similar poll has previously been conducted by Strategic Vision, LLC in Arizona). The poll asked ten relatively basic political knowledge questions that were drawn the U.S. Citizenship Test, such as: "How many justices are on the Supreme Court".

Only 2.8 percent of Oklahoma's high school students passed the test, claim OCPA and Strategic Vision, which is defined by having gotten at least 6 of the 10 answers right. Moreover, the results to some particular questions were strikingly low. Ostensibly, only 23 percent of the students correctly identified George Washington as the first President, and only 43 percent correctly named the Democrats and Republicans as the two major political parties (11 percent of the students, COPA and Strategic Vision claim, provided the answer "Communist and Republican").

For me, some of these results don't pass the smell test. I agree that public schooling in the United States needs to be improved, particularly in the areas of government and citizenship. But only 23 percent of high school students in Oklahoma knew that George Washington was the first President? Really? I have difficulty accepting that claim at face value. In 2008, 68 percent of Oklahoma fifth graders passed the Oklahoma Core Curriculum Social Studies Test. You can read some of the questions on that test beginning on page 50 of this PDF; they're generally quite a bit more difficult than the ones that Strategic Vision asks. (For instance, "Which was the most profitable export of the Jamestown settlement?" and "Which group would most likely agree with ideas presented in Common Sense?"). So either those smart fifth graders were really forgetful by the time they got to high school, or there's something very wrong with this poll.