In July—when The New Yorker ran a long and relatively positive piece about Wikipedia—I argued that the old-media method of laboriously checking each fact was superior to the wiki model, where assertions have to be judged based on their plausibility. I claimed that personal experience as a journalist gave me special insight into such matters, and concluded: “the expensive, arguably old fashioned approach of The New Yorker and other magazines still delivers a level of quality I haven’t found, and do not expect to find, in the world of community-created content.”
Apparently, I was wrong. It turns out that EssJay, one of the Wikipedia users described in The New Yorker article, is not the “tenured professor of religion at a private university” that he claimed he was, and that The New Yorker reported him to be. He’s actually a 24-year-old, sans doctorate, named Ryan Jordan.
Jimmy Wales, who is as close to being in charge of Wikipedia as anybody is, has had an intricate progression of thought on the matter, ably chronicled by Seth Finklestein. His ultimate reaction (or at any rate, his current public stance as of this writing) is on his personal page in Wikipedia
I only learned this morning that EssJay used his false credentials in content disputes… I understood this to be primarily the matter of a pseudonymous identity (something very mild and completely understandable given the personal dangers possible on the Internet) and not a matter of violation of people’s trust.
As Seth points out, this is an odd reaction since it seems simultaneously to forgive EssJay for lying to The New Yorker (“something very mild”) and to hold him much more strongly to account for lying to other Wikipedia users. One could argue that lying to The New Yorker—and by extension to its hundreds of thousands of subscribers—was in the aggregate much worse than lying to the Wikipedians. One could also argue that Mr. Jordan’s appeal to institutional authority, which was as successful as it was dishonest, raises profound questions about the Wikipedia model.
But I won’t make either of those arguments. Instead, I’ll return to the issue that has me putting my foot in my mouth: How can a reader decide what to trust? I predicted you could trust The New Yorker, and as it turns out, you couldn’t.
Philip Tetlock, a long-time student of the human penchant for making predictions, has found (in a book whose text I can’t link to, but which I encourage you to read) that people whose predictions are falsified typically react by making excuses. They typically claim that they are off the hook because the conditions based on which they predicted a certain result were actually not as they seemed at the time of the inaccurate prediction. This defense is available to me: The New Yorker fell short of its own standards, and took EssJay at his word without verifying his identity or even learning his name. He had, as all con men do, a plausible-sounding story, related in this case to a putative fear of professional retribution that in hindsight sits rather uneasily with his claim that he had tenure. If the magazine hadn’t broken its own rules, this wouldn’t have gotten into print.
But that response would be too facile, as Tetlock rightly observes of the general case. Granted that perfect fact checking makes for a trustworthy story; how do you know when the fact checking is perfect and when it is not? You don’t. More generally, predictions are only as good as someone’s ability to figure out whether or not the conditions are right to trigger the predicted outcome.
So what about this case: On the one hand, incidents like this are rare and tend to lead the fact checkers to redouble their meticulousness. On the other, the fact claims in a story that are hardest to check are often for the same reason the likeliest ones to be false. Should you trust the sometimes-imperfect fact checking that actually goes on?
My answer is yes. In the wake of this episode The New Yorker looks very bad (and Wikipedia only moderately so) because people regard an error in The New Yorker to be exceptional in a way the exact same error in Wikipedia is not. This expectations gap tells me that The New Yorker, warts and all, still gives people something they cannot find at Wikipedia: a greater, though conspicuously not total, degree of confidence in what they read.
Nice article. I love the New Yorker, but have noticed that in the latest issue an article cites figures and stats without referring to a source (I should really refer to the page number and article myself!).
Anyhow, a great example of fraudulent journalism can be seen in the movie Shattered Glass, based on a true story about a compulsively lying New Republic journalist.
I’m afraid I haven’t read all the comments; I hope my comment will be read by Mr. Robinson nonetheless. A student in my Professional Editing class posted something on the James Frey incident, and I commented, mentioning, among other things, the apparent decline in The New Yorker’s fact-checking ability. In support of my statement, I cited an old piece (Columbia Journalism Review’s 1994 article, itself fraught with typographical errors); but then found this opinion by David Robinson. I’ll add a link to Mr. Robinson’s article with appreciation for his opinion.
The reason the New Yorker is more trusted is because it is more conspicuous in its mistakes. When an error appears, other people like you take note and call attention. The same is true for Wikipedia, however to a lesser degree so it is not quite as trustworthy.
Both can be wrong, and similarly both can issue a retraction. The New Yorker takes longer to respond and therefore it must be more careful to be reliably correct. Wikipedia in general can respond more quickly, and so as a whole it does not have to be as perfectly accurate.
It’s ironic that the very first comment in the stream above both misrepresents facts (it was Nature, not New Scientist that did the study) and misses the followup discussion in various places that analyzed the study and found it was arguably correct, but that what it actually tested was nothing like a reasonable picture of “trustworthiness” – that the particular things it chose to check skewed the results.
It’s also ironic that none of the commenters who referred to it was able to spell “Encyclopaedia Britannica”.
The wisdom of crowds can be wrong. It can also oscllate between being wrong and being right. Professional fact checking can also be wrong – it’s meant to reduce risk, but you can never eliminate it.
Wikipedia is amazing – it’s breadth and speed are something a printed encyclopedia could never hope to achieve. That doesn’t make it a good substitute for well-written and well-thought-out professional articles; it’s just a different kind of thing.
“This expectations gap tells me that The New Yorker, warts and all, still gives people something they cannot find at Wikipedia: a greater, though conspicuously not total, degree of confidence in what they read.”
With all due respect, new things are never trusted by the majority. When the populace is as familiar with Wikipedia as The New Yorker, then — and only then — might your induction about the meaning of the expectations gap be valid. Unfortunately, Wikipedia still presently falls under the “devil music” curse on progress.
” ‘I only learned this morning that EssJay used his false credentials in content disputes…’ (Jim Wales)”
As has been mentioned, the strength of the Wikipedia isn’t in the credentials of its authors but by its core value that assertions must be backed up by external sources. Further, as this above quote shows, disputes about the veracity (or — more often — neutrality) of presented sourced content are duked out just a tab away — viewable by anyone — where, generally, the majority opinion holds the day. So, to recap: Wikipedia articles aim to always defer to external sources; the choice in sourced content is decided by the majority, for the most part (barring instances where Stephen Colbert tells viewers to change an entry or some such); and these discussions are viewable by anyone wishing to observe or partake in them. That is why the EssJay scandal is in no way near as Earth-shattering as a scandal involving a renowned magazine, journalist, television anchor, etc. We don’t trust the EssJays of the world, and we don’t have to, but we have to take the creators of more traditional news and information at their word.
Trust is almost always based on a risk calculation: what is my risk if I trust this entity, and what is my risk if I refuse to? Reputation may play a part in that calculation–or it may not.
I don’t buy this. Maybe if I’m shopping for a CD player my metric of trust depends on risk calculation. But if I read a factual assertion (say, in Wikipedia,) my trust in that statement is not based on any calculated risk of being wrong.
I don’t even consider the potential repercussions of being mistaken about George Washington’s exact birthdate. Rather, my credulity or skepticism depends on my estimation of the source’s accuracy, and probably the ordinary or extraordinary nature of what is claimed.
This is probably because there is no immediate concrete risk facing me when I decide whether to believe some factoid. You could argue that this would be different if I was a reporter who could suffer in reputation for insufficient fact-checking. But, even if I had such a job, I think I’d still use a lot of the same mental processes for deciding whether to trust a factual statement—especially when the fact in question seems ordinary enough.
Or to put it more succinctly, I think there are different kinds of trust, and trust upon entering a contract and trust in a source of information are different things based on different decision processes.
I doubt if wikipedia was ever meant to be a replacement for New Yorker.
The greatest benefit of Wikipedia, with respect to trust in information, is the fact that it is completely upfront about potentially being wrong. The wiki model FORCES readers to judge all information for themselves, instead of simply having faith that it must be correct simply because it comes from a “trustworthy” source. Traditional media, such as the New Yorker and Encyclopedia Brittanica, are, according to the research mentioned above, no more accurate than Wikipedia. However, Wikipedia does not attempt to disguise the fact that it may contain errors. This honesty is much more important in the pursuit of knowledge and truth than the unattainable goal of perfect accuracy. Even the most trustworthy source is imperfect, so we should all learn to weight the plausibility of every piece of information, no matter the source.
My favourite comment about this sort of thing was simple, and builds on what Jack Lerner said above:
‘Have you ever seen a news article that was in your field of expertise, and were struck by just how fundamentally wrong it was?
What makes you think the rest of the news is any more accurate?’
My impression of news gathering by established print outlets was forever changed when a close friend was involved in a major national news story. In the rush to publish something about this event, major news organizations published bad information, questionable information, and rumors. I realized that reporters get sloppy and that just because something is in print does not mean that every detail of the story is correct. Instead it’s more of a gestalt – the better institutions will tell the overall story and the major facts with some accuracy, but as we all know, even if it’s the Paper of Record we cannot assume that everything in print is the unvarnished truth.
Instead, trust is built on a perceived relationship and history of reliable information.
This is a common misperception, and one that is also relevant to the previous thread about reputation systems. Trust is almost always based on a risk calculation: what is my risk if I trust this entity, and what is my risk if I refuse to? Reputation may play a part in that calculation–or it may not. For example, people routinely purchase goods from Internet merchants they’ve never heard of before and know next to nothing about, because they know their credit card company is legally bound to indemnify them against risk should the merchant be fraudulent. Conversely, they may refuse to lend their life savings to a longtime trusted friend, deciding that the risk is simply too great even given the friend’s absolutely impeccable character.
And just as credit card companies offer indemnification against fraud, professional journalists offer indemnification against embarrassment. Consider, for example, David Robinson’s reaction to the New Yorker/Wikipedia episode. If Robinson’s trust were based strictly on his past experience of discovering inaccuracies, this episode would obviously cause him to trust the New Yorker much less, and Wikipedia (comparatively, at least) much more. But instead, it reaffirms his trust in the New Yorker, “because people regard an error in The New Yorker to be exceptional in a way the exact same error in Wikipedia is not.” In other words, if Robinson himself trusts the New Yorker and his trust turns out to be misplaced, people will consider him the victim of an unlikely error, whereas if he trusts Wikipedia and his trust turns out to be misplaced, people will consider him to have made a poor judgment. Thus, trusting Wikipedia is, for Robinson, a greater personal risk than trusting the New Yorker, regardless of whether the New Yorker is actually more trustworthy than Wikipedia. As in the credit card example, the New Yorker’s puffery about its fact-checking effectively indemnifies Robinson against the risk of looking foolish, in a way that Wikipedia does not (yet, at least).
This sort of indemnification is precisely what professional journalists are defending when they rudely dismiss blogs–or Wikipedia–as inferior. Rather than make serious, careful comparisons between themselves and alternative information sources–comparisons that would inevitably show themselves to be seriously flawed in their own right–they emphasize the process by which they gather and vet their content. They will always treat that process with the utmost solemnity and reverence, without ever being terribly specific about what it consists of. (Too much specificity would destroy the aura of confidence they’re trying to create.) And they will never, ever criticize the process of a competing information source, except when that source dares to contradict the “pack” consensus of professional journalists on some matter of fact–thereby calling into question whether the methods employed by the pack are quite as reliable and professional as the pack claims them to be.
The message behind this elaborate pose is clear: because we (rather ostentatiously) make an enormous effort (of an undisclosed nature) to ensure that your information is accurate, you won’t look stupid if you rely on us. To an information consumer who doesn’t have the time, knowledge or inclination to examine the information they receive critically, this sort of indemnification is well worth the extra price of professionally generated journalism. But, as recent scandals in journalism have demonstrated, it’s very, very far from an actual guarantee of accuracy. And consumers who care about factual accuracy would be well-advised not to treat it as such.
Exhibit A: Fox News.
As opposed to, what? the New York Times, Associated Press, and Reuters?
Sorry to say it, but the media has regressed to the 1890’s yellow journalism. I’m not opposed to a media outlet having a certain editorial point of view, but they should work hard to keep that point of view on the editorial pages, and not on A-1.
The fact checkers at The New Yorker clearly did drop the ball when they took Essjay at his word, just as Jimbo Wales dropped the ball when he failed to vet Essjay prior to appointing him to Wikipedia’s ArbCom. It was a perfect storm of fraud and gullibility, wherein Essjay’s initial deception was bolstered and bootstrapped by his reputation as a trusted Wikipedia contributor, and those who had rational doubts about Essjay’s credentials failed to speak up. Even now, there’s a large contingent of high-ranking Wikipedians with administrative privileges who are loyal to Essjay and are censoring much of the internal criticism of Essjay that would otherwise appear on Wikipedia. This reminds me of the Piltdown Man hoax, which was an obvious fraud that was exposed quite early, but was not properly publicized for decades because so many experts had bought into the lie. At the same time, I cannot fault Jimbo or The New Yorker for being deceived, as they have apologized and taken subsequent remedial measures. What I can and do fault is the culture of credentialism that is so easily exploited by impostors. For a longer tome on my views, please see – http://blog.xodp.org/2007/03/credentialists-and-impostors.html -.
I guess you can only rely on a media organisation as much as you trust their previous work. It would be verging on dangerous to suggest that professional media organisations are trustworthy and accurate simply because the people involved are being paid to present the information.
Exhibit A: Fox News.
Instead, trust is built on a perceived relationship and history of reliable information. If anything, it’s probably easier to trust a single organisation like The New Yorker because you can put names to faces and recollect previous instances of when you’ve believed or trusted their work. Doing the same with all the users of WikiPedia requires a leap of faith which is a little harder to contemplate.
quote:
I claimed that personal experience as a journalist gave me special insight into such matters, and concluded: “the expensive, arguably old fashioned approach of The New Yorker and other magazines still delivers a level of quality I haven’t found, and do not expect to find, in the world of community-created content.â€
Apparently, I was wrong.
quote:
New Scientist did a study on exactly this. They compared Wikipedia to the Encyclopaedia Brittanica. Wikipedia won! iirc.
Actually, I think it was something like “both got four articles right and one wrong”. Where they got it wrong, Brittanica was slightly more wrong than Wikipedia.
So, as regards getting your FACTS right, Wikipedia is actually very good. The problem is that when it’s wrong it can be badly wrong (witness the Professor who complained about a Physics page containing elementary blunders and who gave up trying to get it fixed – he just penalised any of his students who used Wikipedia as a reference), and where the “facts” are controversial you get “edit wars”.
So, at the end of the day, Wikipedia is about as good as print. Where it’s uncontroversial, Wikipedia will be accurate, and where it is controversial, the press is biased and Wikipedia has an edit war.
Cheers,
Wol