Clay Shirky
( Archive | Home )

Liz Lawley
( Archive | Home )

Ross Mayfield
( Archive | Home )

Sébastien Paquet
( Archive | Home )

David Weinberger
( Archive | Home )

danah boyd
( Archive | Home )

Guest Authors
Recent Comments

pet rescue saga cheats level 42 on My book. Let me show you it.

Affenspiele on My book. Let me show you it.

Affenspiele on My book. Let me Amazon show you it.

Donte on My book. Let me show you it.

telecharger subway surfers on My book. Let me show you it.

Ask Fm Anonymous Finder on My book. Let me show you it.

Site Search
Monthly Archives
RSS 1.0
RSS 2.0
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline


« Social Capital and Income | Main | Fired From Friendster for Blogging »

August 29, 2004

Wikipedia Reputation and the Wemedia Project

Email This Entry

Posted by Ross Mayfield

The core issue of collaborative editing, that of accuracy and trust, has reached a point in debate where research is needed to advance the practice of content use and development. Hiawatha Bray of the Boston Globle offered a Wikipedia criticism in July, calling it One great source — if you can trust it:

For it lacks one vital feature of the traditional encyclopedia: accountability. Old-school reference books hire expert scholars to write their articles, and employ skilled editors to check and double-check their work. Wikipedia’s articles are written by anyone who fancies himself an expert…

“I think it’s exactly the right price,” said Michael Ross, senior vice president of corporate development at Encyclopaedia Britannica Inc. in Chicago. Major articles in Britannica are signed by the author; all articles are vetted by an experienced team of editors and scholars. The libraries that pay $1,500 for a set of bound volumes or the family that pays $60 a year for an Internet subscription are buying confidence as well as information. … Ross admits to reading and enjoying Wikipedia, and has even gotten ideas there for future Britannica articles. But the absence of traditional editorial controls makes Wikipedia unsuited to serious research. “How do they know it’s accurate?” Ross asks. “People can put down anything.”..

In 2002, Wikipedia was criticized because it couldn’t scale and have in-depth articles. Turns out that more was put down than expected, surpassing the Britannica.

Hiawatha raised a key issue, that of quality and reputation, and his piece highlighted Wikipedia’s ambition to publish a first print version. Coupling emergent content development and formal editorial process is a very competitive business model for print. But if the public learns to use and trust the content that emerges in Wikipedia as an authority, it is even more disruptive.

This week Al Fasoldt, a Post-Standard Columnist in Syracuse NY claimed Wikipedia is untrustworthy, based upon an interview with a high school librarian:

“As a high school librarian, part of my job is to help my students develop critical thinking skills,” Stagnitta wrote. “One of these skills is to evaluate the authority of any information source. The Wikipedia is not an authoritative source. It even states this in their disclaimer on their Web site.”

Wikipedia, she explains, takes the idea of open source one step too far for most of us.

Mike from Techdirt takes the columnist to task for misunderstanding Wikipedia:

What’s most amusing about this fear mongering piece concerning Wikipedia is that the librarian in question claims that she uses Wikipedia as an example of an “untrustworthy” site in trying to teach students to develop critical thinking skills. If that’s true, she’s doing a dreadful job. If they really wanted critical thinking skills, shouldn’t they do more than trust this uninformed librarian, but do a little research about Wikipedia itself, how it works, and how the power of Wikipedia is the fact that it is edited — but by anyone else using Wikipedia? There’s just something that seems to freak people out about Wikipedia, when they can’t fathom the idea that “the masses” could produce something of value by simply being able to correct each other, allowing them to build something much more beneficial and much more useful than an expensive encyclopedia edited by just a few people.

Mike took another step of contacting the reporter, and the exchange led him to ask, whom do you trust, the wiki or the reporter?

The quality of Wikipedia Articles, at the very least, at a moment in time are better than they were before and will improve over time. Mike offered a Techdirt Challenge: I pointed to the Wikipedia page on Syracuse, NY where he apparently lives, and suggested he change something on the page, to make it provably, factually incorrect — and see how long it lasted. Alex Halavais, for one, is taking the Challenge. While the results of the challenge (update: 13/13)will provide some valuable insight, it lacks an untampered collection methodology and introduces unfair costs to the system itself.

Joi Ito rightly condemns Mr. Fasoldt’s assertion and views this issue as traditional vs. collective authority:

In fact, on very heated topics, you can see the back and forth negotiation of wordings by people with different views on a topic until, in many cases, a neutral and mutually agreeable wording is put in place and all parties are satisfied. Traditional authority is gained through a combination of talent, hard work and politics. Wikipedia and many open source projects gain their authority through the collective scrutiny of thousands of people. Although it depends a bit on the field, the question is whether something is more likely to be true coming from a source whose resume sounds authoritative or a source that has been viewed by hundreds of thousands of people (with the ability to comment) and has survived.

Shelley Powers delves into the issue of truth and authority:

The reason, according to those with more modern views, is though the authors could be considered ‘authorities’ on the topic, they don’t have the ‘truth’ because the truth, in this instance, is held by those who have new, and fresh insight into the existing material–they have reached an epiphany the others, weighed down by the mass of research material and outdated ideas, can’t hope to achieve.

According to these blessed with such insight, they have truth without authority, while the historians have authority, but can’t possibly understand the truth. Who you trust then, depends less on authority or even truth than it does on who you want to believe–literally whose interpretation rings your bell the most.

The Manifesto for the Reputation Society describes Wikipedia as reputation for the community as a whole by helping to create a public good where there is more flexibility as reputation and other motivations substitute for direct reciprocity. As the Manifesto hints, Wikipedia is considering codification:

An item of debate within the Wikipedia community is the degree to which contributors should acquire some form of reputation, which might then be used to make their contributions to the encyclopedia harder to modify. Letting reputation of contributors emerge in a transparent manner will reward higher–quality contributions, and may provide a partial answer to coordination problems if those who make good contributions receive some proportionate ability to decide conflicts. However, the contrary point of view argues that it is the very openness of Wikipedia that made it a success. One suggestion that balances both points of view is to keep the full Wikipedia open, but to use a reputation system to highlight entries that will be periodically copied into an unmodifiable backup; more ideas can be found in the online discussion of a Wikipedia approval mechanism (WikiApproval, 2004).

Which brings me to an lingering thought — that explicitly codifying reputation introduces a cost which can constrain commons-based peer production. Wikipedia was never supposed to work, somehow does because of good club theory and transaction costs, and has gained a reputation as a resource. Introducing reputation for contributors or articles is the greatest risk to the Wikipedia community. Getting a base study on factual accuracy can help inform this decision as well as educate the public on how to use and participate with this commons resource.

I’ve been quitely forming a group of journalism schools, media centers and experts to engage in the Wemedia Project, which begins with a formal Wikipedia Article fact checking excercise and publishing findings. The USC Annenberg Center has already announced their support and next month we will begin the collaborative research process within a Socialtext Workspace. Without getting into defining truth, you can separate issue of fact, value or policy. The approach is to apply a formal fact checking process to a sample of articles to gain a baseline measure of factual accuracy and explore issues of reputation.

More to come, suggestions appreciated.

Comments (16) + TrackBacks (0) | Category: social software


1. Lion Kimbro on August 29, 2004 9:04 PM writes...

One of the reasons Wikipedia is a lot better than Britannica is because it's written in Plain Talk.

Britannica doesn't stoop so low, if I remember reading my old home set. But Wikipedia authors are generally quite willing to step so high.

Permalink to Comment

2. Alex Halavais on August 29, 2004 10:20 PM writes...

The challenge was met, and within a couple of hours, each of the 13 "bad" changes I made to wikipedia aricles was rolled back by watchful community members, including the one to the Syracuse page.

Permalink to Comment

3. Jill on August 30, 2004 3:19 AM writes...

Ross, thanks for the excellent collection of quotes - I think the idea of a research project investigating how the Wikipedia works is wonderful. Good luck.

But reading about this "raging discussion" I find myself a little disheartened. Every blogger for miles around seems to be furiously typing defences for the Wikipedia, referring to this horrendous jouranlist who thinks the Wikipedia is unauthoritative. The article is silly, certainly, and if you read about his subsequent email discussions with Mike at Techdirt the journalist looks willfully stupid and not simply ignorant as the article makes him seem.

But why do we bloggers kick up SUCH a fuss every time some ignorant, smalltime mainstream media person writes something negative about weblogs or wikis or the web? Reading the posts you'd think it was a war, when in fact EVERYONE's supporting the Wikipedia except this rather insignificant journalist.

I suppose at least it gets us thinking. And perhaps we do need to hone our argumentative skills in order to explain these news ways of thinking and writing to outsiders. But don't you sometimes think we might be fighting windmills?

Permalink to Comment

4. Zbigniew Lukasiak on August 30, 2004 3:41 AM writes...

There still is a proble with treating Wikipedia as authoratitative source. All the arguments listed here only prove that it can be authoratitative in the limes. When, in some online discussion, you qoute something from Wikipedia all you show is that at a given time the text at Wikipedia was what you quoted, but in fact you could change the text from Wikipedia by yourself just to support your thesis.

I think this at once proposes a way how to make Wikipedia quotes more authoratitative - you need to include with each quotation the time how long it was there. It could be made more efficient if the modification time was itself in the link for example or on some prominent place on the page.

Permalink to Comment

5. Markus Molera on August 30, 2004 3:50 AM writes...

When writing papers for university, I often look up the info in Wikipedia, and then, if I find it useful, I look it up in an "official" encyclopedia so I can quote it.

Permalink to Comment

6. dpbsmith on August 30, 2004 8:56 AM writes...

Oh, great, just what Wikipedia needs.... reporters "testing" Wikipedia by inserting misstatements into it. Let's hope other reporters trust his results and don't insist on repeating the experiment themselves. Wikipedia has survived a lot of vandalism, and it will survive this, but... shame on Mr. Halavais.

Would he spray-painting graffiti onto a school building as a way of discovering whether the school has janitors?

Permalink to Comment

7. Sheldon Rampton on August 30, 2004 12:07 PM writes...

In fairness to Al Halavais, he was merely carrying out an experiment that had been proposed by a SUPPORTER of Wikipedia. I agree that it's not a good idea to propose this sort of experiment in the first place, but Halavais wasn't acting in bad faith, and he didn't do any harm.

Of course we shouldn't be encouraging vandalism, even as an "experiment," but I've noticed that a fair number of wiki newbies do this experiment themselves. Many people can't really believe that a website of any substance would allow them to simply change things, so the first edit that many newbies make is to stick in a word or two, e.g., "Test" or "Can I really do this?" Sometimes they'll do this sort of edit and then immediately undo it; others do the edit and then leave, and someone else ends up fixing it.

I don't regard this sort of behavior as "vandalism." It's a minor annoyance, but it's manageable and inevitable and part of the learning process for people who are just encountering a wiki for the first time.

Permalink to Comment

8. Alex Halavais on August 30, 2004 2:41 PM writes...

In my own defense...

I am aware of the damage (in terms of potential time and lost reputation) that such an experiment can inflict, but Mr. Smith's likening it to spray-painted graffiti is misleading. Thousands of people every day are writing (in chalk, not paint) on Wikipedia. Some of these submissions are willfully wrong or misleading, others are unwittingly so, and most are helpful, unbiased, and correct. I recognized that by *temporarily* inserting incorrect information into Wikipedia I was doing a small amount of damage.

I weighed this against the damage that could be done if either (a) the report in question became a larger critique of Wikipedia and open publishing, or (b) the reporter was correct and it was easy to place false information into Wikipedia.

Neither of these are correct: my errors were all corrected within 3 hours. The reason I did this publicly is so that those who are questioning the system need *not* put it through a similar test themselves.

I am sorry for the time wasted by those who found my errors and rolled back to the previous revision. I hope that they see the value, however, in demonstrating the claim that Wikipedia as a system is capable of detecting and eliminating many errors.

I think Ross's effort at "fact-checking" the encyclopedia is a very worthwhile one, and in conjunction with a clear indication of how *changes* in the encyclopedia are handled, will help to demonstrate the effectiveness of large-scale collaborative writing.

Permalink to Comment

9. Box of Puppies on August 31, 2004 11:31 AM writes...

Wikis suck because they lack clarity, completeness, correctness, and consistency. Because uninteresting topics don't get updated very often, Wikis also aren't current . Wiki advocates actually want these qualities because they encourage participation. (I'd provide a link to their arguments but their sites change on a whim.)

Wikis also contain contradictory, duplicate, and off-topic (spam, Search Engine Optimization attempts, etc.) information. Attempts to address these shortcoming lead to internecine arguments called edit wars. It's almost good enough for an episode of reality tv!

The Boston Globe focused on "authority" but this is only a partial argument. While facts are facts, how about the issue of giving credit to your sources? *That* would have made their argument more interesting.

Permalink to Comment

10. Dan Smith on August 31, 2004 1:16 PM writes...

Halavais: "Thousands of people every day are writing (in chalk, not paint) on Wikipedia."

Right. Overstated metaphor on my part.

"Some of these submissions are willfully wrong or misleading, others are unwittingly so, and most are helpful, unbiased, and correct. I recognized that by *temporarily* inserting incorrect information into Wikipedia I was doing a small amount of damage."

Again, right, both on "damage" and on "small amount."


"Many people can't really believe that a website of any substance would allow them to simply change things, so the first edit that many newbies make is to stick in a word or two, e.g., "Test" or "Can I really do this?" Sometimes they'll do this sort of edit and then immediately undo it; others do the edit and then leave, and someone else ends up fixing it. I don't regard this sort of behavior as "vandalism."

Neither do Wikipedians. The Wikipedian phrase for this behavior is "newbie test," and we shrug it off. Such changes are easy to spot, and get reverted promptly, and nobody really minds them.

I phrased my comment poorly. What Al Halavais did is not what Wikipedians call "vandalism," either.

The behavior we call "vandalism" is the deliberate, continuous performance of destructive edits for (what we perceive as being) the _intention_ of damaging Wikipedia. If you take a look at Wikipedia's "vandalism in progress" page you'll be stunned by how much of it there is. When I said "Wikipedia has survived a lot of vandalism," what I meant was "Wikipedia has survived a lot of systematic, continuous, deliberate, large-scale vandalism, and it will survive a dozen sidewalk chark-marks performed by Alex Halavais."

Permalink to Comment

11. Mark Federman on August 31, 2004 7:25 PM writes...

Jill's comment, "But don't you sometimes think we might be fighting windmills?" may be heartfelt, but isn't quite correct in my perception. In fact, I think it's quite the opposite.

Conventional mass-media forms (not to mention conventional business models thereof) are fighting against a wind. And although I am highly reluctant to accede to the cliché addition of "of change," it is nonetheless true that the reversals are kicking in, and conventional authority and expertise and objectivity are routinely being obsolesced.

Bloggers are obsolescing journalists, not because bloggers are better at investigating the story, but because they provide an emergent transparency that used to be the purview of the capital-P Press. Wikipedia is obsolescing more authoritative tomes, not because the wiki is better researched, but because it draws from direct experience. And so on, for most of the intellectual-proprietary world.

The force of the new (media) wind against which the older forms are the mills is considerable. The older forms can spin and spin, but ultimately, they are rooted to the spot. The wind blows right past them, unimpeded by their presence.

Permalink to Comment

12. Frank Paynter on August 31, 2004 10:54 PM writes...

No serious researcher sole-sources his research. Both Britannica and Wikipedia can be expected to reflect biases. Inaccuracies will creep into any compendium of facts. Online resources like Wikipedia are more easily corrected than print publications like Britannica. The errata sheet is by its nature a static document amending a static document. The Wikipedia format permits a researcher to follow the development of content and the editorial consensus building that takes place as content is normalized. Wikipedia can provide a useful starting point for real research, and it can provide a quick background in unfamiliar terrain. It's a useful tool.

The more general debate about Wikis is a debate about human nature and is rife with assumptions about motives and opportunities. The real crime is that nobody has run a longitudinal study of the marginal utility of Wikis to members/users that examines a statistically significant population across a meaningful time span. Somebody oughta, somebody oughta...

Permalink to Comment

13. Luke Rosenberger on September 1, 2004 12:59 PM writes...

Ross proposes a baseline study on factual accuracy of Wikipedia, which sounds like a worthy pursuit. You may also wish to consider the study "Wikipedia as Participatory Journalism: Reliable Sources? Metrics for evaluating collaborative media as a news resource" [304 KB PDF] by Andrew Lih of Hong Kong University presented at the 2004 International Symposium on Online Journalism at the University of Texas at Austin this past April. Lih's metrics that are focused more on authority, not accuracy, but it's very apropos to the current discussion and could serve as a helpful starting point for additional research.

[I tried doing trackback to my blog entry but I couldn't find the trackback URL here. I must be overlooking something.]

Permalink to Comment

14. Ross Mayfield on September 1, 2004 8:35 PM writes...

What a great discussion, thanks.

Luke, Andrew Lih is one of the participants.

Permalink to Comment

15. John Tobler on September 2, 2004 4:06 AM writes...

This controversy about the Wikipedia reminds me very much of an analogous project on the Web: the "Open Directory Project," still out there at Unique at the time, the ODP permitted volunteers to sign up as editors with individual, or sometimes joint, responsibility over categories of knowledge within the Open Directory itself.

I was one of those volunteer editors; my mark is still there, in the content, although any connection with my name has been certainly erased (as it well should be).

What we built is now used by others, most notably Google. Google's hierarchical directory uses the ODP as its starting point but modifies it to suit its percieved sense of the needs of Google users. The lasting victory of the original ODP concept is a tribute to the idea of working openly and together for the benefit of human knowledge. We did it with *human* editors, not just algorithms and machines.

A meritocracy emerged over time. Great effort was made to keep contributions to the ODP within certain boundaries. The Netscape employees and others who were most responsible, tried very hard to educate volunteer editors about such arcanities as ontology and categorization theory. A sort of peer-enforcement evolved that allowed the volunteer community to self-police the system, eliminating the bogus "contributions" of self-serving, and often profit-making, induhviduals who sought to corrupt the directory for their own sometimes malicious purposes.

Sigh. Inevitably, the meritocracy became competitive in nature and certain people who managed to insert themselves fairly close to the root of the tree got into playing power games. Some lorded it over other editors who, perhaps because they held full time jobs, were not able to devote their entire lives to the project. People, and I must reveal that this included me, got dissed for not making enough edits within a certain arbitrary time period. And then some of this now middle-layer power group got the power to remove editors who did not meet their quantity standards.

Somewhere along the line, the ODP seemed to forget that the directory was built by *volunteer* editors, people who often cared deeply about the subject matter in the categories they served. The management of the ODP became institutionalized.

I hope that Wikipedia does not follow that part of the ODP model. In the early days, we created something of great worth, we ordinary people who suggested new categories and who researched and added links -- and who sometimes had to recommend that certain links not be admitted (removal subject to a review process). In the later days, I think the project went off track, a victim of its own success. I still proudly point to URLs in email, my weblogs, and other communications. I still think it was worth doing.

I wish all success to the Wikipedia. Let everyone contribute in the true spirit of a Wiki. Think of this as a very long-term development. Time and generations of unknown editors will refine most of whatever seriously flawed content gets entered. The chaff will fall away.

The important point of focus, to me, is that the Wikipedia is a project truly worth doing. It is worthy of our time and contributions, whether as author, editor, reviewer, whatever.

Much material approved and recommended by librarians contains deceptions, outright lies, and simple untruths of varying degrees of severity. Many librarian-promoted volumes contain pronouncements that have been turned into lies by time; other ideas once considered as lies have, on the contrary, been elevated by time into self-evident truths.

Any intelligent person should know, viscerally, that one always must question what one reads, no matter by whom it was written. That fundamental concept has been around longer than any of us who now debate about the future of the Wikipedia. Since when does a true scholar ever quote anything without checking and cross-checking and cross-cross-checking its validity and integrity?

In summary, I say, "Give the Wikipedia a chance!" Let caring volunteers guard its ideals. Let time be its editor-in-chief. We do not need a bunch of control freaks to impose their rigid sense of pecking-order and status-ranking on a community goodwill process.

And never forget: it's really about volunteers!

Permalink to Comment

16. Stbalbach on September 7, 2004 5:15 PM writes...

I contribute Middle Ages history articles including for example "Norman Conquest" .. After 3 years of various user contributions I had to re-write almost the entire article from scratch top to bottom. Why? Because like many articles, over time it becomes a giant hair ball. There is a bunch of facts and sentences strung together with various degrees of specificity. For example there might be one sentence on an important point, while 2 paragraphs on an insignifigant side-show that probably belongs in a diffrent article. What is missing is someone who can pull it all together and highlite the origins and why it is significant and the important elements.. to educate the reader. Many of the articles are of little value because they are akin to 19th Century Encyclopedias.. a long list of facts added by many people with no analysis or meaning or background or glue. Most people will not have the ability to do this because it requires a real education on the subject, but most people can add one or two sentences here and there. And even when someone like myself with the background who does re-write and provide meaning and signifigance to an event, pretty soon other people come in and start adding irrelevant facts (factually correct though). Hair ball effect. There is somthing to be said for a single authors voice when it comes to history who can filter out the noise and focus on a theme, history is an art in that way. With that said, if people like myself keep contributing it might be all that is needed and Wikipedia wins the day. However, there is a lot of work to be done, most of the history articles are very poor, or missing entirely, as it stands my guess it at the current rate it would take another 5 or 10 years in that area alone. Still, it's a great resource to use to quickly look up a name or place when reading other history sources.

Permalink to Comment


TrackBack URL:

Listed below are links to weblogs that reference Wikipedia Reputation and the Wemedia Project:


Email this entry to:

Your email address:

Message (optional):

Spolsky on Blog Comments: Scale matters
"The internet's output is data, but its product is freedom"
Andrew Keen: Rescuing 'Luddite' from the Luddites
knowledge access as a public good
viewing American class divisions through Facebook and MySpace
Gorman, redux: The Siren Song of the Internet
Mis-understanding Fred Wilson's 'Age and Entrepreneurship' argument
The Future Belongs to Those Who Take The Present For Granted: A return to Fred Wilson's "age question"