Clay Shirky
( Archive | Home )

Liz Lawley
( Archive | Home )

Ross Mayfield
( Archive | Home )

Sébastien Paquet
( Archive | Home )

David Weinberger
( Archive | Home )

danah boyd
( Archive | Home )

Guest Authors
Recent Comments

pet rescue saga cheats level 42 on My book. Let me show you it.

Affenspiele on My book. Let me show you it.

Affenspiele on My book. Let me Amazon show you it.

Donte on My book. Let me show you it.

telecharger subway surfers on My book. Let me show you it.

Ask Fm Anonymous Finder on My book. Let me show you it.

Site Search
Monthly Archives
RSS 1.0
RSS 2.0
In the Pipeline: Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline


« Kayak? | Main | Tags != folksonomies && Tags != Flat name spaces »

January 23, 2005

The Innovator's Lemma

Email This Entry

Posted by Clay Shirky

To respond to David’s question about folksonomies Aren’t we going to innovate our way out of this? My answer is yes, but only for small values of “out.” A big part of what’s coming is accepting and adapting to the mess, instead of exiting it.

Seeing people defend professional classification as a viable option for large systems is giving me horrible flashbacks to the arguments in 1993 about why gopher was superior to the Web. Gopher was categorized by professionals, and it was hierarchical. Gotta love hierarchy for forcing organization. You can’t just stick things any old place — to be able to add something to a hierarchy, you have to say where it goes, and what it goes with, next to, under, and above. And if you do it right, you can even call it an ontology, which means you get to charge extra. (I loves me some ontologies.)

The Web, meanwhile, was chaos. Chaos! You could link anything to anything else! Melvil Dewey would plotz if he saw such a tuml. How on earth could you organize the Web? The task is plainly impossible.

And you know what? The gopher people were right. The Web is chaos, and instead of getting the well-groomed world of gopher, we’ve adapted to the Web by meeting it half way.

Part of the solution has been that brilliant minds keep finding ways of extracting value from the mess, but another big part has been that we’ve adapted to the chaos, including satisficing search strategies, acceptance of partial results, the addition of post hoc and edge-based filters, and so on.

Anything that operates at really large scale takes on the characteristics of organic systems, including especially degeneracy, the principle that there is not a one-to-one mapping between function and location in the system. (Christopher Alexander got there a long time ago, in A City Is Not a Tree, to which we might only add that the Web is not a tree either.)

If we’re going to let just any old person write whatever they want and then make it available globally, we’re going to have to extend the same freedom to classifying the resulting flood of material. And critics will be able to say, rightly, that such a system will lack the coherence we would have had, had we not gone and let everyone publish willy-nilly. To which the only sensible reply, as to the gopher people, is “Oh well.”

The best we can do is figure out the second-order problems, such as lowest common denominator classification, and try to fix them. And this is the innovator’s lemma — classification is a sub-problem of the global publishing phenomenon. We have folksonomies because we have a world where professional classification and controlled vocabularies are as broken as gopher was in 1994. But getting folksonomic classification right isn’t the main event, which is why there’s much less flexibility than professional classifiers imagine.

You can find lots of people to blame for this problem — George Boole, Claude Shannon, Vint Cerf, Tim Berners-Lee, and so on. What’s certain is that the explosion of content that makes folksonomies a forced move was a done deal a long time ago. The librarians are now in the same situation as the journalists: “What will happen when everyone can do what we do, without having to be professionally trained?” And the answer, as always, is “We’ll see.”

Comments (7) + TrackBacks (0) | Category: social software


1. Valdis on January 23, 2005 2:36 PM writes...

Clay writes...
"And you know what? The gopher people were right. The Web is chaos, and instead of getting the well-groomed world of gopher, we’ve adapted to the Web by meeting it half way."

Exactly! And we are good at that... many examples throughout history. Those of us who have worked in large organizations, where everything should fit into a tidy hierarchy, know that usually things get done via messy networks. History has shown a constant dance between hierarchies and networks, and as long as things change, we will see the dance continue into the future. Either extreme does not work for us, but together -- adapting to each other -- hierarchies and networks allow us to move forward.

Permalink to Comment

2. Seyed Razavi on January 23, 2005 3:14 PM writes...

What I'd like to understand is why the chaotic and ordered methods can't both exist on the Net? Or why did Gopher not find a market?

Is there some kind of winner-takes-all principle at work? Are there *gasp* too few eyeballs?

Or is it just entropy reducing every organised system into chaos and we're just pissing in the wind even with folksonomies etc.

If so the disquiet in the soul about technological advancement may about to get much worse.

There's something poetic about all these lovely / smart people working piecemeal in making the world a bigger mess by giving small out pieces of goodness (erm, law of unintended consequences). There's a narrative which I think resonates strongly outside of "our" circles.

Permalink to Comment

3. Rup3rt on January 23, 2005 5:13 PM writes...

Path of least effort, crossing paths

I stopped making gopher menus because it was just as easy to do it in html and html offered embedded pictures. Now html code is getting rarer because it is easier to plug in a (php, perl, java) block or module and tart it up with a style sheet. Voilà! A popularity chart of the latest postings with avatars.

Tagging is more related to a thesaurus than to a rigid hierarchy and I'm sure that'll be the way things get figured out. Whether I choose bike, cycle or bicycle as my tag, the thesaurus is a central decoder. That's where the xml will come from.

Amazon has provided a benchmark/shortcut taxonomy for a while. Librarians spent many hours and spilt
blood over categorisation before there were global/central reference points.

I am interested on tags and language in the way that metadata offers an smaller and more copable target for translations. There is some hope that paths can cross in the near future. Maybe google will soon say 'Do you mean globalização de metadados?'

Permalink to Comment

4. Edward Vielmetti on January 23, 2005 11:16 PM writes...

Gopher lost because of pictures, not because of ontologies. At its peak it was as chaotic and unordered as the web a couple of years later - the only way to really get around the whole thing was to do searches ("veronica", the Gopher equivalent of Google).

Delicious is simple enough that you could build a gopher server on top of the data and it would all make sense. Might even be faster to navigate, though the menus would get kind of long.

Permalink to Comment

5. phil jones on January 24, 2005 1:26 AM writes...

"though the menus would get kind of long."

Which strikes me as the whole point in a nutshell. :-)

Permalink to Comment

6. Jay Fienberg on January 24, 2005 1:44 AM writes...

"If we’re going to let just any old person write whatever they want and then make it available globally, we’re going to have to extend the same freedom to classifying the resulting flood of material."

I totally agree.

But, I would argue that, ironically, the usefulness of the tagging systems in Flickr,, and Technorati is that these systems remove the "same freedom to classifying" already available on the web, and constrain tagging within a more traditionally controlled system.

It's true that the tags can be freely assigned, but this is in a context in which there is a fixed definition (e.g., controlled vocabulary) for "tags", and many pre-determined (even tree-like) constraints around how tags are accessed.

In other words, it's true that these tag systems are innovative and give you freedom on *what* tags you use, but, at the same time, they are mostly quite traditional in terms of *how* you use tags. (Actually, the "browsability" of tags in Flickr,, and Technorati includes some nice innovations--I don't mean to slight those.)

This is unlike the web at large, where the general concepts of links and pages (resources) effectively let any old person classify and/or tag whatever they want, whatever way(s) they want.

So, I disagree with the idea that tagging and folksonomies are succeeding due to the rejection of controlled vocabularies. Rather, like what's worked on the web in general, they are succeeding through the smart uses of simple controlled vocabularies that, although they remove freedom for *how* folks talk about things, they do so to enforce a structure that specifically prevents control of *what* folks want to talk about (within that structure).

Permalink to Comment

7. Yoz on January 24, 2005 10:35 AM writes...

Seyed: Some would say that if you have more than one ordered method (i.e. authoritative classification mechanisms which can conflict, such as Yahoo! Directory vs DMOZ) then you have chaos. I believe that it's this to which Clay is referring.

Jay: Yes, the current structure of folksonomy is only freeform within tightly-defined edges. However, it's worth remembering that the entire existing folksonomy structure is a flimsy web consisting of a handful of implementations (and a fair amount of consensus) on top of the existing structure of the WWW, just like Yahoo!, DMOZ, WebRings and a billion other organisational methods.

One of the beauties of the web is that if one particular style of organisation doesn't suit you, it's very easy to build another. I think this is the main lesson here: not that folksonomies rule (Clay's been consistently saying they don't) but that they're a prime example of the kind of agile, makeshift classification of which we'll be seeing much more.

Permalink to Comment


TrackBack URL:

Listed below are links to weblogs that reference The Innovator's Lemma:


Remember Me?


Email this entry to:

Your email address:

Message (optional):

Spolsky on Blog Comments: Scale matters
"The internet's output is data, but its product is freedom"
Andrew Keen: Rescuing 'Luddite' from the Luddites
knowledge access as a public good
viewing American class divisions through Facebook and MySpace
Gorman, redux: The Siren Song of the Internet
Mis-understanding Fred Wilson's 'Age and Entrepreneurship' argument
The Future Belongs to Those Who Take The Present For Granted: A return to Fred Wilson's "age question"