In the Pipeline:
Don't miss Derek Lowe's excellent commentary on drug discovery and the pharma industry in general at In the Pipeline
« Comment Spam Solution |
| Adolescence goes public »
January 13, 2004
I gave a half-hearted answer
to Joi's Are Blogs Just
post, explicitly ignoring the larger philosophical issues he raised from the "Inequality and Fairness" section of Power Laws, Weblogs, and Inequality
That was a cop out. I didn't answer his question because it seems to assume some things about inequality -- that one should take a position for or against it -- that I don't actually believe.
So here's what I do believe: inequality is inevitable, and that being for or against it makes no more sense than being for or against the weather.
Now there are many ways to treat inequality as inevitable -- you can adopt such a posture because you are or have become cynical, worldly wise, passive, or an adherent of realpolitik -- but I have a very particular way in which I believe inequality is inevitable. I believe that wanting large networks without inequality is like wanting mortar without sand. Inequality is not some removable side-effect of networks; inequality is what holds networks together, inequality is core to how networks work.
Networks are deep patterns, but we have often treated them as shallow. Over the last hundred years, we have observed networks at work in a variety of places -- food chains, the spread of gossip, the electrical grid, the connection of neurons -- but we have often regarded those networks as second-order entities, whose behavior is mainly the product of their constituent parts.
This is wrong. Networks are 0th order patterns, _deeper_ than their constituent parts. The way packets moves through the network is similar to the way gossip moves through an office, even though a router is nothing like an office worker. Once you network things -- _any_ things -- the subsequent patterns are more affected by the characteristics of networks than the characteristics of the things the network is made of.
One of the characteristics of networks is a kind of structural inequality that holds things together. Call this a Zipf or Pareto distribution or a power law
or any of the other names it's been given. If a system is large, heterogeneous, and robustly but sparsely connected, it will exhibit power law distributions in its arrangement, and that pattern will be reflected in whatever binds the network together, both statically and dynamically -- link density, popularity, messages sent or received, and so on.
If this hypothesis is correct, there are two workable responses, and one obviously unworkable one. The first workable response is to exit the system by violating one or more of the core conditions.
You can get out of a system with power law distributions by giving up on scale. As Simon St. Laurent points out in Escaping the Googlearchy, one way to avoid the inequality of large systems is not to _have_ large systems. This seems to me to be an incredibly fruitful area of inquiry -- the web made us all size queens for a decade, but now that hardware, system tools, and most importantly programming talent and time are abundant enough, not everything has to scale. I believe we are seeing and will continue to see work on small systems that avoid power law distributions because, as Ross noted long ago in Ecosystem of Networks, a group of 12 is different from a group of 150.
The second way to get out of such systems is to violate heterogeneity. The US Army has systems that don't exhibit power law distributions of links in the command structure, because they control all the units. They can achieve hierarchical coordination by avoiding heterogeneity.
The third way to get out is to drop the idea that the network needs to be robust. If you do not require a network with the six degrees of separation principle, and can tolerate lots of cases where the nodes simply can't reach each other, you have much greater freedom in topological models. This is what Gnutella was like in its earliest incarnation, before the advent of super-peers.
However, simply escaping systems that happen to have power law distributions is a really lousy solution for all sorts of environments we like to be a part of. This trio of characteristics -- large, heterogeneous, and robust but sparse -- is the one that allows us to have, say, billions of people separated by 6 degrees, or an internet where any two machines can reach one another in fewer (usually quite a bit fewer) than 30 hops.
This brings up the other workable response to this sort of inequality -- accept it, but change its terms. Once you accept that there will be a power law distribution, instead of fighting it, you can concentrate on modifying it. There are several possible strategies here as well.
You can tax the system, moving more of X (whatever is being ranked) from where it is abundant to where it is scarce. This is how progressive taxation works, transferring money from those with most to those with least. The result is still a power law distribution (the economist Vilfredo Pareto called the distribution "a predictable imbalance" and found it in all market economies), its just got a shallower slope, and an average that's further from the #1 position and closer to the median.
You can limit the head of the curve. Sometimes this happens naturally -- even Joi Ito falls short of the number of connections required for a raw power law distribution on LinkedIn, while over at Friendster, they have placed an artificial cap at 200 links.
You can truncate the tail, periodically dumping those links that are below median or some other threshold. Brucker-Cohen's BumpList is an example of auto-truncation, though at a scale much too small to demonstrate power laws. I don't know of any systems that use auto-truncation at large scale.
You can also try to make the system more dynamic, by making it possible for new nodes to get attention. This is what Sifry is up to with his Interesting Newcomers list. These are not weblogs with a high _number_ of inbound links, but rather high _growth_.
All of these responses leave the power law intact, just altered.
The unworkable response is to assume you can destroy the power law distribution without also destroying (or at least altering beyond recognition) the factors in which caused it to arise in the first place -- size, diversity, connectedness.
Last summer, a study came out describing the characteristics of the conservative mindset, which included, among other things, a tolerance for inequality. By this definition, I believe we will all be conservatives soon. Evidence that inequality is a core aspect of our large systems, is in many ways the signature of those systems in fact, will make utopian declarations of being 'against' inequality an impossibility for anyone who regards reality as a constraint on their world view.
We can and should talk about the type of inequality we want -- right now, for example, most of the high-flow webloggers are men. We can ask why that is, whether we should do anything about it, and if so, what? We _can't_ ask how we can level out the difference between the high-flow end of the popularity curve and the rest of us, or at least we can't ask that unless we are advocating the destruction of the blogosphere. The interesting and hard question is "Since there is to be inequality, how shall it be arranged?"
I think we are going to see an explosion in work designed to alter the construction and effects of this inevitable inequality (viz Sifry's experiments on moving recent blogs up the Technorati list) and I am optimistic about this change, as I believe the concentration of real thought and energy on what is actually possible, as opposed to cycles wasted on utopian declations, will be tremendously productive.
+ TrackBacks (0) | Category: social software
- RELATED ENTRIES
- Spolsky on Blog Comments: Scale matters
- "The internet's output is data, but its product is freedom"
- Andrew Keen: Rescuing 'Luddite' from the Luddites
- knowledge access as a public good
- viewing American class divisions through Facebook and MySpace
- Gorman, redux: The Siren Song of the Internet
- Mis-understanding Fred Wilson's 'Age and Entrepreneurship' argument
- The Future Belongs to Those Who Take The Present For Granted: A return to Fred Wilson's "age question"