In the comments to my post on Zagat’s, Grimes, and their respective judgments about the restaurant Grocery, Ernie calls me out, correctly, saying:
Clay, although you make your point, it is not always true that one person’s judgement is inferior to that of a mob.
This is right, and I overstated the case earlier. Expertise is not going away, to be replaced by collaborative filtering in all cases. Here's what is happening: experts are being exposed to previously buried contradictions in their jobs.
What William Grimes, the NY Times restaurant reviewer, spends most of his time doing is making refined distinctions — “Whereas the venison carpaccio on the bed of braised fennel was uncomfortably insouciant, the Purina® Hungry Puppy® meatloaf with the sterno reduction was surprisingly flavorful.”
What Grimes is _paid_ to do, however, is make coarse comparisons — This restaurant is better than that restaurant — and recommendations — You should/shouldn’t eat here.
There is a tension between the desires of the writers and readers. Imagine that Grimes went to his editors and suggested that the Times should suspend the star rating system, as it was doing violence to the subtleties of his analysis. He would be laughed out of the room. Though he spends his time on the subtleties, his paycheck depends on his making recommendations.
Grimes can tell himself that his readers care more about great chefs struggling to advance the culinary arts, even if they fail, than finding a restaurant that is consistently a pleasure to eat at, even if they succeed. This is an easy stance to take, since the people whose opinions matter most — the diners — are too dispersed to compare notes or to contradict him. Grimes becomes a proxy diner, and we take his opinions as guides to our own, because we can get to his but not to ours, at least not in aggregatge.
Add collaborative filtering to that situation, though, and things change. The split between refined distinctions and coarse comparisons becomes manifest if the readers have a way of pooling their collective judgment. This is a profound threat to Grimes, and by extension to everyone who lives with a split between taste and advice. The movie critic wants to be known for subtle analyses of the state of cinema, while the reader wants to know whether to see Brideshead Revisited or Booty Call this weekend. So long as the reader gets a recommendation, the writer can layer on interpretation.
The important story from the Grimes re-review is that he had to schlep back out to Brooklyn, because he cannot afford for his function as recommender to be stripped out by collaborative filtering. Zagat’s will never replace his ability to make subtle distinctions, but no one would read reviews that only made those distinctions, without also offering advice.
On the other hand, he doesn’t want to be seen to be priveleging the value of collaborative filtering, so he writes in defensive mode. On the _other_ other hand, he can't dis Zagat's too openly, since their reviewers are his readers, and to condemn their tastelessness would be a Career Limiting Move.
It is exactly this tension —- competition not from another critic, at a lesser paper, but from his own readers -- that gives his review its schizophrenic character. “I went back, and the Zagat’s readers were right, but I was right too, and I’m still right now, and it was good before and now its better, though that's not to say it was bad before, because it was good, though it lacks the haughty grandeur of _La Salope en Papillote_ over on Park and 63rd...”
Given that he said Grocery was better when he recently visited, imagine the backflips he would have had to go through if the Times had asked him to formally re-rate it, relative to his earlier one star review. Giving it more stars would condemn his own judgment, while giving it no more stars would condemn the judgment of his readers. Lose-lose.
Criticism is a bundled service. Collaborative filtering unbundles the direct recommendation piece, and put its in the hands of the people who care most about such recommendations — us.
1. Abe on October 23, 2003 5:16 PM writes...
Interesting stuff, going to be interesting to see if the Times rerates Grocery anytime in next year or two. Not really sure the frequency between ratings...
One thing left out of all this though seems to be trust. Collaborative filtering generates an average, a valuable peice of information, but limited. A reviewer has an identifiable taste and style. That taste might line up with a reader perfectly, or might be quite different. If you know your taste is quite similar to Grimes, his reviews are extremely valuable. You can trust him. On the other hand a reviewer who is drawn toward the meat dishes is worth very little to a vegetarian.
One of the best spaces to see this distinction is music reviewing. An average rating of a musician is not always of much help. But if you know a critic has taste in line with yours then you are set. So personally if a Sasha Frere-Jones recommends something I've never heard of, I might hunt it down cause I like his ear. Jon Pareles how is more hit or miss. His words might draw my attention, but his recommendation alone is not worth much to me.
Can we ever create algorithms that can compare with a trusted critic? I'm not sure. But an untrusted critic? We have them already it seems...
Permalink to Comment2. Lucas Fletcher on October 23, 2003 5:21 PM writes...
Collaborative filtering is not the correct term here. It refers to personalized recommendations based upon previous personal ratings, and not simply mean-value ratings. But it is interesting that you bring it up.
The whole argument of taste, which is what this case essentially boils down to (literally), is just that, a matter of taste. A built-in assumption when ones tries to argue it is that one can, for example, educate a pallete. This is of course possible, but the understanding must be that that there is a a common vocabulary or shared context between reader and writer. A paper like the New York Times cannot usually afford to make this assumption, since it is mass media. Either can Zagat's for that matter, but it realizes this so it simply uses mean-value ratings of its readers. Both are on average wildly innacurate for the common reader because there is limited context.
True collaborative filtering however implicitly establishes context by utilizing the fact that refined distinctions can be infered from coarse comparisons. For example even basic CF algorithms infer that if you like cheese-sauce over brocolli and stir-fried brocolli but you don't like cheese-sauce over turnips then you most likely enjoy cheese-sauce over brocolli because of the brocolli. This is a crude example, but with lots of data much subtler distinctions can be made.
Permalink to Comment3. Francis Hwang on October 24, 2003 10:13 AM writes...
To follow up on Lucas' point, Zagat's isn't tuned to anyone's particular tastes, and if you start to pay attention to the subject at hand (cuisine, in this case) you'll notice stark differences between your own opinion and that of the middle ground.
New Yorkers may have a lot of cuisine at their hands but, like residents of any other cities, their expertise differs according to type. The city is a great place to sample and learn about Italian cuisine or sushi, but a terrible place to eat southeast Asian cuisine. (I have yet to find a good cheap Thai restaurant; I think I'm spoiled from growing up in a city with a lot of southeast Asian immigrants.) Recommendations from Zagat's about Thai food are basically useless to me; similarly, so are recommendations from friends of mine who've grown up in New York. My best bet, I suppose, is to dig deep into the city's foodie subculture to take care of this ...
Permalink to Comment4. Ernle on October 24, 2003 10:19 AM writes...
I read somewhere - in Forbes or maybe some Gladwell
Permalink to Commentwriting - that finance professors like to run the
following demonstration (even if it wasn't Gladwell,
it's the sort of anecdote he loves to relate).
A jar of jellybeans is sitting on a table at the
front of the class, and each student is asked to
guess the number of beans in the jar. Then the
professor takes the average of all the guesses and
reveals the exact number. It turns out that the
average is closer to the exact number than any
individual guess. This "proves" that the market
is efficient. And of course (since we live in
the golden age of metaphor) this proves that in
the marketplace of ideas, the average of many
opionions must be closer to the real value than
any expert. QED (in jest). The moral of the
story is that we should be reporting and studying
the social theory independent of whether or not
true value is being determined. The test of any
theory is how well it predicts. I commend Clay
that he does predict, even if his theory is not
always fully elaborated.
5. Robert on October 24, 2003 12:08 PM writes...
Clay, I wrote a longwinded and probably extremely boring response on my website addressing your post. Prior to, of course, reading your clarification yesterday. I'll clarify my response to clarify your clarification, or something.
Anyway, I'm enjoying your site, which I discovered through Instapundit's link to your Grimes/Fabricant piece.
Robert
Permalink to Comment