Comments on Seb's Open Research: The Greatness and Perils of Following Suit

Then, you also have to take in to account the simp...

2009-02-12T13:02:00.000-05:00

Then, you also have to take in to account the simple psychological variation that comes with the level of enthusiasm between parties that are sharing. A very positive review will tend to skew our perception to the positive. However, if the person passing the information on is negative, your perception will tend to be negative. The question becomes, would you like it more or less depending on who gave it to you or upon a random discovery.

We are human and easily influenced one way or another. Just think about the Digg effect or viral videos. Just because you see them doesn't mean they are good.

I believe that any heuristic that tries to measure...

2009-02-06T13:47:00.000-05:00

I believe that any heuristic that tries to measure a quality index will gain in quality as it takes more information into account. The quintessential example is Google's pagerank, that managed to radically improve the quality index by looking not only at a website global popularity but by looking at which website considered others popular.

A simple way to augment the metric you propose is to combine your friend's popular picks with your knowledge of your friend's tastes.

The problem of adding more information to better your quality index is that, eventually, the index will become harder to evaluate than the problem you were trying to solve.

Another problem of a quality index is that it can be gamed. Playing with the popularity index is what publicity tries to do. In order to counter that, I believe the user of any index has to include a non-deterministic element from his side: don't blindly trust the ranking given by the index, throw a dice and shuffle the first few results!

But what is your definition of 'best song' in musi...

2009-02-03T06:13:00.000-05:00

But what is your definition of 'best song' in music? Is it the song which sells most copies? The song liked by the most people? These are popularity metrics anyway. It's hard to think of any objective measure of 'best' in that context. Will look forward to seeing what your next post adds.

Incidentally I read Dune knowing nothing about its providence - it was just a book that my mum had on the shelf. I still loved it.

Then again, do you hear the song all the time beca...

2009-02-03T04:58:00.000-05:00

Then again, do you hear the song all the time because it's good or do you think it's good because you hear it all the time?

"It is enjoyable now because you know, a priori, that it is brilliant work. Discovering, as you read it, that it is brilliant, requires quite a bit of effort."

@Daniel That's exactly one of the reasons why I never like seeing a trailer or reading a review before seeing a movie. I prefer to know as little as possible about something before I experience it, or else will often be disappointed. Some of my best movie experiences have been with films I knew nothing about in advance, since they took me by surprise. I think it's better to have no expectations and be surprised than having high expectations and be disappointed.

I think this is a bit of a straw man argument. I d...

2009-02-03T00:19:00.000-05:00

I think this is a bit of a straw man argument. I don't know anyone who would argue in good faith that popularity equates to quality.

The more interesting question is whether the relationship between quality and popularity is generally strong enough to justify applying a popularity threshold as a heuristic in order to reduce the search space. I believe that there exists such a threshold in any large domain.

I concede, however, that I don't know how to test this principle, especially since even retrospective judgments of quality are arguably a sort of popularity contest. Perhaps you'll answer this question future posts.

if a song is any good, chances are that you'll lea...

2009-02-02T22:27:00.000-05:00

if a song is any good, chances are that you'll learn about it.

Dune, arguably the most important SciFi Novel of the XXth century (Consider that Star Wars is a mere variation on Dune...), was rejected repeatedly before getting published, according to wikipedia:

"""
The serialized version was expanded and reworked, and ultimately rejected by over twenty publishers before it was published.
"""

Consider moreover that when it was finally accepted, it was by a minor publisher.

This says that you cannot rely on peer review for accuracy. Even if nobody is trying to actively fool the system, the system itself is very fragile.

There was an experiment recently (maybe reported in Wired or in the New York Times) were a famous musician disguised himself and played in the streets. Well, he did not do so well.

Hence, even if you are given Dune, prior to its publication, you may not appreciate it fully because you don't have the knowledge we have now that it has had a profound influence on contemporary scifi. It is enjoyable now because you know, a priori, that it is brilliant work. Discovering, as you read it, that it is brilliant, requires quite a bit of effort.

Similarly, if you go by a famous really good musician in the subway, you may not notice. Hundreds, thousands of people could go pass him and never notice that he is amazingly good. Yet, in a famous concert hall, everyone would applaude... what's the catch?

Why do I display my readers count on my blog? Because I seek to increase it? No. I don't care so much. What is it to me... 1000 or 100 readers? Ah! But to the reader, it makes a whole lot of difference because they feel that the blog is legitimate, so they will pay more attention.

Reference:
http://en.wikipedia.org/wiki/Dune_(novel)