Laserlike

In The Use of Knowledge in Society, Hayek explains why prices are so powerful in organizing our collective resources:

Fundamentally, in a system in which the knowledge of the relevant facts is dispersed among many people, prices can act to coordinate the separate actions of different people… The whole acts as one market, not because any of its members survey the whole field, but because their limited individual fields of vision sufficiently overlap so that through many intermediaries the relevant information is communicated to all. The mere fact that there is one price for any commodity… brings about the solution…

Prices force us to bring our knowledge to bear — the simple act of selling or buying something at a specific price helps us determine value and allocate resources.  To paraphrase Hayek, the collective actions of many buyers paint a tiled picture of distributed knowledge about the value of something.

The click as empirical evidence.

Google’s use of PageRank to re-invent the search market is old news.  PageRank was a key signal to help us develop a hypothesis on a query by query basis on what pages on the internet were a good match for each query.  But the click was the way to continuously test that hypothesis and tune PageRank.  As search engine optimization evolved to help sites game PageRank, hundreds of other factors were required to find the signal again in all of the noise.  Again, the objective function in optimizing the system is the click.  Search engine vendors watch the click-through rate of the top result in a query religiously.  After all, the ideal would be that the first result would give the perfect result 100% of the time.

Much has also been said about Google’s spell checker and language translation efforts.  While Google knows nothing about spelling rules, grammar, and syntax, the site is able to help me identify spelling errors through the use of mounds of historic data.  And to assist me on [simple] translations.  But advocates of inductive methods often fail to appreciate, once again, that the empirical value of the click is the last step in the process that allows for continuous tuning of the system.

Perhaps a little help from Big Brother?

What could a system do if it could see everything everyone sees and then run empirical tests to optimize the system to improve my life?  

What if one vendor tracked my every move?  Imagine that index.  For every page I’m on and for every object I see (video, photo, chunk of text), what if they could offer me further alternatives (people who liked this video also liked these videos)?    Perhaps this explains Google’s intense focus on Toolbar distribution?  What if Microsoft used their installed base to glean that data and finally acquired Yahoo! (or at least the search division of the company)?  Despite the conventional wisdom that the search game is over, I think Google has reason to worry (in particular about Microsoft due to their Windows and IE installed base).

For me?  You shouldn’t have.

Over ten years ago I envisioned a personalization service that tracked everything I did and used that data (including the click data) to personalize the web just for me.  After a venture capitalist convinced me that Microsoft was about to launch something that would do just that (it was Passport), I took another path.  But I’m now convinced that the world is ready for just such a service.

And that brings me to my final thought — Facebook should drop social media advertising and pursue personalization as their path to revenue.  They should build a toolbar installed base through distribution deals and by building really cool features that customers value.  They should then take that data (what user does everywhere on the web) along with the the social graph to rebuild PageRank based on a user’s social graph.  Weigh my actions highest, my first degree connections second, and so on.  Search just for me.  Or even start by delivering buckets if the storage and compute costs are too high to have an index for every user on the planet [today].  And, finally, open that data-set up to the world’s publishers so that we can finally deliver a personalized web to the world.