Don’t you just love those “aha” moments in life?  Those experiences where you learn something and the world just makes more sense?  I remember sitting in Gary Loveman’s last class before he joined Harrah’s over a decade ago.  Before that class I had heard of the 80-20 rule, but hadn’t really internalized its meaning.  Understanding this principle has been the single most important discovery in my career.

The Pareto Principle (aka the 80-20 Rule, the Law of the Vital Few)

Italian economist Vilfredo Pareto developed what is now known as the 80-20 Rule after he discovered that 20% of the Italian population accounted for over 80% of the wealth.  He later found that the rule applied for many other economies (and still does, see Table below from Wikipedia).  

Source: United Nations Development Program. 1992 Human Development Report

Back to School.

It turns out that this statistical phenomenon applies to much more than wealth distribution.  The realization came for me when Professor Loveman explained how to use the Pareto Principle to shape business strategy.  I believe the specific case was Harrah’s, where he is now CEO and was a consultant at that time.  

First you segment your customers into some number of groups.  Let’s say five groups based on revenue per year (although you can pick whatever value suits your fancy).  Next find out what percent of revenue your top 20% customers account for, the next 20%, and so on.  You will likely find that the distribution is a power law.  It may not be exactly 80-20 — actually, it’s often far more skewed.

Now find out why those customers are so loyal.  Call them and find out what separates the top segment from the other segments.  If you have multiple products, do they buy certain products more than others?  Use certain features more than others?  Where do they come from and can you tune your marketing to find more customers like your best customers?  Does your analytics effort measure overall metrics as well as the health of your business by segment?

Another big learning was that, for most companies, the average cost of moving a customer from the second most loyal to the most loyal category was extremely small while the cost of going from a new customer to the most loyal customer was often a money losing proposition.  That is, the costs of acquisition exceed lifetime value, on average.  Most companies focus on customer acquisition way too much!  Even very early stage businesses are best served by having evangelists amongst their early adopters.

Other applications.

While I was working at my first startup, an extremely talented marketer from Goodby, Silverstein suggested that I read Eating the Big Fish.  It was a fun read and my takeaway was that positioning is really about finding the message that resonates with that top 20% of your customers.  Most people try to communicate a general message to all potential customers and have limited appetite for negative feedback.  If you have hundreds of millions to fuel your marketing efforts and play in a mature industry, pursuing a broad generic message may be the right thing for you.  But if not, you will clearly get the best bang for your marketing (and PR) buck by honing your message for your best customers.  And don’t forget to chug some Pepto Bismol, because you may ruffle a few feathers along the way.  Just focus on getting the message through to your best customers.

Years later I was working on a large acquisition (the acquisition didn’t go through).  The target had hundreds of customers with extensive contracts.  Our law firm evaluated each contract with equal vigor.  I discovered that the top 10 customers (of several hundred) accounted for over 80% of revenue.  So I read just ten contracts and found some very bad stuff in those contracts that the law firm missed.  Think about the psychology of sales people in organizations – it’s not worth breaking the rules for small deals.  But when you’re talking about millions of dollars in commissions and making a quarter, well rules are made to be broken, right?

And in many companies [certainly large companies] 20% of the employees create a disproportionate amount of the value.  While this is so obvious to so many, why aren’t salaries and bonuses distributed in the same way?  Why do HR professionals tell executives that everything is fine because we only have 15% of our workforce turning over each year of their own volition, which is only slightly up from years past?  Don’t you really want to know what’s happening to your top performers?  

The Long Tail.

There are some cases where using this thinking can get you into trouble.  Chris Anderson did a great job in The Long Tail of describing how the 80-20 Rule can get you into trouble.  With that caveat in mind, many business leaders in many functions would be well served understanding how to put the Pareto Principle to work.

The experience curve describes cases where the cost of doing something decreases the more often the task is done.  It is a cousin of the learning curve and has been a cornerstone of corporate strategy since the late 1960’s when Bruce Henderson of the Boston Consulting Group (BCG) applied cumulative learning gains to corporate strategy.  An early mover in a market characterized by a significant experience curve should pass on cost savings to consumers with the objective of gaining share.  A positive feedback loop is then triggered leading to a potential monopoly.

Brief Summary of the Experience Curve (from Wikipedia)

The experience curve… states that the more often a task is performed the lower will be the cost of doing it. The task can be the production of any good or service. Each time cumulative volume doubles, value added costs (including administration, marketing, distribution, and manufacturing) fall by a constant and predictable percentage.

These effects are often expressed graphically. The curve is plotted with cumulative units produced on the horizontal axis and unit cost on the vertical axis. A curve that depicts a 15% cost reduction for every doubling of output is called an “85% experience curve”, indicating that unit costs drop to 85% of their original level.

Mathematically the experience curve is described by a power law function sometimes referred to as Henderson’s Law:

• ^[4]

where

•  is the cost of the first unit of production
•  is the cost of the nth unit of production
•  is the cumulative volume of production
•  is the elasticity of cost with regard to output

I have a hypothesis that there is also a learning curve for machines — I have yet to formally test this hypothesis and would be interested to hear from you about whether you think this makes sense and to see if anyone has heard of any tests that would support or refute this argument.

The Machine Learning Curve is likely limited to particular “industries,” just as is the case with the Experience Curve in manufacturing.  So the ML Curve for Search may have little to do with the ML Curve in some other market — for example, genetic testing.  In other markets I suspect that we will find out just how closely the two are linked — for example, behavioral targeting of text and display ads.

Arguments for the likely existence of a Machine Learning Curve.

1.  More data = better hypotheses.   The availability of massive stores of data to both regression algorithms and to human beings allows for broader investigation into potential relationships between variables that were not previously known.  Furthermore, experience in finding these relationships will improve the ability to spot *potential* patterns in the future.  For example, a group may learn that a set of actions may proceed another set of actions by 30 days (e.g. reading user reviews on cars prior to purchase).  This time-based delay is a potential candidate for future experiments — from targeting content on autos to understanding the consideration period for other items.  This could have a cumulative impact, as every relationship builds on previous relationships.

2.  Larger audience = more experiments.  When it is time to conduct randomized tests, a larger audience will allow for more granular testing.  Let’s say a test requires that we put something in front of 1,000 users.  We need at least 2,000 users as we need a test group and a control group.  So the number of experiments that can be run is (audience-1,000)/1,000.  If this sort of process is well executed, one would assume that the early mover would have the largest audience.

3.  Time-based effects.  While you could make the argument that a new player could buy a ton of equipment and spend a huge sum on marketing to neutralize an incumbent on arguments 1 & 2 above, there are significant time-based effects that make such a strategy an unlikely winner.  It takes time to measure certain effects.  For example, the consideration period for buying expensive products like cars can only be measured over a period of time that is commensurate with [at least] the consideration period.  The probability that a particular user is a real person and has a worthy reputation grows with time.  The half-life of some data (e.g., news) has a very different value with respect to time than other data (e.g., historic financial results).  By the time a new player figures these things out, the incumbent has grown her lead.

Just as with the Experience Curve, the most likely way that the Machine Learning Curve loses momentum is a discontinuity in the market.  As a Stanford professor of computer science recently told me, machine learning is not a product but rather a way to take a product to the next level.  I agree and would add that the ML Curve may increase the odds that a market leader who aggressively adopts machine learning has the potential to lock out the competition.

Industrial organization helps us judge the attractiveness of an industry by understanding “the five forces.”  In the economic food chain, the best positioned segments are those with the greatest level of consolidation relative to buyers, suppliers, and competitors.  So, for example, Amazon entered the book market as an intermediary between tens of thousands of publishers (fragmented suppliers) and millions of buyers (consumers).  

In the video “market,” there are thousands of producers (millions including user-generated content) and millions of consumers — YouTube had less opposition from the incumbent video “suppliers” than Napster or Imeem did with the incumbent music “suppliers,” of which there are four that dominate the market.  The video content market is extremely fragmented while the music businesses is massively consolidated.  With the latter a small group can take coordinated action while the former suffers from a free rider problem.

On the Web we can extend this thinking to the data food chain.  Google and Yahoo! Search (metadata players) sit between billions of pages (data players) and hundreds of millions of consumers.  In these types of markets, there is the potential for natural monopolies from economies of scale (brand, operations, the “machine learning curve”).  When YouTube was clearly on its way to a “video monopoly,” Google had good reason to act.  When data ownership is fragmented, incentives for sharing metadata are high (think SEO).  But when a content market is a data monopoly or oligopoly the company that owns the data has the potential to have a monopoly not only on the data, but also on the metadata.  For example, Google should be able to offer dramatically better search on the corpus of YouTube videos because they have access to tons of non-public metadata (how long users watch a particular video, for example).  Furthermore, that metadata should improve non-video search — if not now, perhaps once all audio from video is transcoded into something search engines can grok.

So what does this have to do with Twitter?

As I have mentioned before, I think Facebook is an incredibly interesting property.  Well beyond what today’s conventional wisdom suggests.  They have a very consolidated level of ownership of a high fidelity social graph mapped to profile information, communications relationships, photographs, application usage, and so on.  In my mind, Facebook is the one company that really has the potential to give Google a run for its money, if it ever goes after search.

But Twitter also has very interesting data, and will surely carry a lower price tag.  Check out Twitter Search, to find out what is happening right now.  In addition to a real-time search capability, you can find out what topics are trending upward (sound familiar?).  Google has aggresively injected RSS feeds into search results as of late, and Twitter offers a potentially superior signal to noise ratio than blog content.  And note that Twitter’s API has been locked down a bit since the acquisition of Summize (now Twitter Search).  

Twitter has the potential to offer a major search vendor the ability to not only inject real-time content into search results, but also to offer high-quality metadata to tune the ranking algorithm.

Companies invest enormous sums of money on data warehouses.  They then spend much more on various business intelligence tools and training.  I have used many of these tools in previous roles, and basic use of of the Data Analysis package in MS Excel is far more advanced than what I have seen used at most companies.

1.  Identify correlations within your data.  

Many companies’ data mining efforts bring to mind Jorge Luis Borges’ Library of Babel.  Raw data is worthless.  While this statement self-evident, it’s clearly not appreciated as most firms make poor use of their data warehouses.  Business intelligence should help us make predictions about the future.  A good place to start would be to identify correlations in data and then to extract functions (e.g., sales increase x% when customers spend y minutes on page z of or our web site).  

To start, simply take whatever data you have and do a scatter plot in Excel.  If it looks like the data “lines up”, run a regression to extract the function (on Excel for Windows, use the Data Analysis tool; on the latest Excel for Mac you will need to do the least squares work manually).  Brief tangent – I would love to see someone build a simple “correlation machine” that could ingest any CSV file, find the best correlations within the data, and then spit out functions in descending order based on correlation coefficients.  

2.  Use randomized testing to identify causation.

Correlation does not equal causation.  The best way to find out if a correlation is meaningful for predictions is to do A/B testing.  If we send 50% of a small group of customers to page z1 of our site, sales are x dollars.  If we send the other 50% of that small test group to page z2, sales are 1.5x dollars.  If the function can make predictions, great.  If not, go back to #1 and try again.

3.  Your intuition sucks.

If you haven’t already, read Michael Lewis’ Moneyball.  Just as with baseball scouts, most business people believe that they have good intuition.  And, just as with baseball scouts, most business people would be well served to add statistics to their arsenal of decision making inputs.  

The larger the organization, the more people will differ with respect to their intuition.  Of course, the decision is ultimately made by the most senior or the most vocal person (more often than not, the correlation between these two things is very high).  Oddly enough, senior people are often further away from the evidence and, consequently, are not well positioned to make the call.   Intuition may be an entirely reasonable replacement for #1, but it is most certainly not a good replacement for #2.

This is not a suggestion to replace the way decisions are made in your organization today with algorithms, but rather another tool.  As basic as this all sounds, I have been surprised by how few people really use basic statistics to help make decisions.  And if most executives [continue] to ignore this advice, it’s just more opportunity for the rest of us…

A key reason for Facebook’s success was a very well thought-out access control system.  To see my Facebook profile you either had to (1) be a friend as confirmed by the profile owner or (2) a member of my network.  In the early days of Facebook, network membership was confirmed by authenticating the domain of a user’s email (e.g., student@harvard.edu).  Since the vast majority of users do not alter default access controls, most sites previously had opted for a public default (MySpace) or direct social graph only (instant messenger buddy list).  Protecting a user’s information by very thoughtful access controls encouraged Facebook users to share their real names and early norms encouraged users to accept only real friends as connections.

While this solution was a big step in the right direction, it is far from perfect.  People implicitly use very sophisticated access control systems in their daily lives.  In addition to the coarse controls encoded in Facebook, we also vary what we share with others by topic (e.g., I may share work-related information with a different group than I share information about a recent trip to Las Vegas) and over time (e.g., relationships evolve over time).  

It looks like the current social graph owners are very well positioned to own a very important piece of the social computing future.  But if someone can figure out how to dynamically represent the REAL social graph in all of its glory, they will create an enormous amount of value.  

While I’m not sure what this solution looks like, I have a few thoughts.

1.  It must be passive.  Consumers will not actively manage the minutiae of their online social graph.  Simple actions like “accept friend” are fine.  But for atomic permissions, the solution is auto-magic — with some sort of override, of course.

2.  It must be dynamic.  If you can’t keep up with my life, I’ll stick with what I have today. 

3.  A new user interface.  A big part of the solution lies in figuring out how to visually represent the complexity of these permissions.  The system should make it easy for me to see where my permissions stand today, as well as allow me to quickly see what changes have been made in the past.

I remember a conversation with a venture capitalist in 2003 during which I suggested to him that social networking would be a very good place to look for the next big, big thing.  He asked, “so you don’t think it’s a fad?”  He wasn’t suggesting that it was a fad, but he clearly wasn’t sure.  Most of the other investors I met back then were certain that social networking was a fad.

Fast forward to the Summer of 2007.  I was having sushi with a good friend who is one of the best engineers I know.  At that dinner he told me that *all* future web applications would be developed on the Facebook platform.  Everything.  The pendulum had swung to the other side of the hype cycle.  Many investors were also excited about the opportunity and invested in everything social at silly [in retrospect, at least from the perspective of this moment in time] valuations.

With the recent financial downturn, Silicon Valley is heading back to the “social networking is a fad” side of the hype pendulum.  ”These sites are like nightclubs, and users will eventually find something else” is a common refrain amongst many in Silicon Valley these days.  ”Even if they aren’t like nightclubs, you can’t build a big business on $0.25 CPMs, right?”  And others are suggesting that even Facebook is in trouble.

While the hype cycle clearly got ahead of itself in the past few years around all things social, the fundamental trends are wildly positive for the future of social computing — both from a usage perspective and, eventually, from the business perspective.  Yes we are going through some hard times and everyone will feel the pain — even Facebook.  There is no doubt that we must turn audience attention into revenue.  Despite all of these challenges, to channel John Doerr, I believe social networking is underhyped.  

Facebook and others in the industry think about their mission as exposing and documenting the *real* set of relationships that people have in the offline world, rather than creating a separate online social graph.  While users navigate this graph online today by surfing from profile page to profile page, or perhaps through some entertaining third party application, the future is going to look more like the offline world.

Think about how you make so many of the important decisions in your life.  Need a doctor?  Ask people in your network.  Looking for a job?  Most people are better off leveraging their real social networks rather than using an existing online job site.  What about doing a search on Google for your friend Mary Jane.  No, I wasn’t looking for the Urban Dictionary to tell me that Mary Jane = Marijuana.  For many queries consumers make today, the collective wisdom of PageRank is of limited value.  What if there was an option to do a standard [universal] search or have results filtered through your social graph?  Imagine augmenting Ebay’s trust system with your real world social network.  How about a payments platform built on your social graph?  

The opportunity goes well beyond these obvious ideas.  People socialize with each other, in part, because they share similar interests.  A system that knows about actions within your graph can increase the probability of delivering a dramatically more relevant experience to you.  Imagine going to Yahoo.com and getting the news stories that are likely to be most interesting to you based on actions by users from your social graph.  How about a social version of Amazon’s recommendation system for all ecommerce players on the web?  In the coming years, developers will use the social graph to replace noise with signal in just about every corner of the web.  

Facebook Connect, Open Social, and other efforts are working to allow consumers to use their own social graph on third party sites.  I’m not sure if these efforts will work out better than the Facebook Platform or Facebook Beacon, but I applaud Facebook for not being afraid to fail.  They are pushing the entire industry to realize the broader dream of social computing.  

In the next decade social computing will change the face of the web as we know it today.  And smart entrepreneurs and investors will earn healthy returns by challenging [the latest] conventional wisdom.

“Things on a very small scale behave like nothing that you have any direct experience about. They do not behave like waves, they do not behave like particles, they do not behave like clouds, or billiard balls, or weights on springs, or like anything that you have ever seen… Because atomic behavior is so unlike ordinary experience, it is very difficult to get used to, and it appears peculiar and mysterious to everyone–both to the novice and the experienced physicist.”   -Richard Feynman, The Feynman Lectures on Physics

The Tipping Point.

Malcolm Gladwell offers an analysis of sociological phenomenon in The Tipping Point to help us understand how to identify “the moment of critical mass, the threshold, the boiling point.”  A key tenet of his argument is that a small group of hyper influential people called “connectors” help launch exponential growth by bringing the world together.  This idea is a deeply held belief amongst many product and marketing executives, entrepreneurs, and venture capitalists.  It’s a compelling story.  Unfortunately, it’s dead wrong.

Feynman explained that quantum mechanics is so counterintuitive that it’s hard to grasp for anyone, even Feynman.  You must have a deep appreciation for mathematics and follow the data in order to really understand what is going on.  The same thing is increasingly true in developing consumer products, particularly consumer web properties.

Fast Company had an extensive interview with Duncan Watts of Yahoo! Research, which I highly recommend.  Watts has done a number of experiments including an updated take on Stanley Milgram’s 1967 Six Degrees of Separation study.  But instead of having 160 people send physical mail, Watts’ experiment involved 61,000 people using email.  Like Milgram he found that, on average, messages passed through approximately six people.  But he also found that connectors had very little to do with the functioning of the system.  He also ran a Sims-like experiment where he found that “the average Joe” was as likely to be the source of the propagation of exponential growth as “connectors.”

“Watts thinks trends are more like forest fires: There are thousands a year, but only a few become roaring monsters. That’s because in those rare situations, the landscape was ripe: sparse rain, dry woods, badly equipped fire departments. If these conditions exist, any old match will do. ‘And nobody,’ Watts says wryly, ‘will go around talking about the exceptional properties of the spark that started the fire.’”

Enter Joe the Plumber.

After working on numerous consumer web products and reviewing the data of hundreds of consumer web companies in my current role, I have come to appreciate Watts’ forest fire metaphor.  Most entrepreneurs are sure that they have the right idea and continue doing the same thing, sometimes for years, in the hope of hitting a tipping point.  ”Now that we have a product, we are raising capital for marketing,” is a refrain that I hear far too often.

Most of today’s massive consumer web properties experienced exponential growth fairly shortly after launch.  A few thousand users over a few months is probably sufficient to find out it you have hit a nerve.  So forget the connectors.  Any average Joe will do once you find some dry timber.

In an op-ed piece on October 16, Warren Buffett advised the American public to, “Be fearful when others are greedy, and be greedy when others are fearful. And most certainly, fear is now widespread, gripping even seasoned investors.”  Mr. Buffett went on to say that over the long-term equities were the right call and that he wasn’t going to call the bottom.

I did a quick analysis to impute the compound annual growth rate (CAGR) of an investment in the DJIA at various prices.  In the table below, pick the number of years that you believe that it will take for the DJIA to return to it’s October 9, 2007 price of 14,164 and then pick the level at which you would invest in the DJIA.  The result is the annual return you would get.  

Today’s closing price of 8,691 is highlighted.  If the DJIA returns to it’s previous high in anything less than 10 years, it’s not such a bad time to buy a DJIA index fund.  As always, a copy of my file is here (it’s called buffett.xls).

As I am sure many Americans have done over the past few weeks, I have had many discussions with friends about the recent stock market gyrations and potential economic fallout from the financial crises.  To get some perspective, I went to Yahoo! Finance and looked at a chart of the DJIA prices from 1929 to the present.  I took a rough estimate of the high price before a precipitous fall and then tried to find the low before things started to get better.  Below are a the results of this exercise, which are by no means scientific.  You can find the spreadsheet here.  

The first three columns are fairly obvious.  In the third, I applied previous percentage loses to the relatively recent high fro the Dow Jones Industrials (~one year ago).  This is not a prediction, but rather just fodder for the discussion.

“Science is not a body of facts.  Science is a state of mind.  It is a way of viewing the world, of facing reality square on but taking nothing on its face.” -Natalie Angier, The Canon

About a week ago I came home from work and found my kids perplexed with our TV.  The volume wasn’t working and they didn’t know what to do.  My first grader knew about the “volume” and “mute” buttons on the remote, but neither worked for her.  This was clearly a family crises, so I made correcting this situation my top priority.  

First I turned the amplifier mute on and then off.  Nope.  Then the cable receiver mute on and then off.  No again.  Next I rebooted the cable box.  Didn’t work.  But when I tried the mute button on the cable receiver again, voila!  We have all used a simple sequential problem solving process like this for issues with a TV, computer, or some other home appliance.  By developing a clear hypothesis, isolating potential solutions, and testing these solutions one at a time, we are implicitly using the scientific method.  The feedback loop on whether our experiments worked or not is abundantly clear — the volume is either on or off.

Then we go to work.  And despite a degree from XYZ university in engineering or business, we often apply less scientific rigor to product development than we do to fixing the volume on our televisions at home.  Think about it.  Product releases are often packed with features.  Why?  The product development leadership has decided that those features are what the customer wants, right?  But how will they know if their hypotheses are right if they are simultaneously running a large number of experiments (features)?  Sure they can develop statistical inference models to isolate certain variables, but why go there?  It’s like trying to solve a home PC internet connectivity problem by trying every potential solution simultaneously.  That’s a terrible algorithm, yet it dominates product development.

Less is More?

There is a fashionable design philosophy in product development circles these days — let’s call it less is more.  37signals has been a champion of this approach, but there are many others.  A key tenet of the philosophy is that uncluttered products with fewer, better features are preferred to similar products with more features.  I agree with the less is more product development approach, but for a different reason.

The reason I like less is more as an approach is that it allows for a more scientific approach to product development.  By starting a new product off with as few features as possible (1?), you can be incredibly scientific.  With 10 features in a single release, you may spend more time trying to figure out what is working and what isn’t working than it took to build the thing in the first place.  As you incrementally experiment with your product, you can observe the impact of a particular feature one at a time and adjust accordingly.  

Scientific product development.

So this leads us to the following approach to developing new products.

Step 1.  Have a very clear idea of the problem you are trying to solve.  

Step 2.  Develop a hypothesis about the minimal feature set that will address the problem (ideally just one thing). You can do research or just have a gut feeling about the answer.  A good “product picker” offers significant leverage in this step of the process.

Step 3.  Test your hypothesis by shipping product quickly.  A killer engineering team provides massive leverage in this step of the process.

Step 4.  Observe the results of the experiment.  Did the results of your test match what you expected?  If not, kill the feature and start over at Step 2.  If things worked, continue feature development by starting over at Step 1 again.

Most experiments fail.  Many teams do a relatively good job on Steps 1 through 3, but forget the importance of Step 4.  Instead of killing bad features, they simply add more hoping that feature 10 will somehow make crappy features 1 through 9 better.  

By embracing a scientific approach to product development, not only will your business have a much higher probability of success, but it will also be a more fun and creative place to work.  Nothing kills innovation like the fear of failure.  And nothing leads to failure like a process that resembles astrology more than it does astronomy.

With the move to digital distribution of media, content and physical distribution have been untethered.  For example, a decade ago a consumer would get her music by CD — she would buy an entire album in order to get the one or two songs she really wanted.  Or she would pick her favorite paper and get her political news, sports news, and the time and locations of movies in her area all from the same paper.  The separation of content from physical delivery is well known and has been widely discussed.  Yet, media companies haven’t really changed their approach to business.  The New York Times online is a digital replica of its atom-based cousin.  Fifteen years ago I read the New York Times cover to cover just about everyday.  

The New York Times used to be a place where people started their day.  It was also an authoritative source of news and entertainment.  And being both of these things was possible because of the delivery mechanism physical paper.  Now many consumers start their day with the Yahoo! homepage or MyYahoo, Google Reader, Facebook, Twitter, or even with Google Search (in the past year, I find myself starting my day with Twitter and Facebook).  On the content side I still get some of my political and international news from the New York Times (much less, though), but I much prefer Yahoo! Sports for my sports news, Fandango is a much better source of movie information for Silicon Valley, and TechCrunch and Techmeme are where I go for technology news these days.

Time to focus.

Media companies cannot ignore the dramatic changes that have taken place in the past decade.  Rather than try to protect their franchises they must completely rethink their strategies.  The best defense is a good offensive.

First, media companies need to figure out if they are in the business of telling people where to go or if they are the place where consumers end up.  They cannot be both any longer.  If they want to be the place where people start, they should put strong technology executives in leadership positions and should also consider buying technology companies as a way to kick-start the revolution.  Twitter is one such candidate.  Their recent acquisition of Summize (now Twitter Search) would make a great homepage for a media business.  Big media executives must get comfortable with the fact that they should focus on getting the consumer the best product even if that means sending users to the competition.  

If a media business would like to keep its focus on content, that’s a fine alternative.  In fact, while “winning” is likely a much smaller outcome than the “place to start” path, it’s a much higher probability bet — particularly because the DNA of these businesses is deeply content centric.  But rather than have just another one-size-fits all solution, these business must focus like crazy on doing one or two things really well.  So, for example, the New York Times might want to consider dropping sports, regional news, technology, science, and travel and instead nail politics, arts, and style?  It would be a painful transition, but the alternative is likely to be far more painful.

In ten years, most of the existing media giants will be a shadow of their former selves or gone altogether.  The winners will either focus on being the best place to start or nailing a relatively narrow content category and becoming the definitive place to go for just those things.

Cargo Cult (background from Wikipedia).

A cargo cult appears in tribal societies in the wake of interaction with technologically advanced, non-native cultures. Focused on obtaining the material wealth of the advanced culture through magical thinking, religious rituals and practices, the cargo cult believes the wealth was intended for them by theirdeities and ancestors.

“The most widely known period of cargo cult activity…was in the years during and after World War II. First, the Japanese arrived with a great deal of unknown equipment, and later, Allied forces also used the islands in the same way. Manufactured clothing, medicine, canned food, tents, weapons, and other useful goods arrived in vast quantities to equip soldiers. Some of it was shared with the islanders who were their guides and hosts. With the end of the war, the airbases were abandoned, and cargo was no longer being dropped.

In attempts to get cargo to fall by parachute or land in planes or ships again, islanders imitated the same practices they had seen the soldiers, sailors, and airmen use. They carved headphones from wood and wore them while sitting in fabricated control towers. They waved the landing signals while standing on the runways. They lit signal fires and torches to light up runways and lighthouses.

Cargo Cult Science.

In his 1974 Caltech commencement address, Richard Feynman compared examples of modern “science” to cargo cults, and called the phenomenon cargo cult science.  As I read the text of Feynman’s address again this weekend, I couldn’t help but to think of the similarities between what he was describing and so much of what I have seen and read in modern business management.

Cargo Cult [business] Management.

Why are most of the meetings you attend so worthless?  Perhaps those in leadership positions are themselves subject to the same thing from their managers and are simply replicating the same behavior?  ”It works for him and he is the CxO, so if I do it I will be CxO someday?”  Why do so many presentations contain so much content yet so little information?  In my experience, professionals are trying to fit the “proven” formula within the company (e.g., executive summary, lots of detailed text and charts to show you how smart you are, and a conclusion).  ”Jack does it that way and he’s successful, right?”  Trust me on this — the formula sucks.  Try telling a story instead.  Why do leaders hold so many all-hands meetings, yet neither inspire anyone nor communicate a byte of new information?  They are likely doing what they think other good managers do, based on some vague association they gleaned from an MBA case study, a business book, or even a story about Jack Welch on the news?  This is almost never the result of a single manager, but more often an organizational cancer which will eventually prove malignant.

Hit the BUSINESS section of your local bookstore for insights into why cargo cult management runs so rampant — it’s hard to find authors that understand the difference between causation and correlation.  Jim Collins’ Built to Last and Good to Great are two of the biggest piles of cargo cult crap out there — oh, and best sellers.  In Good to Great, Collins and his team of “researchers” examined historic stock returns of potentially “great” companies relative to a market index.  The research team took the top historic performers and then narrowed the list down further by looking for only those companies that did much better than their industry peers (e.g., if an entire industry did exceedingly well, the company was dropped).  They compared these “great” companies to companies with lower stock returns which are just “good.”  How did they compare the companies?  By reading news articles and doing interviews and “systematically” coding the results into “strategy, technology, and so forth.”  They then used all of this research to see patterns so that they could tell a story about how any manager could take a company from good to great.  It’s a great story, but would be more appropriately located in the FICTION section of the bookstore. 

What if aerospace engineers employed a similar analytical approach — look at historic data and use induction to find the answers in patterns gleaned from the data?  I’ll bet you that our airplanes would fly about as well as the one in the photo above.  Without the empirical testing (observational) part of the scientific method it’s quite hard to know if what you have is causation or correlation.

Entrepreneurship.

While there are examples of cargo cult entrepreneurship, startups run this way have a dramatically higher probability of running out of money.  Of course, as startups that survive earn the air-cover of a business with momentum they often allow inefficiency and bad management to creep into the system.  Or they were just lucky to begin with.

I believe that the biggest advantage a startup has over a big competitor is intellectual honesty.  Most of the entrepreneurs I know start their own companies or join startups in order to do what they wanted to do at their larger [previous] employers — to do something worthwhile.  Then they finally reach the conclusion that the only way to avoid cargo cult management is to start from the ground up.  This is particularly true of technical professionals who view intellectual honesty as core to their job.  How often do you hear business people poke holes in their own arguments the way engineers do?

Which brings us back to the last line of Feynman’s 1974 address:  

“So I have just one wish for you — the good luck to be somewhere where you are free to maintain the kind of integrity I have described, and where you do not feel forced by a need to maintain your position in the organization, or financial support, or so on, to lose your integrity. May you have that freedom.”

Well said.

With the technical feats accomplished by the new Yahoo! Mail, the new Hotmail (Windows Live Hotmail), and Gmail, who needs an email client these days?  All three major web-based email products have done an incredible job making Javascript in the browser act like an email client.  Yet, there are a few key reasons that so many people (like me) opt to use an email client today.  

I would like to see someone build a peer-to-peer email system which combines all of the functionality of Outlook and Exchange in a distributed client model — similar to what Skype has done with VOIP and IM, but for email, contacts, and calendaring.  Perhaps you could start with Eudora now that Qualcomm has said that they are going to open source one of the best email clients of all time?  Or perhaps you could just build on Thunderbird as Songbird has done with the browser?

One of the reasons why P2P worked so well with Skype is that users have an incentive to keep the client open all day — they want to receive messages.  Continuous client operation in a P2P system increases the resources for the network relative to applications which are running only while being used.  Email also benefits from being left open all day (either the client or the tab in the browser for web-based mail).

P2P email advantages.

1.  Your personal Exchange server.

During registration users would have the option of getting a new email address with our domain, picking a web-based email vendor (as long as they have IMAP/POP), or using their own domain (e.g., laserlike.com).  They could create as many email addresses as they would like (joe@laserlike.com, jane@laserlike.com, etc), as well as control their own mailing lists, their own directories, and do anything else that you can do with Exchange.

2. Disconnected use.

When I hop on a flight, I love to use the downtime to crank through my inbox.  Today, you simply cannot offer the power of Outlook + Exchange through the web browser when disconnected.  Yahoo! recently announced that the Zimbra client would integrate with Yahoo! Mail, which is a very smart move by Yahoo! — but it’s still a client.  And you can get free access to Gmail’s POP/IMAP features with any POP/IMAP client, like Apple Mail.  Yahoo! offers the POP/IMAP service to any client for a premium (the iPhone is an exception — it’s free) and Microsoft has no such offering for any price.   

You must have a client of some sort to have access to email when you are disconnected from the Internet.  A secondary benefit of a client is caching — even when you do have Internet access, the client offers a better experience as variations in the quality of a connection are smoothed out for the consumer by a client.  

3.  Availability.

Related to the first point is the notion of being able to have access to my email whenever I want it.  While web-based email systems are amongst the most reliable applications in the world, they do experience downtime.  And when they are down, you cannot get anything!  With a client, if your service provider is down (corporate Exchange or web-based mail), at least you can access the most recent snapshot of your email, address book, and calendar.  If your computer crashes, there is always a copy on the server which can be accessed through a web-based interface (for Exchange or web-based mail) from another computer.  With a client, you get an extra layer of protection (or several if you have a computer, blackberry, and web-based access to your email server).  It’s highly unlikely that you would experience failure on your computer, Blackberry, and your email server simultaneously.

4.  Security.

If you are using Outlook connected to your company’s Exchange server, there is a good chance that everything you do is archived forever.  Archiving software like Symantec’s KVS product even keeps copies of emails that are deleted by the user and allows executives to “mine” employee data without notification.  The major web-based players are required to turn over data if they are subpoenaed – again, you may not even know that your email is being reviewed.  On a national level, certain governments may also review email communications for other reasons. 

A P2P system could be built that would encrypt all communications so that nobody could retrieve your communications without your permission (or at least your knowledge, if the requesting party is armed).

The flip-side of the cloud computing coin is that everything you do can be retrieved without your knowledge — I wonder what happens when the government subpoenas Salesforce.com for a customer’s records?  Are they even permitted to notify them?  If they have Siebel installed, at least the authority making the request must approach the subpoenaed party.

5.  Large files.  

When you want to send a really large file, email often doesn’t work.  With our system, you can email files of any size.  In fact, we will use client-side deduping, which will allow for very fast “transfer” of large files which are common.  Forget YouSendIt, we’ll send it for you.

6.  Search.

It’s amazing to me that client-side (Outlook in particular) search is still so bad after all of this time.  Our client will include a PST “sniffer” which will find all Outlook files on a client and ingest them into our P2P client.  We will then index the mail and make wicked fast search.  We will also allow users to ingest all web-based mail for a complete search experience.  You don’t need Google Desktop, our system will handle search for you.

As I noted earlier this week, there are many opportunities in the email space.  While Thunderbird has lagged Firefox in adoption, it still has 67 million downloads – and that’s an okay a client.  A P2P email client-server combination with some of the features I suggested has the potential to make a bigger dent in the >1.2 billion user market.

According to the Radicati Group, the number of email users worldwide will reach 1.6 billion in 2011 from an estimated 1.2 billion users in 2007 (393MM of these accounts are corporate users).  According to Internet World Stats, just under 1.5 billion people use the Internet today.  According to Nielsen//NetRatings, the digital media universe was approximately 550MM in May 2008.  However you cut the data, it’s striking that about as many people use email today as use the entire web.

Despite the fact that email is still the killer app on the internet, it seems like there is very little in the way of new businesses being formed around the space.  Sure, social networks are mainly a communications solution (as long as security issues are addressed).  And companies like The Mechanical Zoo, Xobni, and Twitter are doing some really interesting things in the communications space.  But it just seems like there is so much more to be done…

Here are a few brief thoughts, although I would really like to hear from you if you know of or have an interesting business in the communications space [you can reach me at mspeiser at gmail dot com].

1.  Dynamic social networking.

There is much talk about “the social graph” in social media circles. The idea is that we all have relationships that exist in the real world, and firms like Facebook have worked to digitize the graph of those relationships.  I love this guiding principle.  

However, real world graphs have far greater fidelity than their digital copies.  For example, let’s assume that everyone you know is a friend on Facebook.  You have a crazy weekend in Las Vegas at a friend’s bachelor party and you want to share the photos with friends.  That’s probably not something you want to share with co-workers, which is what happens today when you upload those photos to Facebook.  Instead, you discriminate and limit distribution (likely by email) to those who attended the event.  Or let’s say that you would like to ask someone for advice on getting a vasectomy?  Perhaps it’s the same guys as those from the bachelor party, or perhaps you want to save that one for your doctor?  How about advice on negotiating an acquisition?  You get the point.  Different people help you with different things and your relationships change over time.

It’s not as if I’m the first one to figure this out, of course.  But sites that have asked users to proactively decode the real social graph through explicit permissions have failed miserably — the vast majority of consumers will chose not to spend their time programming such a system.  

About three years ago, Microsoft Research launched SNARF which aims to user actual communication history to help users organize things.  More recently, Xobni has gone much further in helping consumers understand more about their social graph derived from actual usage.

There are so many directions that this can go.  One thought — why not use the social graph to build a federated white list for anti-spam purposes?  In addition to the current techniques, such an approach could be used to assess the probability that a particular sender is legitimate and then the existing approaches could be modulated accordingly.

2.  Data mining.

In addition to the relationship data embedded in email there is, well, everything else.  I don’t even bother using desktop search anymore.  Email search (at least with Apple Mail) gives me 99% of what I need.  Most of what matters to me is either in the text of an email or is an attachment to an email.

Yet, it seems like the only time email is mined is as part of a legal discovery process.  Oh, and Gmail’s content indexing for ad targeting.  But there is so much more that can be done!  If something is really important to me, I share it as an URL or as an attachment through email or IM.  Aggregating that data would be immensely valuable for discovery (people who like this URL like that URL) and for search (use this data to separate signal from PageRank noise).

And things will get really exciting when my email data and metadata is mined to help personalize the web just for me.  Now that’s behavioral targeting.

3.  The consumer service bus (CSB).

According to Wikipedia, “a bus is a subsystem that transfers data between computer components inside a computer or between computers. Unlike a point-to-point connection, a bus can logically connect several peripherals over the same set of wires.”  I use the term consumer service bus (CSB) as a play on the enterprise service bus (ESB).  

The idea is that, unlike today’s point-to-point email, CSBs “logically” connect entities with entities.  The first generation of CSBs follow the publish-subscribe model – Twitter, FriendFeed, Google Reader.  These systems pivot around sources to which I have subscribed (a person or an RSS feed) and send me a sequential list of everything these publishers put out.  But data volumes will soon grow out of control.  Today a very small percentage of the internet population use these services.  Once everyone I know is publishing a good chunk of what they do (publishing will need to be made a passive exercise for this to happen), consumers will drown in the noise.  

Facebook’s News Feed is already picking the top ~10-15 “articles” for me from a sea of choices.  I would love to see a CSB emerge that could do the same thing for everything — summarize my favorite blogs and news sources (e.g., Techmeme-style), find the best video on the web (perhaps by email URL extraction from my social network), find the most important emails from my inbox, and let me know tha traffic on my commute home will be especially bad today (because the system has a feed from my calendar and has done a mashup with Yahoo! Maps).

It’s time for email innovation.

I have seen a huge number of startups in the advertising, search, and Facebook applications segments.  Surprisingly, there have been comparatively few in the communications area, despite the obvious opportunity for innovation.  I smell opportunity.

Don’t look now, but the Microsoft’s browser dominance is going away.  According to data from w3schools, it looks like IE peaked in September 2003 at 87% share of the worldwide browser market.  At that time, Mozilla had a share of 6%.  In July 2008, IE had a share of 52% and Firefox (plus Mozilla) was 43%. 

To answer the question of when (if?) Firefox will pass IE, I simply used Microsoft Excel’s extrapolation feature (take a series in a consistent order, drag right corner of last cell to extrapolate).  This is what I got.

According to these crude projections, Firefox will pass IE in May 2009.  And if I’m wrong, blame Microsoft

PS.  Stop developing ActiveX plugins if you can use AJAX or Flash.

PSS.  If you must develop an ActiveX plugin to obviate the browser security model, you better also build a Java plugin so that Firefox and Safari users can use your product, too.

Update 8/23/08:  As TimTrueman points out in the comments section, the w3schools data likely has a selection bias towards alternative technologies.

Last week I wrote a post about the Facebook spam problem.  The malicious thing about this spam is that it’s coming from friends’ accounts, but it’s clearly not being authored by them. 

I have received more obvious spam on Facebook in the past two weeks than I have cumulatively received in the [approximately] past two years.  In addition to the spam on my Wall and in my News Feed, I am now getting personal messages from friends which are clearly not authored by them [below are two that I received today].

A few friends have changed their passwords and have continued to have issues, which is inconsistent with a standard phishing attack (people would need to be phished twice in order for this to make sense — highly improbable).  Are Facebook applications being used as a Trojan Horse for spam?  Has some cracker launched a worm again?  Or have a large number of people installed malicious software on their computers that is logging keystroke information or polling IE for Facebook credentials?

Whatever the case, Facebook needs to immediately take the following steps:

1.  Add a “flag” item in the “edit” menu which is attached to all FB content in the new Facebook design.  They should add a second level to identify the content as spam or inappropriate content.

2.  Send an email to every user discussing the situation and asking for help from the community to catch the bad guys.

3.  Give Max Kelly more resources.

SPAM in my Facebook inbox.

SPAM in my Facebook News Feed.

Update 8/23/08:  Changed the title from “Is F8…” to “Is the Facebook Platform…” to correct the title as F8 refers to Facebook’s developer conference rather than their platform.  Thanks to Ryan Waggoner for the correction.

Ontology, taxonomy, folksonomy.

An ontology is concerned with the categorization of things.  A “thing” can be just about anything — a person, a company, a particular news story, a video, an application, or a web page.  Ontology is a broad concept and subsumes taxonomy and folksonomy. Google Search delivers relevant results because it leverages a massive corpus ontological information about web sites and entities.  PageRank is but one device in the search arsenal that infers ontological information about both the classification and quality of a page.

A taxonomy is a more formal construct concerned with the ordering of things, usually in a hierarchical structure.  Yahoo’s Directory is a taxonomic organization of the web, for example.  The Open Directory Project (DMOZ) is also a taxonomic organization of the web, but rather than leveraging a group of internal “surfers” as Yahoo! did, DMOZ is an open source directory.  

In the early days of tagging, the term folksonomy emerged to describe a non-hierarchical ordering of things through collaborative input.  For example, when a user tags a photo in Flickr, she is creating an informal taxonomy.  These tags are aggregated to help consumers find photos — either by “tag surfing” or by making search more precise as a result of better meta-data.

Who needs philosophy?

Let’s review some of the big wins over the past decade on the consumer internet.

Yahoo!  

Started with a directory built by web surfers.  Employed a chief taxonomist to overlook the organization of the directory.

Google 

As the amount of data on the web grew exponentially, taxonomies will too structured to keep up — a more flexible ontology was required.  Google replaced the directory (a taxonomy) with search (a less formal ontology).

Facebook

Social networks are ontologies.  When you se