"Dark pools" of competitive information

Tim O'Reilly has written two fascinating articles recently on the role of Web 2.0 social networking sites and herd behavior in hedge funds and business. Part 1 is Facebook, the Quant Fund Meltdown, and the Techmeme Leaderboard; Part 2 is More on Techmeme and Financial Markets. To grossly simplify his argument, O'Reilly's claim is that technologies like Techmeme and real-time financial data create a herd behavior where everyone chases the same stories or opportunities. You can see the phenomenon on Techmeme easily; someone breaks a story like "Apple To Ship Mac OS X Leopard On October 26", and within a few hours there are literally fifty commenting stories on that same idea. Everyone ends up talking about the same ideas. And the same thing happens in hedge funds -- as soon as a fund identifies a pricing inefficiency in the markets, it becomes in everyone's interest to exploit it. The result: everyone chases the same opportunity, and it disappears to everyone's loss.

What I find most intriguing is the concept currently the rage on Wall Street of dark pools of liquidity, where price and trading information is hidden. The problem with that technique: when there's no transparency in prices or activities, how do you know the price is the correct one for the underlying information? The answer: you don't. That was one of the root causes behind the credit crunch this summer and promises to be a topic for financial mavens -- and possibly regulators -- to address for months and years to come.

Yet, I think that O'Reilly has hit on a business trend that extends far past Web 2.0: markets increasingly value businesses by those things that make them unique or different, not by the things that make them the same as their competitors. Public data, processes, and knowledge, by their very nature, are rapidly integrated into the woof and warp of the marketplace. Private data, processes, and knowledge, allow the market to identify the unique value of a business, while that business can shift and move to opportunities the market hasn't yet recognized. Private data can be anything from KFC's secret recipe for fried chicken to the proprietary card fraud detection systems at American Express. Privacy means differentiation, and differentiation is necessary (but not sufficient) for business success.

Private data also prevents the business from succumbing to market commoditization. After all, if a business has exactly the same offer as its competitors, the only marketing differentiator it has to work with is price. And when competitors lock horns in a commodity price war, they're locked in a race to failure and bankruptcy, not success. Need proof? Ask Dell how much fun the commodity PC business has been over the past ten years.

This fact is why there's been such a hue and cry about disclosure requirements for public companies; the more public information they release, the lower their business advantage. And it's also why the hedge fund business is very concerned about government regulation. It's not the government they actually fear; most fund managers don't think the government could understand their strategies to say nothing of acting on them. They worry that public disclosure of their private data would erase their trading and money-making advantages over their competitors. And you know what? They're right.

Regular readers know that I believe private data -- outright secrecy might be a better phrase -- is at the root of Apple's success over the last eight years. No one I knew of had any idea that Apple was working on a music player when the iPod was introduced in 2001. Apple's worked on its Macintosh transition to Intel processors for five years before it was made public, yet I would argue it fundamentally reshaped its computer business. Apple was developing the iPhone for two and a half years before Jobs announced it in January 2007. Time and time again, Apple has generated immense publicity and buzz about its products simply because at the time they were introduced, no one knew the market opportunities it was attacking. And since its competitors required years to respond to those opportunities, Apple has captured significant market share in each market it has attacked. Yet, if Apple had publicized those efforts before launch, those initiatives would never have seen such success.

O'Reilly recaps what this means to the Web business at the end of his second article:

What does this tell us about the future of Web 2.0? It tells us that the pendulum is going to swing from public data to private. I'm not sure when this will happen, but I'm quite confident in the trend. (If you see signs of areas where this is already happening, be sure to let us know.) There's still a LONG way to go in making all the world's information accessible, but maybe one sign of the maturity of the Web 2.0 trend will be when more companies are started on the premise of keeping information hidden than on building publicly accessible repositories that grow through user contribution.

The only thing I can add is that this isn't just a Web 2.0 effect; the Internet's free flow of business information is reshaping every business on the planet. IBM has probably done the best job of articulating this issue with its "What Makes You Special?" series of ads and case studies around innovation. But the real leaders will likely be those that aren't in IBM's ads. Why those companies? Because their competitive advantage will be a secret, not a case study.

UPDATE: for a contrary view claiming that Wal-Mart's recent swoon is the result of too much IT differentiation when it should be using commoditized software and systems, see this recent article by Nick Carr at his Rough Type blog.

Full disclosure: the author owns Apple stock.

Technorati Tags: Apple, Credit Crunch, Dark pools, Finance, Hedge funds, Marketing, Markets, Nicholas Carr, Tim O'Reilly