« October 2007 | Main | December 2007 »

November 28, 2007

Introducing The Long Tail

The book I am going to attempt to introduce on this blog is The Long Tail: Why the Future of Business is Selling Less of More by Chris Anderson copyright 2006. Anderson is the editor of Wired Magazine. He maintains a blog of the same name here.

Part #1: The Basic Framework

Many of the readings we have come across in this class have noted the prevalence of long tail distributions. Pareto distributions, power laws, and Zipf distributions are all examples of what can be called a long tail distribution. Graphically speaking, a long tail looks something like this:

You can see where the name comes from. That line that goes off to the right looks like a long tail. Notice that if you graphed sales on the vertical access with product type on the horizontal access, a relatively small number of products account for large portion of the overall sales. This is the short head. Anderson uses the term Long Tail to refer to that area off to the right; the area where you have tons of products that sell very little. This book considers the rising importance of the Long Tail in the commercial world.

Now a couple of basic concepts in the Long Tail

Hit vs. Niche
The short head is composed of “hits.? In the context of music, the hits are those tracks you will find on the shelves at a store like Wal-Mart, which apparently carries around 60,000 tracks. There is a whole ton of music available every year that does not fall into this category. For example, at the time the book was written, online music provider Rhapsody maintained an inventory of 1.5 million tracks. These tracks include many hits and many more tracks that are not hits—they comprise niches of every kind. More interesting than just carrying all the tracks (at least to me) is that virtually every track will sell at some point. The point: niche markets and niche products characterize the Long Tail.

Scarcity vs. Abundance
Economics of scarcity fuel a hit-driven world. Think standard movie rentals. Scarce shelf-space at a Blockbuster means that only a limited number of movies can be carried. These movies must pay their way and must appeal to a sufficiently broad local market to justify their existence. This requires that the movies be relative hits and severely limits the potential of niche movies to reach niche markets. Economics of abundance do not have the same effect. Think Netflix. There is no scarce shelf space or inadequate local market placing severe limits on the numbers of movies available in the Netflix catalogue. As a result, Netflix carries a much broader selection of movies and serves many more niche markets than a bricks-and-mortar movie rental place. The point: when scarcity becomes less of an issue (or a non-issue), niches thrive. Of course, the Long Tail is characterized by abundance.


Part #2: Central Idea of The Long Tail

Anderson summarizes the theory of the Long Tail as this:
“Our culture and economy are increasingly shifting away from a focus on a relatively small number of hits (mainstream products and markets) at the head of the demand curve, and moving toward a huge number of niches in the tail. In an era without the constraints of physical shelf space and other bottlenecks of distribution, narrowly targeted goods and services can be as economically attractive as mainstream fare.?

In other words, the Long Tail can compete with the hits. But how does this happen?

The world of online retail and distribution move us from a world of scarcity to one of abundance. The costs of connecting supply and demand shrink in this world enabling an offering of far greater choice to the consumer (more niche products). Offering niche products helps reveal latent consumer demand. This demand in turn improves the economics of offering these products creating a feedback loop that expands the niche market itself. This world is not a world of mass markets, but a mass of niche markets.

The initial key to enabling a Long Tail , then, is reducing the costs of reaching niches. In this vein, there are usually three forces at work:
[1] democratizing the tools of production
[2] cutting the costs of consumption by democratizing distribution
[3] connecting supply and demand

I will take each of these in turn.

Democratizing the tools of production - “Give enough people the capacity to create, and inevitably gems will emerge.?
Cheap and ubiquitous technology spreads the tools of production broadly. For example, a PC and an internet connection makes everyone a publisher. This spread of technology changes people from passive consumers to active producers resulting in a stunning rise in the amount of stuff produced. Some of this stuff is bound to be good as the technology crosses the paths of creative, visionary, and talented people.

As a side note, the spread of the tools of production to so many people alters the landscape of information and knowledge. The accuracy of information in a system characterized by peer production is not guaranteed by a typical top-down approach. In the top-down approach, editors and other professionals verify information in an attempt to make it accurate. In the peer production world (Wikipedia is a great example), accuracy or intelligence is an emergent property. These are systems governed by probabilistic statistics – accuracy is measured as a probability based upon the wisdom of the crowd. “[T]here is only a statistical level of quality.? There will be some terrible information. But the real miracle is that the whole does not fall apart. Instead there is a breadth of information that is simply unmatched by any top-down approach, especially in the way it is unbounded by any space, production, and time constraints – it is constantly updated, self-correcting, and self-healing. (I just thought this was an interesting thought he had).

Democratizing Distribution – The Internet makes everyone a distributor
Of course more content is only better if it can be consumed. So we need cheaper distribution. Of course the internet provides just this (note: however that many other things combine in the cheap distribution of physical goods including massive warehouses, assembly lines, and other parts of the increasingly efficient supply-chain. Anderson argues that the internet really just brought a bunch of these innovations together and coupled them with a e-catalogue that is virtually infinite and free). The web makes everyone a distributor. This democratized distribution gives rise to a new kind of what Anderson calls aggregators.

An aggregator is “a company or service that collects a huge variety of goods and makes them available and easy to find, typically in one place.? Aggregators lower the costs of selling things allowing more to be sold. Traditional aggregators deal in atoms (physical goods) – a retail store is a good example. The web allows for two new types of online aggregators: the hybrid retailer and the pure digital retailer.

A hybrid retailer crosses physical and digital economics – they deal in atoms and bits. The goods still come through some physical delivery system. The costs are lowered because these retailers can use centralized warehouses and an online catalogue that can list virtually infinite goods. A pure digital retailer, on the other hand, deals only in digital economics – only bits, no atoms. Products are a database entry, distribution is broadband megabytes incurred only when a product is ordered. The result is “near-zero marginal costs of manufacturing and distribution.?

The point is that both of these aggregators have a reduced cost of delivering a product to the consumer. This makes Long Tail products more viable.

Connecting supply and demand: driving demand down the Tail/lowering the search costs of finding niche content.
More content and cheaper distribution is all fine. The problem is that the shear breadth of content is quite overwhelming, and, frankly, lots of it just not that good – this could result in all of it looking like noise. Enter the third force: elements that drive demand down the curve connecting/exposing people to what they might like.

Connecting supply and demand is not a new thing. In fact, isn’t it the goal of traditional marketing? The nature of this drive does appear new, however.

Traditional marketing relies heavily on top down advertising to drive demand. Driving demand down the Long Tail in our heavily digitized world seems to happen more by peers trusting peers. Amplified word of mouth is the manifestation of the 3rd force. The new “tastemakers? are those peers who have respect of others. These people are tools in connecting supply and demand in the Long Tail.

The general word for tools that help you find quality in any Long Tail is “filters.? The person at Universal Studios who selects what movies to make is a filter just as a search engine that returns results with more links to it first is a filter. The difference between the two is a matter of pre vs. post-filtering. Pre-filters predict the market. Post-filters measure and channel consumer behavior and amplify it. The Long Tail thrives in the post-filter environment.

Part #3: Potential Impact of the Long

We live in a Long Tail world, so what? In a commercial and consumer sense, a couple of changes should occur. Hits will matter less as consumption of Long Tail products matters more. For consumers, in theory we will be better off. When hits ruled, we were subject to the lowest common denominator. In the entertainment context, we had access only to things that appealed to the broadest number of other consumers. Now we can access the niches. In some areas they should contain things that we really love that most other people really hate. That’s cool.

Outside of the commercial world, this theory provides neat way of looking at things. You can see the Long Tail of everything. Green and Libertarian parties as the Long Tail of politics, home-schooling as the Long Tail of education, and maybe terrorism as the Long Tail of violence (really big violence used to be the realm state actors). It seems a compelling argument to say that the same technology that aids commercial Long Tails has a similar potential to bolster the non-commercial Long Tails I just mentioned. I would go further and say that the Long Tail provides a crucible for creating things that can take down a hit. This is powerful stuff and perhaps a different perspective if you are in to change.

That’s all I got – please let me know if you have questions or seek clarifications.

November 18, 2007

Fun With Attractors

I've been working with iterative functions and have come up with some custom software using Jitter that explores these iterated functions as their parameters vary with time. Check out what I came up with....

jitatt-animate.gif
FAKE-animate.gif
att-animate.gif

Each is generated with the following set of iterative equations:

x2 = sin ( a * y1 ) - cos ( b * x1 )
y2 = sin ( c * x1 ) - cos ( d * y1 )

The top two videos interpolate values of all four variables (a,b,c,d) between two known states. The bottom one only interpolates the value of d between two known states. As you can probably see the functions seem to drop frames. This is due to parametric states that produce fixed-point attractors where the equations produce relatively few possible coordinate states. This is equivalent to the bifurcation model where we observe the critical bandwidths, but here these are represented in 2D+Time.

November 17, 2007

Presentation on "Critical Mass" by Philip Ball

Download file

Here are the slides from my presentation on 11/14.

Shireen Stone

November 14, 2007

golly

Follow these links to download golly.

Windows

Mac

You can also go to the golly site.

November 9, 2007

New Cool Cellular Automata Site

Here is a generalized applet for exploring different cellular automata:


http://jmge.net/java/csprings/

Its way more functional than my previous examples. Cheers, Jeremy

November 6, 2007

Complexity Illustrations

I found the course website for a class last spring at Southern Oregon University on Complexity. The site has a lot of links to computer applications and illustrations related to concepts we have discussed this semester. There's among the great links, one is to an advanced version of Conway's Game of Life that I will be showcasing in class this week.

Chaotic Attractors Application

I ran across this applet while perusing the "intertubes" --as the kids are saying these days. This little java applet iterates the following function to produce pretty images:

x2 = sin ( a * y1 ) - cos ( b * x1 )
y2 = sin ( c * x1 ) - cos ( d * y1 )

...in the application you can set the values of a, b, c, and d or you can let it run at random. Its a fun little toy for working with iteration. Download it here:

http://attractors.sourceforge.net/

Natural v. Explanatory Simplicity

Hi all:

There's been a convergence in my class notes over the last few weeks that I wanted to share, surrounding two approaches to complexity studies. David brought this up first, with his idea of 'Natural' v. 'Explanatory' simplicities.

As I transcribed it, there are two methods for creating models and theories (simpler explanations) of complex systems. The first, 'Natural Simplicity', is epitomized by Stephen Wolfram's extremely bottom-up methodology, creating rules and taxonomies of order from the bottom up using CA behavior. The characteristics of this approach is that is doesn't over-limit the kinds of behaviors its models. It attempts to realize every possible combination of CA models, systematically in Wolfram's case, to grasp the entire landscape of possibility (phase space) of a system. This approach is highly empirical and inductive, prescribing the outcomes as loosely as possible.

On the other hand, 'Explanatory Simplicity' might be something like Robert May's model of predator/prey relationships, a model that comes with many more sets of assumptions. The simplicity here lies in the way in which a single approach can lead to dramatically complex (and elegant) behavior. (May's famous equation was the one with many bifurcations that had periods of intermittency and fractal scaling.) Here, we are being slightly more deductive than before, and 'deducing' using theoretical models (empirically grounded earlier, hopefully) to demonstrate the relationship between the simple and the complex.

I mention this because Luke's description of computational mechanics (and attempting to find the predictability of the wind) seems to fall along this theoretical/deductive v. empirical/inductive complexity divide, as well. As I understand it, computational mechanics is an approach that makes very few assumptions about the mechanisms of the underlying system, and simply tries to look at the order within the forward and backward linkages in the system. In other words, it simply asks how predictable a set of data may be, without constructing a model of how these changes (behaviors) occur.

So, I'm throwing this out there to see whether anyone else thinks this contrast is interesitng. Is it a difference that makes a difference? Clearly, one approach requires far more computing power... but apart from that, is it always a good idea to be as inductive as possible? Do we lose something? Do we lose the causal connections that we find in more tightly proscribed models?

And finally, how would these two approaches relate to the kinds of social systems many of us are thinking about?

Bonus: Order & Chaos on the Streets of Berlin

The Strib's Transportation journalist, on his 'Roadguy' blog, recently tried to use 'order' and 'chaos' as a lens to view bicycle traffic in Berlin, Germany.

Here's the link for you to click!

The seeming chaos of thousands of bicycles becomes a vast, highly ordered system because everyone is following simple rules.

What do you think? Is this is the kind of thing that is interesting, or merely annoying?

November 3, 2007

Interesting

An interesting blurb in the October issue of Harper's: "Physicists at the Max Planck Institute for Extrateraterrestrial Physics announced that electrically charged dust can organize itself into double helixes resembling DNA; these objects, known as plasma crystals, also exhibit some of the characteristics of living things."

(Dust to dust, and back again?) -- Bill Comiskey

November 2, 2007

"At Home in the Universe" Presentation

For those of you who would like to access the "At Home in the Universe" presentation from last Wednesday's (October 31) meeting, here is a download link (it takes a minute or two):

Download file