Content Log

Scaling Out Like Technorati

My fellow World Economic Forum Technology Pioneer, David Sifry, the founder of Technorati, was also in Dalian, China for the “Meeting of New Champions” or “Summer Davos” as the Chinese like to call it. During Davos in January, I had the great misfortune of pitching Alfresco against Technorati in a competition between tech pioneer companies. As fantastically well as Alfresco is doing, Technorati has the temerity to compete against Google in blog search and win.

I got the chance to talk to Dave during the conference and ask him some questions on the technology and architecture behind Technorati, the internet blog search site. I thought that someone who could take ordinary computer components and build a huge internet architecture could possibly teach something to people running enterprise architectures that are puny in comparison.

Technorati is a web site that tracks blogs, pictures and any user generated content and allows you to search those sites about what people are thinking, seeing and hearing. When a new or urgent situation breaks out, you can do worse than to search Technorati for immediate reaction. Every day, every hour, every second, Technorati is indexing over 10 million blogs with over 10 billion objects. Technorati’s user base is doubling every six months and quick and accurate response is critical for retaining those users.

David Sifry, Founder and Chairman of Technorati

I asked Dave about his architecture and what applicability their might be for enterprise architectures.

John Newton: In building Technorati, what were some of the issues that you had in architecting your systems.

David Sifry: I was looking at just temporal information. I had no idea how big it could get. When I looked at the architecture, instead of architecting it right, I architected it for right now. I had no big budget and I didn’t want to wait six months to build it. Also, I had no idea what the killer app would be.

I focused on data flexibility. At the time, that meant putting everything into a relational database. That was okay while the size of the indexes is less than RAM and about a million blocks of data. That was okay while there were less than 20 million blogs.

The next generation took advantage of data parallelism. That meant upon update send a signal to all the other systems. We expanded the data over several “shards” [segments of data partitioned on different databases on separate machines].

What was surprising was that we were writing as much data as we were reading. At this point Technorati was as big as some of the biggest OLTP. Even so, maintaining data integrity was important, because you would want the link count [count of how many other blogs point to a particular URL] to be out of sync. This put real pressure on the system. At the same time, we realized that time was more important dimension than URL. People didn’t want to sort or search on URL, they wanted to search on time. [i.e. what are the latest blogs on a particular subject?]

By this point, we understood the application more and more. The app [Technorati] is about real time access. You need to be able to count on finding latest information on a subject. That’s when we built the third architecture. Scaling was well understood and we build the shards on time rather than on URLs. Instead of putting data into a DBMS, we put it into special purpose databases. It was more of a bus-based architecture. Each database could be scalable and grow as big as we needed.

JN: The notion of shards - did you call it that at the time? I have been looking into shards and I was only aware of or heard of them for about the last year.

DS: Back in 2002 when we were pitching this to VCs, I at least explained the theory. All I just thought through the problem carefully. Doing it this way, we could add hundreds of systems, lots of cheap CPUs, RAM and disks. It provides inherent parallelism. I can’t believe that I was the first one to think this up.

JN: How big does this architecture scale?

DS: We are loading one terabyte a day into Technorati. That’s 100 million blogs or about 10 billion objects. A lot of is new types of tagged data. There are about a half billion videos and photos.

With all that data, you have to think about what do you throw away?  We can’t really delete anything, because we are potentially losing an asset. We don’t delete anything. So we take data out of the spin cycle. [Transitory data used in preparation.] We take the long-term data and put it into low latency storage.

When data is doubling in size every six months, that means that only one quarter is a year old. We don’t need to worry old data.

JN: How do you deal with large number of users with very large data sets?

DS: Any off the shelf tools falls over. There is a lot of interesting analysis on old data, but no off the shelf tools can handle that much data. It’s only just now that some tools can handle it.

JN: What are those tools?

DS: One is Green Plum by a bunch of O’Reilly guys. If you use ordinary data warehouse tools, they would just scream and shout.

JN: Actually what I was originally referring to was the fact that you are showing lots of data that are not users used to enterprise information management tools. How do you present this information to consumer-level users? How do you deal with the user interface and visualization of all this data?

DS: Gotcha. It depends on what the user wants to get out of Technorati. If the user wants search results, then we give it to them. Sometimes they want to browse or discover information. We have spent a lot of time on visual design. Then we give them lots of bright, shiny things for them to click on.  Things like metadata, video or other links.

We have used enterprise class web tools to analyze what users are doing? We look at the click stream and see what is successful or not. That helps to make the information contextual.

One of the big mistakes that we made is to not do this [buy click stream analysis tools] sooner. It was only $80K. Up to that point it was so much trial and error. I’m glad we finally did it. Now we can see how much time a user spends on a feature. We can see page views, goals per visitor.

JN: So what do you measure on Technorati?

DS: Measuring a web site is like forecasting the weather. Yesterday it’s sunny and today it is cloudy. Why is it cloudy?  Sometimes you have no idea. Sometimes you realize that that a change in barometric pressure has a lot to do with it.

We look at the number of newbies, number of reports, session lengths and then measure them against prior periods. It’s not always consistent.

I had never built a B2C site before. I just focused on me, on what I wanted. That worked well for a while when I was the target audience. But we have to build for a broader audience.

JN: At Alfresco, we measure conversions. Are you measuring things like performance? Does that affect retention of users?

DS: Of course, but if the system is falling down, then even performance doesn’t matter. So I don’t get too stressed out about it.

JN: When we met at Davos you wanted to move Technorati to be the Internet Now! Is that still the case?

DS: Everything is shifting. I wanted it to be a site that everyone is able to use. We forgot about the core users that just wanted to find out about blogs and any real time information. In an attempt to jump the chasm, we chased after 100 million users and tried to be everything to everyone. Now we try to make blogs and user driven content available for those looking for that.

Also performance is improved significantly. Now I notice how slow other sites are. This is a total tribute to the engineering team. Everything is easier and faster.

Pretty soon we will have a whole lot of stuff that we have been working for a year.

JN: Can you say what it is?

DS: I don’t pre-announce.

JN: What does the Technorati brand stand for today?

DS: Good question. What’s popping up now on the internet, especially user generated content? It’s about users tagging user generated content and finding it.

JN: Who are your competitors?

DS: I probably sound like the typical entrepreneur, but nobody really seriously. Google provides blog search, but other than that nobody really. Other people are trying to identify and tag information like Digg and del.icio.us, but they aren’t really competition.

JN: What do you want Technorati to be in two years time? Five years would be ridiculous.

DS: I would like Technorati to be a profitable business that is strongly differentiated. It will be the place that you would go for mobile, RSS or push information. For all that you would come to Technorati.

Blogging on ZDNet is Hard!

Toward the beginning of this year, blogging got a lot easier for me. Rather than look for things to blog about, I would just jot down something that I found interesting and then blog about it the next day. I would think about something in the middle of the night or even more frequently in the shower. When I got into the office in the morning, I would just start typing away having enough caffeine momentum out of my first cup of coffee.

The opportunity came up to blog on ZDNet and I thought that it would be easy. I just do what I was doing before, but I would have a bigger audience. Well, it has turned out a lot harder than I thought. Rather than being easier, I took their blogging guidelines to heart and endeavored to be as neutral and unbiased as possible. I also strove to have a theme that could encompass content management, but appeal to a wider audience. Rather than just write what comes to me, I started to look for subjects and read blogs for what might fit my chosen subject of Information Management. What had become a liberating activity had tied my frontal lobes into knots with inhibition. This is slowing my blogging substantially on both on ZDNet and on this blog.

Nothing illustrates this more than my latest blog on Microsoft’s initiative to spread its Open Office XML standard around the world. This started on the 4th of July with a conversation with Ian Howells, but it has taken me until today to write it. My natural inclination was to write the obvious, which is that Microsoft is scared of ODF and OpenOffice and it is using its massive power and installed base to force governments to reckon with it. Instead, I ended up doing a substantial amount of research on both sides of the argument to present a balanced view. I learned a lot about ODF and OOXML as well as XPS by writing a more thorough piece, but it took me a lot longer than expected.

Back in 1988, I made the mistake of trying to do enterprise sales myself rather than just supporting the sales people. As a result, I got a much better appreciation of what sales people did and how valuable they are to the sales process. Engineers in general are somewhat dismissive of what sales people do. Just read any Dilbert cartoon. Although I didn’t start with quite the same lowly opinion of sales people, I have come away with a greater appreciation of what journalists go through. I am clearly not a journalist nor am I an analyst.

Most people blogging on ZDNet are journalists, reporters or analysts whose full time job it is to analyze the industry and to comment on it. I have a great deal respect for most of those writing and particularly like Mary Jo Foley’s column on Microsoft and Dan Farber’s broad range of subjects. Some people are full time employees at vendor companies and some appear to even begin to hold an unbiased opinion on what it is they are writing about. I have a full time executive role, have been in the industry a lot longer and I am attempting to have an unbiased opinion.

One of my professors at Berkeley was Laura Nader, Ralph’s sister. I took an Anthropology course from her where she would constantly opine about the nuclear industry. Taking a similar belligerent attitude as Ralph Nader, she would say that it made no sense to try to be objective, because we all have opinions. She would then use a big chunk of the lecture to rail at nuclear power, rather than talk about cultural anthropology. She had some points to make about nuclear power, but I couldn’t have disagreed with her more on the importance of objectivity. I will keep slugging away at both blogs, but probably focus more on this one. However, I will try to keep the objectivity factor in mind, while keeping the spontaneity of what’s on top of my mind.

There! That only took 20 minutes to write rather than 3 days!

Try My New Blog "Newton's Theory" on ZDNet

I have been invited to blog by ZDNet on their blogs site and it is a real thrill to be involved with the blogging on such a widely read and heavily trafficked site. The new blog, called Newton's Theory, is a more general and balanced look at the larger information management market, although with an open source bias. Well, open source is the future of software after all. ;-)

My first blog post was on a Theory of Information. It is something that I think about a lot and someday I might write a book about. My latest post, REST battles SOAP, is about how information will be consumed in the not too distant future, which has relevance in a post iECM world.

I am still going to blog on Content Log and post back relevant ZDNet entries to this blog. So I hope you subscribe to both. Also, anything you are interested in me posting, please let me know.

Normal procedure is to be paid for blogging, but I plan to give 100% of any advertising revenue I receive to  Nicholas Negroponte's One Laptop Per Child. I saw him present and got to speak to him while I was a Davos and what he said convinced me that this could really change the world.

Googling my Blog

When I write my blog, I tend to write about what is at the top of my mind first thing in the morning. Usually it comes to me in the shower. I am not targeting Google when I write these entries. However, sometimes I find that my blog gets Google hits more often than not for things that I didn't intend. It says a lot about Google and its algorithms than it says about my blog. It also says a lot about what content is important in a new media world of participation. It also shows how important a title for reaching people with your blog.

Here are the top blogs that get unintended hits as a result of Google searches:

1. Theory X, Theory Y and Theory  - This was originally written as a result of a conversation. However, it showed me that Theory X and Theory Y are not dead, since people are looking it up every day.
2. What the heck is Microsoft Sharepoint 2007? - Apparently, I am not the only person asking this question. This blog gets hit with a number of different combinations of Sharepoint 2007 and other related terms. Closely related and almost hit as much is Heard on the Street: Sharepoint Scalability. Seems people are constantly searching for how scalable Sharepoint is.
3. An American Entrepreneur in Europe - This gets hit whenever an American is looking for a job in Europe or people want to know how to be an entrepreneur in Europe.
4. Katie Couric's Blog - Well, this was a shameless attempt on my part to build traffic. As soon as I saw that she was going to start a blog, I thought this was my chance to get hits on it. Little did I know that my fears expressed here were well founded and that she (or really her staff) have been accused of plagiarizing other people’s material. Her “page from my notebook” posts have turned into YouTube-like, tepid sound bites with no depth.
5. SQL vs. Xpath vs. XQuery - A Query Language for Content Management - Seems lots of people want to compare SQL and XQuery as well as Xpath. It is still a hot topic of debate in the JSR-283 committee. It is a typical the potential future vs. the strong legacy debate.
6. I want my Joost TV - I was only going to do the top 5, but I have been intrigued by the response to this post. I only recently posted this, but it shows how hot Joost is going to be.

Special mentions go to Game Theory and Open Source because Game Theory is still an academic subject and Is Crossing the Chasm Dead? because Geoffrey Moore, who I greatly admire, has such a powerful impact on Silicon Valley. Short term Google hits came from my posts on the departure of Marc Fleury and Dave DeWalt where my blog was a high up on Google when searching their names. I am also quite happy that when you Google my name, John Newton, it shows up on the first page despite the fact that there is a more famous John Newton.

Shai Agassi and the Long Tailpipe

It's good to see that Shai Agassi has found an blogging outlet now that he has left SAP. I found it by accident when somebody was searching my blog for the Geek Dinner at Davos. Shai's blog was probably the funniest moment during the dinner. The video that Shai mentions doesn't seem to be on AlwaysOn's web site. That is probably because I don't think Tony actually got permission before he brought his cameraman into the room. I was at the same table as Sergey and Shai, so I saw the whole thing. The trick is to take 3 glasses widely separated, 3 butter knives and rest the bottle on top of the knives. The answer looks something like this:

Shai is apparently off working on things environmental now and I guess still in enterprise software. While we are at Davos, we were in the same working group on what life would be like in the year 2015. Shai discussed what I thought must be one of the most innovative ideas I have heard in a decade - a physical analog to the internet. In other words, a world-wide mechanism that handles the logistics of physical things the way the internet handles packets. You don't care how things get there, you just send them. In the process, objects can actually be assembled and packaged as well. Having earlier come from a session on city design where Larry Page, the other Google founder, envisioned a world where "if I want a tomato, I just ask for it and it arrives a few minutes later," I could see a perfect meshing of these two visions. In fact, from my phone, I could arrange to have a physical good routed to my phone, no matter where I am with that phone. Thinking about it later, the whole notion of transportation could revolutionized with a physical goods infrastructure that didn't even need people or roads. The possibilities are too endless to mention. I hope this is a concept that Shai explores with his blog.

It is also a good time for him to ponder his previous reluctance to open source. When I was at Documentum and even immediately after, I just didn't get open source. Like Bill Gates, I compared open source to communism. It took Marten Mickos from MySQL to explain it to me before I actually got how you could make money with open source. Shai has not been quite as strident about open source, but he hasn't exactly been embracing. Perhaps freed from the mind set of SAP, it will be easier to see the opportunities that open source make possible. In fact, although I think the idea of a Physical or Logistical Internet would be brilliant, I don't think it would be possible without large portions being driven by open source concepts, just like the current internet is.

I wish him luck in his future endeavors.

How You Read My Blog

I recently wrote a blog asking how you manage the blogs that you read. Thanks to everyone who contributed advice, which was very helpful. As a result of that advice, I was led toward publishing my blog using the feedburner service. This allows me to push content to multiple channels and to track how my blog is being subscribed and read.

I was actually very surprised at how many people subscribe to my blog. Feedreader provides me with the type of blog subscription tools people are using and from what country they are reading it. But don't worry, it doesn't identify who is reading the blog. If you are interested in subscribing, you can subscribe at this subscription link.

The primary tools people use to access Content Log are Google Reader or Homepage (28%), Bloglines (17%) and a bit of NewsGator (8%). I thought most people were accessing it mainly through Google search, but many of the links in Typepad do not show the source of link.

Here is the list of all the different types of feed readers that people are using and feedburner's description of those feed readers:

• Akregator  -- Akregator is an open source aggregator that you can download at Sourceforge
• Apple CFNetwork Generic Client  -- Most of the time, this the Apple OS X screensaver, which acts as a desktop feed reader. This user-agent can also represent other independent software that used Apple frameworks to build a feed reader.
• Attensa for Outlook  -- Attensa for Outlook is an RSS reader that brings up-to-the-minute news and content from Websites, blogs and Podcast sites directly into Microsoft Outlook.
• BlogBridge  -- BlogBridge is a desktop feed reader for Windows, Linux and Mac. BlogBridge provides a full suite of feed reading tools including feed search, discovery and categorizing capabilities that are unique to BlogBridge
• Bloglines  -- Bloglines is a web-based aggregator that makes it easy to keep up with your favorite blogs and newsfeeds. With Bloglines, you can subscribe to the RSS feeds of your favorite blogs, and Bloglines will monitor updates to those sites.
• FeedDemon -- FeedDemon enables you to quickly read and gather information from hundreds of web sites — without having to visit them. Don't waste any more time checking your favorite web sites for updates. Instead, use FeedDemon and make them come to you. Written by Nick Bradbury, creator of TopStyle and HomeSite, FeedDemon makes RSS/Atom feeds as easy to access as your email.
• FeedReader  -- FeedReader is a freeware Windows application that reads and displays RSS feeds. It supports the RSS formats 0.9, 0.91, 1.0 and various extensions such as Dublin Core. FeedReader does not support Atom.
• Firefox Live Bookmarks -- "Live Bookmarks is a new technology in Firefox that lets you view RSS news and blog headlines in the bookmarks toolbar or bookmarks menu.
• Because of how the browser operates, version 1 of Firefox might be overstating the number of Live Bookmarks subscribers and some casual visitors may be counted as subscribers. This is fixed in version 2 and will show up as a separate entry called ""Firefox Live Bookmarks""."
• Firefox Live Bookmarks  -- Live Bookmarks lets you view RSS news and blog headlines in the bookmarks toolbar or bookmarks menu. With one glance, quickly see the latest headlines from your favorite sites. Go directly to the articles that interest you - saving you time.
• Google Desktop  -- Google Desktop doesn't just help you search your computer; it also helps you gather new information from the web with Sidebar, a new desktop feature that shows you your new email, weather and stock information, personalized news and RSS/Atom feeds, and more.
• Google Feedfetcher  -- Feedfetcher is how Google grabs RSS or Atom feeds when users subscribe to them in Google Reader or the Google Personalized Homepage. Subscriber counts include Google Reader and the Google Personalized Homepage. Feedfetcher collects and periodically refreshes these user-initiated feeds, but does not index them in Blog Search or Google's other search services.
• GreatNews  -- GreatNews is a downloadable RSS desktop client for Windows. GreatNews is optimized for "full page reading" so that you can scan through a number of articles quickly without having to navigate between feed items.
• Gregarius -- An open source feedreader for inclusion in web sites
  
• intraVnews  -- Keep up with hundreds of websites and have RSS and Atom news items delivered into your Outlook folders, as easy as email! With the powerful interface of intraVnews you get total control over your feed subscriptions on the Internet and your intranet.
• Java-based feed reader  -- There are a number of feed readers, some quite sophisticated, some hacked together, that only identify themselves as having been built in Java and provide no further information about their identity. Thus, this particular "category" of feed reader is a catch all for one or more readers that are only partially identifying themselves when accessing your feed. If you know people are accessing your feed through a client that you don't otherwise see in this list of clients and aggregators, it's quite likely that the client is either sending no identifier or using this catch-all java identifier.
• Liferea  -- Liferea is an open source feed reader for Linux
• Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) --
• My Yahoo  -- A web-based newsreader that allows you to select and manage RSS headlines within a My Yahoo! account.
• NetNewsWire  -- NetNewsWire is an easy-to-use RSS Web news reader for Mac OS X. Its familiar three-paned interface - similar to Apple Mail and Outlook Express - can fetch and display news from thousands of different websites and weblogs, making it quick and easy to keep up with the latest news.
• Netvibes  -- Netvibes allows you to create a personalized home page incorporating different kinds of weather, sports, news, product content and RSS/Atom feeds
• NewsGator Online  -- NewsGator online is a free web-based aggregator with a clean user interface that makes it easy to arrange and track your favorite news feeds. Newsgator supports all the major feed formats and has a number of additional features for ranking feeds, getting recommendations, and permanently saving specific feed items.
• NewsGator Outlook Edition  -- NewsGator is one of the most popular feed readers. It runs within Microsoft Outlook. The latest versions of NewsGator provide the ability to read feeds on multiple machines running Outlook with full synchronization.
• Omea Reader  -- Omea Reader is an easy to use, all-in-one RSS/ATOM feed reader, newsgroup reader, and web bookmark manager. But what really makes it unique is the level of information organization and management features including lightning-fast searches, flexible filing, contextual access, and extensibility.
• Onfolio -- Onfolio 2.0 includes a number of new features including a feed reader, Firefox integration, blogging support, shared collections, improved capture, Outlook integration, EndNote integration, folder publishing, and even more.
• Outlook 2007 -- "Using Microsoft Office Outlook 2007 to subscribe to an RSS Feed is quick and easy and does not involve a registration process or fee. After you subscribe to an RSS Feed, headlines will appear in your RSS folders. RSS items appear similar to mail messages. This number may be inflated, since some requests from Internet Explorer 7 also identify themselves in this manner."
• Rojo  -- Rojo (pronounced like Mojo with an R) is a web-based service dedicated to helping Internet users efficiently manage online content and information flow.
• RssReader  -- RssReader is a free news reader for RSS and Atom feeds. Features include support for authenticated feeds and OPML import/export.
• Safari RSS (OS X Tiger)  -- Safari RSS is the built-in news aggregator found in Apple's Safari web browser, starting with the version found in Mac OS X "Tiger." More information can be found on the Apple Safari RSS page.
• SharpReader  -- SharpReader is a popular feed reader for Windows that handles all RSS versions and Atom. SharpReader has a number of powerful features like support for HTTP Authentication, making it possible to subscribe to authenticated feeds, drag-and-drop feed subscription, and keyboard navigation between content items.
• Thunderbird  -- Thunderbird is Mozilla's next generation e-mail client. Thunderbird makes emailing safer, faster, and easier than ever before with the industry's best implementations of features such as intelligent spam filters, a built-in spell checker, extension support, and much more.
• Topic Blogs  -- Topic Blogs is under construction as of Autumn 2005
• Vienna  -- Vienna is a freeware, open source RSS/Atom newsreader for the Mac OS X operating system
• Windows RSS Platform  -- This represents a subscription from the Microsoft Windows RSS platform. The actual client can vary, but your feed is being regularly checked and read.
• WordPress  -- Wordpress bot

Ian Howells: With Open Source you can't be half-pregnant

My colleague, Ian Howells Chief Marketing Officer at Alfresco, has posted a blog on how Geoffrey Moore's marketing models describe what is happening with the commoditization of the ECM space and how the network effects of open source are accelerating the pace of commoditization of enterprise software. He quotes Geoff:

"Enabling technologies commoditize extremely well, allowing them to proliferate into markets far afield from the original starting points and generate a high degree of network effects. These in turn put pressure on the overall marketplace to standardize exclusively on a single set of components driving market shares to extraordinary levels …”

Ian sites Alfresco move to GPL as embracing this network effect and that companies should embrace open source whole heartedly. Ian says:

"You can’t be 'half-pregnant' and similarly you can’t be half open source and half proprietary. It is open source, not hybrid models that will drive true disruption, commoditization and benefit most from the network effect. The GNU General Public License (GPL) is the ideal license to drive forward this industry disruption and accelerate the network effect. That is what drove us to move to GPL."

How James McGovern can improve his blog

James McGovern recently asked how his blog can be improved. I would point him to this article on enlightenment on his path to his ultimate blog:

Reaching Blog Nirvana (courtesy of ValleyWag)

What his and every blog can use is more sex and celebrities and more vitriol and schadenfreude poured onto vendors other than us. Use Britney Spear’s bald head as a metaphor for the decline in morality of traditional enterprise software vendors. Use Anna Nicole Smith as an object lesson in the excesses of greed and investment in new, shiny technology.

Seriously, James’ blog is an extremely thoughtful discourse on the needs of a large enterprise in its use of technology, techniques, governance and best practices in enterprise architecture. There is a lot a like, such as the length, funny images and directed questions. I learn a lot from his blog and his task is a difficult one, but here’s what I would like to see:

• Your blog disassociates itself from your employer, but that shouldn’t prevent your from presenting use cases that we can actually use to build product
• As abstractly as possible, what are the application domains that you are tackling and what role does new technology, such as some of the new Web 2.0 like AJAX, REST, tagging, etc. play
• You have blogged some good things about open source, but where are you actually use it? Where do you draw the line of open source vs. traditional?
• You posted an article on user advisory boards. It seems to me that blogging may be a more potent form of user advisory board. How about trying a user-led versus vendor-led user advisory board organized through the blog
• Write about how you find time to blog - this is one of the hardest problems that I have
• More tagging and tag clouds to find the posts that are relevant to me
• Some insight into your purchase process and purchasing decisions - very important for vendors
• My blog on your blog roll
• A post about how you are going to buy a lot of support for Alfresco

Right Brain Goes Shopping on Web 2.0

I recently wrote a post on how Web 2.0 is about the right-side of your brain. Yesterday Om Malik wrote about a new on-line shopping site, Browse Goods, that uses a much more visual shopping experience as differentiation in the very crowded shopping and directory business. The service is currently limited to shoes, watches, toys and a limited selection of sporting goods, but more is coming.

Browsing is more like going into a department store where all the goods are organized into separate, but very visible parts of the store. You walk in and see what catches your eye. You can imagine yourself stumbling across something that you weren't planning on buying, but impulse and the devil made you do it. I'll take one of those, these and that. This isn't about analysis and comparison, real left-brain activities, that are delivered by most on-line sites.

The left-brain in me sees a pattern in the presentation. This is very similar to drill-down heat maps that have been used in financial services for a while now. O'Reilly uses a heat map on his book sales to indicate what subjects are hot and what are not. At the moment there isn't enough data on the Browse Goods site to drive a heat map of sales, but you can see it coming.

I believe that the set of visualization mechanisms that are available for use in Web 2.0 presentation are limited, but can be stylized in infinite number of ways. We know that the left-brain likes linear and tree structures, where the right-brain like abstract concepts and spatial structures. So here is my first approximation of what those structures might be:

• Maps - this was the first wave and we saw this with Google maps being mashed up with things like real estate in sites like Zillow and similar concepts being applied to Flickr maps. This can be metaphorical or fictitious maps as well, such as those found in Second Life
• Heat Map - Already discussed, this is a more compelling mechanism of displaying the same information as analytic charts. Pie and Bar Charts provide good targets for drill-downs of popular or hot topics, but are not exactly right-brain friendly. Go to where the heat is.
• Color and Image Coded Lists - Top 10 lists will always be popular, but can use color and images to emphasize position, trends and rank like Newsweek's Conventional Wisdom chart
• Tag Clouds - Simple presentation of various levels of interest through typeface and font size
• Mind Maps - Related to tag clouds in presenting concepts, but more directed with explicit relationships. These may be too personal to use in mass market environments.
• Fish-eye and Hyperbolic browsers - This concept was originally developed at places like Xerox Parc and MIT Media Lab in the 80's and early 90's with few great applications, but could apply to things like tag clouds
• Time lines - What makes Technorati different? Time lines provide a tool to move in dimensions that the mind moves with scrolling walls and spheres (like Picasa). These can be stylized into lines or as calendars.
• Family Tree - A way of presenting the complex flow of relationships over time, especially for product lineage
  
• Radar - As a metaphor for what's over the horizon, radars present how far objects are from you and what direction they are coming - good for presenting future events, interest or threats
• Human Body - Presenting biological organisms is a powerful metaphor and none more powerful than the human body. You can use the human body as both a map and as part of parts explosion
• Rogues Gallery - There is a reason it is called Facebook. Your brain is extremely tuned to look at a large crowd of people and pick out a face.
• Parts Explosion - Taking a photograph of an ordinary physical object and blowing it apart with other photographs and animation as a way of drilling down a subject and providing a means of connection other objects (or things to buy - particularly in high-cost goods like autos)
• Analog Controls from the physical world. We already see touch-tone phone pads and dials and volume controls on many web sites. This was an equalizer control from Facebook to control output of stories based upon characteristics:
  

• Floating Detail - By floating over any of the above visualizations, Pop-up, hover dialog that provides a quick summary of information along with a representative thumbnail. This can take on the characteristic of a post-it or a magnifying lens

What these have in common with each other is modeling and presenting relationships between objects or describing a property or characteristic of an object. This is what the web has been doing with lists, tables, forms and charts since the beginning. More artistic types, such as those at Dotted Pair, are transforming the user experience by using the spatial and conceptual models of the right brain.

Have I missed anything?

Keeping up with Blogs -- Seeking Advice

The number of blogs that I am trying to keep up with has been growing considerably. Originally, I would just ping blogs occasionally. Mainly since the people I tracked were people who blogged occasionally just like me. However, I found that I could learn more from blogs than I could from news.

Then I started using RSS feeds through Yahoo or Google, but those became too cluttered. Then I tried using a combination of Technorati and Bloglines, which really required me to go check those web sites.

Lately, I have been using Feedreader. It seemed counter-intuitive to use thick, client-based software to track a very web-based phenomenon, but Feedreader has really provided me with what I was looking for to keep up with all the stuff going on in Web 2.0. I can use .opd files to synchronize all my feeds between Bloglines and Feedreader, so I still get web-based access. However, now I can be aware of new feeds when they happen. And it's free.

This has led me to add even more feeds and probably get more distracted by what's going on in the world. I can also see how the hell all these people were more informed than I was about all the new stuff going on.

What do you use to keep track of blogs? How do you avoid getting distracted by what's going on?

Is there a client-based blog creator that makes publishing to Typepad easier? That would certainly make it easier for me to keep up to date on my blog.