Nimble Code: Category Web Services

The Future of Feeds

2006-01-24T17:12:00-08:00

Earlier this year, I made a New Year’s Resolution to write more postings to the blog, but I have failed abjectly at that so far sadly. January is sadly a busy month for me, and I usually don’t feel like coming home after a long day to stare at a computer screen some more. But occaisionally inspiration strikes, and I feel like sharing some thoughts I have about the future of Syndication via RSS and Atom feeds in the next few years.

I have to agree with my company’s CEO Steve Goldstein when he says that increasing RSS/Atom adoption will be one of the big trends in the information industry in 2006. Speculating wildly, I think that 2006 will be The Year of RSS and Atom. And so, I’m going to make a few bold predictions of my own about the state of syndication and the software industry. To avoid spending even longer without posting to the blog, I have decided to not bother with research, so if something I have predicted has actually happened already, be sure to make a snarky comment about it to me below. Thanks. And now here are big things I see happening in the next year or so:

Mainstream Feed Adoption – Currently, most of the sites producing feeds are still blogs and news sites –- those for whom the article paradigm within feeds is a natural fit. However, non-newsy sites like del.icio.us and Flickr have earned a lot of buzz not just for their front-end interfaces but for their innovative and pervasive use of feeds everywhere and using syndication for non-article content. But this is still under the radar of the typical Internet user for now. I think this will change this year. I expect a major online site to follow suit this year and provide syndication of their data in a similar fashion. For instance, I would not be surprised if Amazon began to offer feeds for user wishlists or upcoming recommendations or music alerts. That would rock.

More People Will Embrace RSS, But Still Not Know It – A survey from last year by Yahoo! revealed that only 4% of Internet users knew what RSS was, but this will most certainly change. The name is obviously somewhat to blame (there’s a joke that if it were called SpeedFeed more people would use it), but I think it’s not the real problem. In truth, most people just don’t care. For your typical web user, the Web is a toy more than a place to siphon information from; they are therefore not exactly vexed enough by the slowness of visiting sites to want to setup an aggregator. Instead, I think some different killer app beyond mere aggregation will be the only thing to make most people want to use RSS. I have no idea what that app might be, but I’m sure one of you could think of something awesome.

RSS Is A Terrible Business Though – For companies that produce or sell content, RSS will be a good fit, providing additional ways for users to access the site and contributing a small boost in sales or page visits. But companies in the RSS software business (particularly online aggregators) will have a terrible time. There are just very few barriers to entry for competitors, and the whole we’ll make money through advertising model seems so very pre-boom to me. That said, there are still some promising markets for companies doing RSS software. Software that allows filters or other mechanisms for mitigating information overload from feeds will find a market. Bridges from the pull-based mechanism of RSS/Atom to push-based messaging services like SMS, Email, or IM will probably also do well. And of course, it would be smart for makers of web analytics software to also look at measuring conversions and clickthroughs from feed links.

Syndication Will Drive REST Web Services – One of the common complaints leveled against REST-based Web Services is that the simple model of REST does not specify the XML representation for data (unlike the case of XML-RPC and SOAP). This will never officially change, but I think it will become increasingly more common to see REST Web Services that return RSS-like data for searches. Indeed, throw in Amazon’s OpenSearch extensions for RSS and you have support for presenting pagination within RSS documents. The practical upshot of this is that programmers can use the exact same code for RSS, Search Engine Plugins (if any other engines besides A9.com support OpenSearch_), and REST-based Web Services. More significantly, this makes REST more compelling to these programmers, since supporting SOAP’s data encoding would require supporting an additional data format for searches and more code to implement. Similarly, the arrival of the Atom Publishing Protocol will also bring awareness of REST Web Services to a wider market of people who will be using it without even knowing it.

Feed Extensions Will Get Wider Support – As RSS/Atom grows in its usage, it should be possible to find more aggregators and process that also support the more popular extensions to feeds (these are normally specified within the document in a separate namespace). I think a few extensions will become essential in the years ahead (and I don’t even know if they exist currently!): support for threading of RSS items (for feeds based on email lists); partial encryption of RSS documents to be decrypted in the reader (would allow banks to offer feeds for instance); geographic tagging for marking stories (see the breaking news/photos on a map). In any event, I think good aggregators will show in some fashion all tags embedded in the feed so that users could search or filter based upon them.

Syndication in the Enterprise – Finally, it is obvious that RSS/Atom will become more common within corporate intranets as well. And the credit for this belongs to Microsoft of all companies. Mainly, by placing RSS within all aspects of Windows Vista, Microsoft will drag all sorts of big IT Departments into accepting RSS as a solution for messaging and event notification. This will in turn make them more likely to also accept other solutions based on syndication. In fact, I’m optimistic enough to think that B2B Syndication-based products will do better than B2C (remember that vexing 4% recognition rate from above; CTOs can mandate use in their companies). Smaller companies will be quick to embrace the immediacy of RSS and larger companies will also enjoy it for Windows Vista integration.

Atom vs. RSS2.0 Holy Wars Won’t Matter I know it vexes some readers of this posting that I’m using RSS sometimes as a shorthand phrase for RSS2.0 or Atom-based syndication. And I agree with them that Atom is technically superior while RSS2.0 has better market share. But my ultimate feeling is that the differences between the two for the consumer will be largely academic (like caring about whether your audio player is playing MP3 or AAC), since any good aggregator or processor should just be able to handle both formats effortlessly.

I will probably follow up on some of these points in more depth anon, but feel free to comment and quibble in the comments section below. I am no industry expert, but I do so like to make guesses about the business. And I’m curious what you think as well. Check in next year and we’ll how well they turned out.

Amazon Shakes Things Up

2005-12-14T17:42:00-08:00

I really must hand it to Amazon/A9/Alexa. They have consistently delivered more innovation to the developer than those two combined. Amazon started to impress me when they allowed any application to search their extensive database of product information. Directly they have earned nothing from this, but it more than makes up for itself in good will and inspired purchases. Amazon understands Web Services.

Amazon also has a search engine. But here they are not the market leader like they are for books. Poor A9, always ignored while Google and Yahoo and MSN get copious amounts of press and praise. But this has been good to spur on Amazon and Alexa (their search company) into creative new territory for their search engine. A9 has already received some attention for their YellowPages data that includes photos of storefronts; that’s certainly cool, but it’s just content. On the other hand, OpenSearch is a cool change to the spider-driven notion of search content. Google may have specified their sitemap standard for giving hints to spiders. But A9 did that one better with their OpenSearch RSS extensions, which allowed any site to package itself as a channel for A9.com. This allows sites to handle search requests dynamically at the time of request, rather than creating documents to then be spidered and cached.

To illustrate why this is a cool idea, suppose I had a site that did hotel bookings. I could create an A9 channel that could accomodate search requests for things like “nyc hotels” with actual hotels and rates from that moment. If people had date information, I could be more clever, but the key of this is that Amazon is handing over control of their search results to the end sites. As long as a user is willing to trust a channel enough to add it, A9 is happy to get out of the way.

Of course, there is nothing to make syndicated OpenSearch sites obey search rules in the same way A9 does, but who cares? As Google’s success with a single bar shows, people care less about all the little details of searching (boolean vs. web syntax, stemming, relevance scoring) than just being able to find stuff. Amazon gets this, but Google seems to have forgotten it.

But Amazon/A9/Alexa’s latest move has really blown me away. In recent years, a lot of search providers have found additional revenue in peddling search appliances to companies. Essentially, you sign a contract with a company Google and buy a local web spider that crawls content you specify. You then pay Google a price based on support, how many documents you plan to index, and how many users will search your index. There are limitations to keep you from turning Google against themselves and all pricing is set up front rather than growing with use, and prices can approach up to $2 million up front. More importantly, you are not actually using Google’s index, just recreating part of it independently. And while Google has much vaunted data centers across the globe, enterprise customers get only a single box or two to place in their own data centers if they have them.

Amazon has been interested in entering the enterprise market, but they did not want to be in the business of creating hardware servers for enterprise clients (Google will always be better at that). And that’s when someone at Alexa had the brilliant idea of opening up their entire search index data to the outside world. And what makes it additionally brilliant is that you install nothing locally (everything is done via Web Services) and pricing is set by consumption and not by contract, meaning even the most modest users can do something with the backend data. Users can write programs to process data sets, implement their own additional indices on top of A9’s basic web search, or even just analyze a massive collection of web pages for statistical goals. In some sense, Amazone allows customers to not just outsource spidering but also maintenance of indices they’ve built.

It’s a brilliant move, and I must applaud Amazon for the creativity of their vision. Forget Google and Yahoo, Amazon is doing more to turn Search into a utility than the any other company around.

The Silliness of SOA

2005-11-10T09:06:00-08:00

There seems to be a lot of buzz around the concept of Service-Oriented Architecture (SOA), as long as you don’t ask any of its proponents what it actually means. For an example of the ways in which English can be mangled by technical jargon beyond recognition, check out some of the definitions from A Defining Moment for SOA and Revisiting the Definitive SOA Definition to see what I mean. My personal favorite from the bunch is

SOA is a form of technology architecture that adheres to the principles of service orientation. When realized through the Web services technology platform, SOA establishes the potential to support and promote these principles throughout the business process and automation domains of an enterprise.

which really tells me nothing at all about why this is cool and where you would start if your CEO suddenly decided you needed it. Cutting through the bombast and hype, it seems like SOA is basically

Wrapping a backend database or similar resource in a small component for business logic you can talk to with XML.

This is definitely cool for many reasons (loose coupling, security, building big things from small parts), but I don’t understand why there’s so much hype about it now. Not to boast, but we’ve been doing this at Alacra for 8 years now, and I’m really gobsmacked that this sort of architecture would be a revelation to anyone, especially since it builds on design patterns good sense and the older Unix and later Internet philosophy that Small Pieces Loosely Joined can make great things. And SOA is pretty much what the earliest XML-driven web services were.

I can only assume that the latest hype about SOA is less about what the technology is, but what it is not. Hint: add a P and you can see what businesses find so refreshing about SOA. SOAP is a heavyweight protocol that can be good for some external APIs, but is a lot more work than a simple quick and dirty internal application often needs. Imagine being able to drop the requirements of WSDL bindings, XML typing, even limited data structures to get the most lightweight, low-ceremony solutions you can feel comfortable with using. And since it’s internal, you don’t have to worry about the “shame” of not being fully SOAP+WS-whatever compliant for your API. Looking at it this way, no wonder they’re excited. I’d be excited too if I were able to stop documenting and start hacking for change.

Great Picture, But One Quibble

2005-09-30T00:09:00-07:00

Courtesy of Tim O’Reilly’s Foo Camp (which I am definitely not cool enough to be invited to), there is now a picture to match the exciting flow of ideas and themes coalescing into Web 2.0. I think this assemblage of bubbles and trends is a great thing to see, especially since it serves as a better executive summary of high-level ideas than gleaning bits and pieces of the big picture from blogs and demo sites across the web.

That said, I think one thing is missing from the picture they provide. Maybe I am a bit preoccupied with the subject, but I think RSS (or Atom here, I’m just using RSS as shorthand for syndication) is really one of the biggest things driving Web2.0 services and adoption these days, but it hasn’t even gotten a mention in the top as an influencing technology (unlike blogs or Gmail). I think blogs were great at establishing RSS as a way of keeping track of changes, but the really influential aspect of Del.icio.us and Flickr is not just tagging, but establishing RSS as a mechanism for tracking any possible view of the system you might want in as light-weight and user-friendly mechanism as possible (as opposed to the awkwardness of SOAP or even REST to the end user).

I think the source of my unease here is that I’m mostly a backend guy. A lot of my work at Alacra has been making sure that all sorts of information flows agilely between processes and servers. Backend stuff. It makes it happen, but if it’s working, you never notice how critical it is to success. Similarly, AJAX and other front-end browser mechanisms are very nice in my mind. But the biggest joys and successes of Web2.0 are all driven by the fluidity and ease of RSS and REST. “Hackability”, “Data as Intel Inside”, “Right to Remix” ... RSS made this a possibility and these are what drives me to take Web2.0 seriously and not just as another wave of web hype. All I’m asking for is a little recognition. Thanks.

Update: The good news is that it seems like I’m not alone in this view. The bad news is my company is Dave Winer.

RSS Comes to the Enterprise

2005-09-23T03:03:00-07:00

As you may have guessed from this site, I am hardly a person you would expect to praise Microsoft for their technical insight and forward thinking. Like the story about generals, they always seem to be fighting the last war, with their late arrival to the Internet and Web being a current and recurring example (now, rather than Netscape, it’s Google that is seen as their greatest threat). However, occasionally they get a thing right along the way, and I feel honor-bound to note it in this case.

Over at The Register, there is an article RSS Goes To Work In Windows detailing the ways in which Microsoft has jumped onto the RSS bandwagon with Windows Vista (shipping next year, maybe?). For those of you following along at home, Microsoft is planning to ship a built in RSS-store capability in Windows which will allow any application compiled against the appropriate API to be an RSS aggregator and manipulate feeds programmatically. This is a nice touch, but hardly a stretch yet.

What is interesting to me is that Microsoft discusses their plans to have the next release of their Dynamics CRM software to create RSS feeds that can be syndicated to. Other server applications with RSS extensions in the works are Sharepoint Portal, Exchange, and possible Office. As the article quotes it

“CRM is one of the first examples of how we see RSS unlocking data in the back end data systems,” Amar Gandhi, Microsoft Internet Explorer group program manager, told The Register during a recent interview. Microsoft revealed plans to RSS-enable its CRM last week at the Professional Developers’ Conference (PDC)

Chris Caposella, vice president for Microsoft’s information worker product management group, told software developers attending PDC Microsoft believes RSS would be transformed into a platform that embraces business applications.

That last paragraph was something I’d never see. Because honestly, I never thought people would “accept” RSS in the Enterprise. While we can joke about how meaningless the term Enterprise Software is, RSS is definitely not it by any usual sense of the term for several reasons:

RSS in general has no typing or generalized schema. It’s really a schema for news stories that can be generalized for events, but it’s not for serialization of complex typed objects like SOAP.
The RSS model is extremely simple. Pull requests over HTTP. No pushing, no notification mechanism for new content.
In itself, RSS has no security for content (I know, Atom has support for encrypted content), and access security is generally only done through obscurity (giving an individual user a feed with a long unique token in it, maybe unguessable, but not unsniffable)
Finally, RSS is not really controlled. There are no authorization lists, no ACLs or DRM or other such mechanisms to control who can subscribe and where they can do it.

That said, I think all of these aspects are why RSS has succeeded so widely in the world of the Web2.0, while SOAP has largely failed to gain traction. In fact, considering that one of SOAP’s biggest backers is Microsoft, this RSS announcement reveals what a disappointment SOAP has been as a web service platform. So, why is a company like Microsoft so willing to reject one of its software initiatives and embrace RSS so, in effect giving RSS an mantle of legitimacy in the Enterprise it might’ve taken years to achieve?

I think that if we want to consider the biggest Web2.0 trend for the coming year, it won’t be the proliferation of AJAX or increasing maturity of Web Frameworks. It’ll be the proliferation of RSS into every aspect of server-side content. One of the greatest things about Flickr and Del.icio.us is that you can subscribe to an RSS feed for every possible page. And since pages are dynamic content (combinations of photos, groups, tags, users), what you are really subscribing to is not a list of documents, but the latest matches for a search (ie, “find me all photos tagged Italy by this user”). And it’s easy to do, regardless of the site.

One of the cool features of Mac OSX Tiger’s Spotlight (and probably Windows Vista) is the ability to save searches as virtual folders which can be opened and browsed like regular folders in the Finder. This is a really cool way to manage ever-growing content. Now, imagine being able to do similar functionality with server-side content, and suddenly the appeal of RSS makes a lot of sense (admittedly since RSS usually shows only the last 20 items or so, there is a you snooze, you lose problem, the general metaphor holds). Most of the Web2.0 is about taking the applications of the desktop into the browser, this brings the web back into the machine and applications.

No wonder Microsoft suddenly has the RSS Religion. Done well, this might be their best chance to keep people hooked into Windows and all their software products for business. Of course, they are trying to do the old embrace, extend, extinguish trick, but this time it may not fly because Microsoft’s not in the driver’s seat for the technology. That said, I must admit that Microsoft has made a smart and proactive move for the future here, and I think it’ll benefit them well. But more importantly, it lends a large amount of authority to RSS in the Enterprise, which makes RSS and the Open Source Standards-Based community the biggest winner of all. At Alacra, we’ve felt that RSS will be much bigger in the Enterprise for a while. It’s neat to see we’re not alone.

RSS as an Error Reporting Mechanism

2005-07-17T18:37:00-07:00

An interesting idea occurred to me the other day...

First, a disclaimer. This is not a specification. Specifications are fixed and obsessively spelled out, and I would rather explore the idea fully first. Besides, specifications are notoriously boring and opaque, and I think that's part of the problem I want to address. So, let's get started.

The Wonders of RSS

RSS is a pretty neat thing when you think about it. In the past 3 years we have seen a veritable explosion of sites that create RSS feeds and interesting readers that consume them. It's led to some pretty sophisticated ways of reading the web, and your standard aggregator allows grouping, regular checking, unicode support and searching. In addition, newer apps even allow users to do things like create "virtual feeds" by specifying search queries that can be run across multiple feeds, while services like Bloglines allow you to take your subscriptions anywhere around the globe and will probably soon over ways of transposing RSS items into emails or SMS content or such. I imagine the next big thing will be helping us to better cope with information overload from RSS feeds. Which is fascinating stuff. Especially fascinating is that this is in spite of (or rather because of) RSS is painfully stupid. It's not the first or the "best" mechanism for distributing content across the Internet (I think many would argue that NNTP is far more elegant and scalable), but its basic simplicity and ease of manipulation has made it a winner (although IMHO it took the further simplification of RSS2.0 and Atom vs. RSS1.0 to ensure this.)

So, what was my point in a nutshell? There are several awesome things about RSS that I think should be noted in a bullet list:

Very simple. The RSS2.0 spec is so basic that deserves the name Really Simple Syndication
Doesn't care about transport. If HTTP is good enough for everybody else, it's good enough for us. Of course, you could put RSS on top of other things, but the important thing is that we don't need to worry about implementing the transport layer here.
Extensible. One good thing about RSS is that you can embed additional content in the basic RSS feed. As long as you use a namespace for your elements, any good RSS reader should ignore them, but savvy ones can use them.
Regular checks. One of the common complaints about RSS is that it's a dumb pull-based mechanism and thus can cause many redundant server accesses from every RSS client checking a feed once an hour. This however is a virtue in my mind. It keeps things simple (no subscription tracking).
Time management. With RSS you're not so much browsing the web; instead you are monitoring it.

Read that last point again. Because this is where I went from Modern Web 101 to a Reese's peanut butter cup moment (I'll explain the product plug in a bit). When you get down to it, there is nothing in RSS2.0 (or RSS1.0 or Atom; I don't care which RSS spec you prefer) that says the items in an RSS channel have to be documents somewhere (ie, new stories, new photos, etc.). And in fact, I would argue that the meaning of an RSS item is really as an event which just happens to be tied to a particular story in many cases. But it doesn't have to be.

RSS and Error Reporting

Okay, the title here gives away where I'm going with this, but I still want to provide a little more background on things. In my office, I'm a backend guy. This means I do a lot of work tying together various systems and making sure data flows around smoothly and efficiently. It works really well actually and I'm quite proud of our simple system (more on that later). But errors are inevitable, and they occur in many different areas. Imagine you have a web server farm with applications. You can think of where errors occur in terms of domains (listed below with some standard error reporting mechanisms)

Hardware - if the underlying machine has problems, the OS will hopefully catch it. If you're using Unix-based systems, you can configure syslog*. Otherwise, on a Windows-machine it might get logged into the System Events and perhaps picked up by a third-party tool like Patrol or such.
Network - Routers, firewalls, etc. can go down. SNMP seems to be the standard mechanism for reporting problems in the network.
Availabilty - A machine may be up but unreachable due to high load or other things. NAGIOS is a way of catching this.
Web Server - the web server may have problems with some requests. This are generally logged to a local HTTP log. Logs from multiple HTTP servers can be sent to another host via syslog, but this is rarely done.
Web Application errors - in the best cases these might be logged to a local file on the server, but usually they are just displayed on the screen to the user.

There are a few interesting things to note here:

Some of these error reporting solutions also include a networking protocol in addition to an error format, meaning you could assemble a centralized console for error reporting; no such luck in the web server.
For applications, it's pretty much left to each developer. As a result, errors if they are logged are often local, since not many people think to use syslog for error reporting.
I think the ideal thing for site maintenance is to have some sort of centralized view of errors in the system as they occur.

Anyhow, to cut an already long story somewhat short, I was working on an application that would automatically generate and process RSS feeds that I wanted to be robust and I wanted to know whenever errors occurred without having to login to the machine (ala syslog). So, I looked into some ways of reporting/logging errors when it struck me: why not use RSS as an error reporting mechanism?

You Got Your RSS in My Syslog, You Got Your Syslog in My RSS!

And so we get to my Reese's Peanut Butter Cup reference. Think about it, RSS has some real advantages to error reporting:

Very simple. RSS has a lot of support across many platforms and with a wide range of viewers/aggregators. Does syslog or other networked error reporting mechanism. And it has support in programming language libraries too.
Doesn't care about transport. HTTP is established and allowed through most corporate firewalls. No dumb network debugging or NAT fiddling and such.
Extensible. One size doesn't fit all. I might want to record basic messages in some cases (ie, "printer on fire") or capture detailed stack traces in another (Rails logging). XML + Namespaces is a great way of layering stuff in there (although you need an RSS reader that will allow you to look at the underlying XML to get the extra data)
Regular Checks. I like the pull communication of RSS because it acts like a heartbeat to your error reporting services; when something is wrong with one of my sources, I know it because my RSS reader tells me so. (to make an analogy, suppose you had a phone to be called for emergencies; does not getting any calls mean that there are no emergencies or might the line be dead?). Granted, I might not get an error immediately, but to be honest I don't think I've encountered many errors that a 5 minute delay would've made impossible to handle; don't use it your nuclear reactor though I guess.
Time Management. So, if I wanted to I could have errors, warning, statistics and other things coming into my central console. I don't have an insight into the best way to manage all this information effectively, but I have a pretty good idea that problems of information overload will be tackled much sooner in the RSS world than they have been in the Syslog user base.

So, here's the idea. Define a basic set of RSS error fields in a particular XML namespace (like http://www.nimblecode.com/rsserror). This will have a set of optional tags that could be added to items. If users want to define their own extra tags they could just by declaring a new NS like (like http://www.nimblecode.com/rsserror/syslog) and add them. Really, you can add anything to XML, but the NS tags help consumers of the information to know what to expect (so, if I want to look for syslog-type errors). An example of this might be produced soon.

The Minimum Error Fields

So, if we were to make a minimal RSS error specification, what optional and required tags would it include. I have an idea:

<error:msg> The text of an error message. Note that this could also be put in the title or description of the regular RSS item, but I think it's good to define an explicit place in a namespace it will always be.

And that's it. Obviously, there are other things we might want to record. For instance in my case, I think of optional tags like timestamp, extended, code, hostIp, application, etc. Not to mention some of the error information available with syslog traces, but I want to weigh what tags should go in a given namespace before I start adding options willy-nilly (indeed, there's no reason why you couldn't define your own extensions.)

So for instance, if we were defining an addition syslog RSS error namespace, we could also add the following syslog fields as tags:

timestamp - a timestamp for the error
priority - the priority of the message (EMERG, ERR, WARNING, INFO, etc.)
facility - the facility option for syslog (AUTH, MAIL, USER, etc.)

On the other hand, if I were working on extensions for Ruby on Rail's logging format, I could include the following tags:

timestamp - the time the error occurred.
severity - the severity of the error
sql - the SQL executed against the backend server
backtrace - the call stack where the error occurred

Both of these would still use error:msg for the main message with the error. It looks like I should also make timestamp part of the standard set.

So there you have it. I will probably be working on this a bit more, but it's an interesting idea and I'm curious what other people might think.

The New Web Math, Or How SOAP Adds Up To Nothing

2005-04-25T00:06:00-07:00

Last week saw the intriguing web mashups of combining two popular web services in one place. So, first there was combining Craigslist Real Estate listings with Google Maps, which although slow is really quite amazing to behold. Then someone else hacked Gmail + Del.icio.us, using Google's mail service to capture contents on bookmarked pages at the bookmarking time. Not to mention Adrian's trailblazing mix of Google Maps and Chicago transit info.

All of these are amusing and technically well-executed hacks, but what is most amazing to me is not the front-end combinations, but the ease in which it is possible to mix web tools in the back end to create new powerful services. And I have been especially struck by how SOAP has been utterly unnecessary for this to happen. In fact, I would argue that the strict COM-style model of SOAP is exactly the problem here, in that SOAP requires such effort and planning, it resists such freeform discovery of capabilities beyond what the creators of products imagine. REST seems to have a natural advantage here in that it builds on the inherent procedural model of HTTP and thus can evolve as naturally as websites do. Combine that with dynamic HTML tricks and we can see how nimble code can outmaneuver the heavyweight mechanics of SOAP again and again.

For a more lengthy and detailed critique of SOAP, see my earlier post, Radical Simplification

Radical Simplification

2005-04-16T10:43:00-07:00

David Heinemeier Hansson has posted an interested find over at his blog Loud Thinking about IBM’s stated new goal to pursue Radical Simplification in their enterprise work. Essentially, Big Blue is starting to acknowledge that most enterprise web development is just too cumbersome and daunting for agile and powerful web development. Things like SOAP and WSDL are examples of this. What should be a simple task like retrieve a weather forecast from a remote site becomes a complicated mess of debugging SOAP calls, tweaking WSDL specifications and using wizards on the server to generate code that is impossible for the end user to debug, let alone understand.

Or as Sam Ruby puts in his excellent presentation to IBM on the topic Hello From The Open Source World:

For normal people, the perceived usefulness of a computer language is inversely proportional to the amount of theory the language forces you to learn.

Sam illustrates this in his slides by showing the classic “Hello World” example in a variety of programming languages on different slides. But when he reaches WSDL to specify it, it’s a blank. Because, quite frankly, WSDL is so unwieldy to use, it’s hard to just build it on the fly, forcing many people to plunk down bucks to Microsoft or IBM to get their compilers to build it for them. It’s good business for those companies, but bad news for the Internet as a whole I think.

Ruby says that essentially for a framework to succeed on the web, it has to enable a situation he terms Zero Training, a state where it is easy to get going in the language and to adapt examples to your own needs. Good programming languages have it, some web technologies have it, SOAP and WSDL don’t. Indeed, I feel like the growth of SOAP/WSDL in the enterprise has been in spite of the difficulties of developing for it (mainly because of Microsoft and IBM pushing it). Because, quite frankly, it’s a beast for anything but the most basic RPC-style calls. The reasons:

SOAP allows you to abstract away the underlying transport protocol. But for 99% of SOAP communication, this protocol is HTTP, and the abstraction limits the control you have over HTTP.
SOAP needs to validate the entire message before applications can operate on it. This usually means it has to load it up into a DOM as well, so calls that return lots of data or might take a while (the fun ones) are frowned upon
SOAP also assumes the COM/CORBA marshalling mechanism, so no streaming data or parsing data until the entire document has been received, parsed into an XML DOM tree, and then mapped into an in-memory object tree. For large amounts of data, forget it.
WSDL has a lot of syntax to map procedure names/arguments onto HTTP. Contrast this to REST, which embraces the inherent naming conventions of HTTP and applies them to a simple procedural call model
The assumption of these tools is that you will use their SOAP wizards instead of rolling your own code. The wizard is meant to be the only way, not an additional way.
In fact, the idea that SOAP = Simple Object Access Protocol has become a profound irony.

In his presentation, Mr. Ruby points to PHP as an alternate model. PHP is not really a pretty programming model and it tends to shy away from enforcing higher-level abstractions in favor of lower-level models. But, it is precisely for this reason that PHP has succeeded where many other dynamic web application programming models have failed. Indeed, as it says in the page Do You PHP?

Database abstraction is mostly a myth. There is nothing wrong with direct database calls’ making use of all the tricks and cheats your chosen database has to offer, to tweak as much performance as possible out of it.

And this is the kicker here. Complicated database models tend to abstract away the lower level details of individual databases, even when those details can be the difference between fast and sluggish performance. Remote invocation models like SOAP abstract away the details of the underlying calling mechanism, even when tweaking HTTP requests can be the difference between a fast web app and a slow one. The lower-level mechanism might require more work for the developer, it might be ugly as sin, and it might be wrong and inefficient on some levels, but it works, and more importantly, it works quickly. Because ultimately, the customer doesn’t care what technology is used on the backend, they just want their data fast.

Bringing it back to my own experience, I once had to work on an application where we searched a remote document repository via SOAP and got metadata back on matching documents in XML. The service would send back all matching results for a query and in some cases, this would result in returning 150000 documents over 5 minutes of streaming XML to us. Knowing that the user probably would want to see data as quickly as possible, I decided it would be good to render whatever data I got after I received 10000 documents and tell the user to perhaps narrow their search down. And I tried to do this in SOAP in Visual Studio, and I failed. I could error out completely, but the mechanism had abstracted away the underlying XML-over-HTTP communication that I had no hooks to get lower level.

And so, I chucked it out and went back to basics. The upcoming call to the remote service was faked by filling in an XML string I grabbed from the original SOAP call via a packet sniffer. On the return, my program screams through the data with stream-based XML parsing. Once I hit 10000 records, then I can stop reading the input stream, display what I have, and tell the user that they need to narrow their search criteria. All within 30 seconds.

Those people who use SOAP are probably horrified by this and would rightly point out that I had to do a lot of lower-level work to get some of the same functionality promised by the one-click higher-level SOAP abstraction. But I think that’s precisely the problem. I needed something to work, and I didn’t need as much for it to be pretty. The SOAP framework locked me into a particular model, and once I needed more than that, it failed to deliver. Abstraction is nice, but abstraction always reduces speed, abstraction always reduces flexbility. In most cases, it is possible to strike a balance where the abstraction helps more than it hurts, but until SOAP is able to handle the heavy lifting needed for serious web work, it isn’t worth my time.

Google Maps and the User Experience

2005-04-07T10:05:24-07:00

Jason Kottke nails one of the reasons why Google Maps has generated such excitement in the last few days with the addition of satellite data. Certainly it's not that the technology is especially new (Microsoft and other companies have had satellite data on the web, and some localities like Washington, DC have provided maps that mix GIS information with aerial views). But the thing about Google Maps that inspires people and creates such a possibility for serious play is the user interface. In his post on Google's user experience, Kottke quotes an earlier post with a statement I like so much, I'm going to quote it here:

Advances on the internet and the web are typically heralded as technology-driven. Robert Morris from IBM argued last year at Etech 2002 that -- and I'm paraphrasing from memory here -- most significant advances in software are actually advances in user experience, not in technology. Mosaic was not an advancement in technology over TBL's original browser. Blogger is a highly-specialized FTP client. IM is IRC++ (or IRC for Dummies, depending on your POV). The advantages that these applications offered people were user experience-oriented, not technology-oriented.

Packaging and interface do matter. Indeed, I think the most successful online web applications ultimately will be those with the best interfaces, those that allow people to easily grasp what they can be used for and to dream up new ways to extend their functionality and uses in ways the designers haven't conceived of.

I Can See My House From Here. And Yours Too.

2005-04-07T07:00:00-07:00

Second day blogging and I am already late to the party in talking about the changes Google has made to the Maps program by incorporating the satellite imagery from Keyhole. The result is very cool, not just because it's neat to see things from space, but because the map functionality is overlaid right onto the satellite data as well (as John Park shows with his bike route). What I like about this is what I also like about people using the maps with Flickr's annotation capabilities to present memory maps or people doing "virtual tourism" of sorts to get a feel of geography from afar, in that there's a real sense of play and fun and hacking going on with Google Maps that I haven't seen with a web service for a while. I will talk more about this later, since I think it's the difference between a successful web service and a dud.