Rails and the Distribution Question

Posted by Jacob Harris Tue, 23 Aug 2005 08:43:00 GMT

I like Ruby on Rails a lot. It’s a pretty awesome web development frameworrk and it’s generated a serious amount of buzz for its young age (just over a year). One major milestone that’s ahead for the language will be a 1.0 release that among other things promises to deliver more API stability than there has been with more recent releases (as well as freeing developers and writers from creating code that’s partially outmoded on arrival). Will that mean though that the big development work for Rails will be done? Hardly. If Rails is really to succeed as a development platform, I believe it can only happen if the distribution problems are solved. And Rails 1.0 seems to be a necessary precursor to that.

I like to think with web application frameworks there are 3 Ds that are important for the platform to succeed:

  • Development – how easy is it to develop?
  • Deployment – how easy is it to place on your own servers and systems?
  • Distribution – how easy is it for third-party users to deploy on their servers?

I think Rails has done very well in the first case. In the second case, I think tools like Switch Tower will help to make the second problem as nice as the first. But I think the third problem is the most problematic in Rails right now. Essentially, there are no nice and standard ways for developers to package up their Rails-developed programs for other users to easily install (without Rails experience).

Let me give a concrete example here. As you may know, I recently switched this blog from Wordpress (a PHP-based service) to Typo (a Rails-based service). I currently am hosting on Textdrive, which is rare among hosting providers in that it supports the latest Rails versions, so at least I did not have to fight with my ISP to get Rails as an option. But even then, I must admit that Wordpress’ setup was a dream to use, and not knowing a lick of PHP, I was able to get a blog up, installed, and running in no time. In contrast, I do have some knowledge of Rails, but Typo was definitely harder. For starters, I had to manually create tables in a MySQL database by running a MySQL file to create the tables (good thing I didn’t use one of the more obscure adapters like MS SQL that didn’t have a manually written schema file for them). I had to go an edit the database.yml config file manually in a shell (not confugured via an install script) Importing the data from Wordpress was a bit difficult too, and I had to fix the import script. Finally, I discovered that I have actually been dispatching Rails methods with the FastCGI dispatcher, even though I’m not using it currently. So, my blog went down over the weekend and I didn’t know. Not good.

This is not to trash Typo, which is a truly excellent product. Nor is it to condemn Rails, which is still too young to even explore a thorough solution to the distribution problem yet. But I think if we start thinking about the future of Rails, the distribution problem is important, because without easy distribution, Rails will be doomed by the network effect. What do I mean by that?

There are a lot of PHP developers out there these days, far more than develop for Rails, Django, Seaside or some of the other frameworks out there. Why? Is PHP a better framework? Arguably no. But is PHP in a whole lot of publicly deployed web products, like Wordpress? Or more wikis than you can shake a stick at? Yes. This in turn leads to more people thinking of developing in that language, books being written, etc. which accelerates adoption of PHP even further. The network effect.

I think 37 Signals, Robot Coop and such have done a lot to advance Rails over the last few years. But ultimately, I think it’ll do more to sell their particular style of visual design than Rails as a development platform. Sure, there are people making custom solutions from scratch that might be wowed by the backend magic that makes it all happen. But I think there’s a bigger pool of programmers who might find themselves having to setup a quick Wiki for their intranet or such? Given the choice, how many of them will write a system from scratch in Rails? And how many of them will just grab an existing PHP-based solution? And once they’ve done the latter, how likely is it they’ll look at Rails ever again? The best way for Rails to grow is for there to be many Rails apps to choose from. And to get there, Rails needs to make distribution as easy as development.

(I hope to follow up on this in a few days with some further observations and ideas.)

Tags , ,  | no comments

Emerging from the Deep Web

Posted by Jacob Harris Tue, 16 Aug 2005 09:55:00 GMT

After a long period of working in relative obscurity to the web at large, my employer Alacra has now launched a store to sell some of the premium content to the public. Gentlemen, I present to you (in Beta form), Alacra Store

It’s a very interesting time for us here at Alacra. Although we are a small company compared to the media behemoths that surround us (ie, Thomson, Lexis, etc.), we have always managed to thrive by delivering agile and powerful solutions to our customers and making their happiness our #1 priority. And although I can’t really talk about it, there are great things afoot. For more analysis, see the always excellent Fred Wilson. I particularly like it when he says

I have been involved with a lot of companies over the years, but very few, if any, are as loved by their customers and are as unknown to the rest of the world as Alacra.
But that may change shortly. Hopefully not the “loved by their customers� part. I am talking about the “unknown to the rest of the world� part.

That’s us. And if there is one thing I love even more than the new technologies we will be using with the store, it’s the fact that I can finally show people the stuff I (and my able colleagues) have been working on.

Posted in  | Tags  | no comments

Moved to Typo

Posted by Jacob Harris Thu, 11 Aug 2005 09:50:00 GMT

Just wanted to note that I have moved this blog to Typo, an excellent blogging system written in Ruby on Rails . If you’re not reading this in an RSS reader, the change should be pretty apparent, although I must apologize for not having rethemed yet. Otherwise, I look forward to hopefully spending more time blogging and less time doing maintenance in the future.

Posted in  | Tags , , ,  | no comments

O'Reilly Emerging Technology Conference (March 6-9, 2006)

Posted by harrisj Wed, 03 Aug 2005 23:47:00 GMT

Check out the Call for Papers of the O’Reilly Emerging Technology Conference This looks like a pretty amazing subject and I am curious what the presentations selected will be. It looks like an amazing forum for really thinking about how the make the new web.

Posted in  | Tags , ,  | no comments

RSS as an Error Reporting Mechanism

Posted by harrisj Mon, 18 Jul 2005 01:37:00 GMT

An interesting idea occurred to me the other day...

First, a disclaimer. This is not a specification. Specifications are fixed and obsessively spelled out, and I would rather explore the idea fully first. Besides, specifications are notoriously boring and opaque, and I think that's part of the problem I want to address. So, let's get started.

The Wonders of RSS

RSS is a pretty neat thing when you think about it. In the past 3 years we have seen a veritable explosion of sites that create RSS feeds and interesting readers that consume them. It's led to some pretty sophisticated ways of reading the web, and your standard aggregator allows grouping, regular checking, unicode support and searching. In addition, newer apps even allow users to do things like create "virtual feeds" by specifying search queries that can be run across multiple feeds, while services like Bloglines allow you to take your subscriptions anywhere around the globe and will probably soon over ways of transposing RSS items into emails or SMS content or such. I imagine the next big thing will be helping us to better cope with information overload from RSS feeds. Which is fascinating stuff. Especially fascinating is that this is in spite of (or rather because of) RSS is painfully stupid. It's not the first or the "best" mechanism for distributing content across the Internet (I think many would argue that NNTP is far more elegant and scalable), but its basic simplicity and ease of manipulation has made it a winner (although IMHO it took the further simplification of RSS2.0 and Atom vs. RSS1.0 to ensure this.)

So, what was my point in a nutshell? There are several awesome things about RSS that I think should be noted in a bullet list:

  1. Very simple. The RSS2.0 spec is so basic that deserves the name Really Simple Syndication
  2. Doesn't care about transport. If HTTP is good enough for everybody else, it's good enough for us. Of course, you could put RSS on top of other things, but the important thing is that we don't need to worry about implementing the transport layer here.
  3. Extensible. One good thing about RSS is that you can embed additional content in the basic RSS feed. As long as you use a namespace for your elements, any good RSS reader should ignore them, but savvy ones can use them.
  4. Regular checks. One of the common complaints about RSS is that it's a dumb pull-based mechanism and thus can cause many redundant server accesses from every RSS client checking a feed once an hour. This however is a virtue in my mind. It keeps things simple (no subscription tracking).
  5. Time management. With RSS you're not so much browsing the web; instead you are monitoring it.

Read that last point again. Because this is where I went from Modern Web 101 to a Reese's peanut butter cup moment (I'll explain the product plug in a bit). When you get down to it, there is nothing in RSS2.0 (or RSS1.0 or Atom; I don't care which RSS spec you prefer) that says the items in an RSS channel have to be documents somewhere (ie, new stories, new photos, etc.). And in fact, I would argue that the meaning of an RSS item is really as an event which just happens to be tied to a particular story in many cases. But it doesn't have to be.

RSS and Error Reporting

Okay, the title here gives away where I'm going with this, but I still want to provide a little more background on things. In my office, I'm a backend guy. This means I do a lot of work tying together various systems and making sure data flows around smoothly and efficiently. It works really well actually and I'm quite proud of our simple system (more on that later). But errors are inevitable, and they occur in many different areas. Imagine you have a web server farm with applications. You can think of where errors occur in terms of domains (listed below with some standard error reporting mechanisms)

  1. Hardware - if the underlying machine has problems, the OS will hopefully catch it. If you're using Unix-based systems, you can configure syslog*. Otherwise, on a Windows-machine it might get logged into the System Events and perhaps picked up by a third-party tool like Patrol or such.
  2. Network - Routers, firewalls, etc. can go down. SNMP seems to be the standard mechanism for reporting problems in the network.
  3. Availabilty - A machine may be up but unreachable due to high load or other things. NAGIOS is a way of catching this.
  4. Web Server - the web server may have problems with some requests. This are generally logged to a local HTTP log. Logs from multiple HTTP servers can be sent to another host via syslog, but this is rarely done.
  5. Web Application errors - in the best cases these might be logged to a local file on the server, but usually they are just displayed on the screen to the user.

There are a few interesting things to note here:

  1. Some of these error reporting solutions also include a networking protocol in addition to an error format, meaning you could assemble a centralized console for error reporting; no such luck in the web server.
  2. For applications, it's pretty much left to each developer. As a result, errors if they are logged are often local, since not many people think to use syslog for error reporting.
  3. I think the ideal thing for site maintenance is to have some sort of centralized view of errors in the system as they occur.

Anyhow, to cut an already long story somewhat short, I was working on an application that would automatically generate and process RSS feeds that I wanted to be robust and I wanted to know whenever errors occurred without having to login to the machine (ala syslog). So, I looked into some ways of reporting/logging errors when it struck me: why not use RSS as an error reporting mechanism?

You Got Your RSS in My Syslog, You Got Your Syslog in My RSS!

And so we get to my Reese's Peanut Butter Cup reference. Think about it, RSS has some real advantages to error reporting:

  1. Very simple. RSS has a lot of support across many platforms and with a wide range of viewers/aggregators. Does syslog or other networked error reporting mechanism. And it has support in programming language libraries too.
  2. Doesn't care about transport. HTTP is established and allowed through most corporate firewalls. No dumb network debugging or NAT fiddling and such.
  3. Extensible. One size doesn't fit all. I might want to record basic messages in some cases (ie, "printer on fire") or capture detailed stack traces in another (Rails logging). XML + Namespaces is a great way of layering stuff in there (although you need an RSS reader that will allow you to look at the underlying XML to get the extra data)
  4. Regular Checks. I like the pull communication of RSS because it acts like a heartbeat to your error reporting services; when something is wrong with one of my sources, I know it because my RSS reader tells me so. (to make an analogy, suppose you had a phone to be called for emergencies; does not getting any calls mean that there are no emergencies or might the line be dead?). Granted, I might not get an error immediately, but to be honest I don't think I've encountered many errors that a 5 minute delay would've made impossible to handle; don't use it your nuclear reactor though I guess.
  5. Time Management. So, if I wanted to I could have errors, warning, statistics and other things coming into my central console. I don't have an insight into the best way to manage all this information effectively, but I have a pretty good idea that problems of information overload will be tackled much sooner in the RSS world than they have been in the Syslog user base.

So, here's the idea. Define a basic set of RSS error fields in a particular XML namespace (like http://www.nimblecode.com/rsserror). This will have a set of optional tags that could be added to items. If users want to define their own extra tags they could just by declaring a new NS like (like http://www.nimblecode.com/rsserror/syslog) and add them. Really, you can add anything to XML, but the NS tags help consumers of the information to know what to expect (so, if I want to look for syslog-type errors). An example of this might be produced soon.

The Minimum Error Fields

So, if we were to make a minimal RSS error specification, what optional and required tags would it include. I have an idea:

  1. <error:msg> The text of an error message. Note that this could also be put in the title or description of the regular RSS item, but I think it's good to define an explicit place in a namespace it will always be.

And that's it. Obviously, there are other things we might want to record. For instance in my case, I think of optional tags like timestamp, extended, code, hostIp, application, etc. Not to mention some of the error information available with syslog traces, but I want to weigh what tags should go in a given namespace before I start adding options willy-nilly (indeed, there's no reason why you couldn't define your own extensions.)

So for instance, if we were defining an addition syslog RSS error namespace, we could also add the following syslog fields as tags:

  1. timestamp - a timestamp for the error
  2. priority - the priority of the message (EMERG, ERR, WARNING, INFO, etc.)
  3. facility - the facility option for syslog (AUTH, MAIL, USER, etc.)

On the other hand, if I were working on extensions for Ruby on Rail's logging format, I could include the following tags:

  1. timestamp - the time the error occurred.
  2. severity - the severity of the error
  3. sql - the SQL executed against the backend server
  4. backtrace - the call stack where the error occurred

Both of these would still use error:msg for the main message with the error. It looks like I should also make timestamp part of the standard set.

So there you have it. I will probably be working on this a bit more, but it's an interesting idea and I'm curious what other people might think.

Posted in  | Tags ,  | 2 comments

We Now Resume Our Regularly Scheduled Broadcast

Posted by harrisj Sun, 10 Jul 2005 07:01:36 GMT

Apologies for the silence recently (here I pretend I actually have readers), but I have been overwhelmed with work of late. Soon I shall be back with more posts and what I hope is insight into my little realm of computer science.

Posted in  | no comments

Intellectual Pastry Law

Posted by harrisj Fri, 17 Jun 2005 23:40:00 GMT

Intellectual Property Law is not really an area I'm expecting to write about often in this blog. Besides the fact that other people do it much better than me (see Lawrence Lessig or Declan McCullough's Politech for instance), and I must admit that I hew largely to the standard technocratic view on copyright, mainly that I am not opposed to it in principle (I'm happy to buy my music and movies), but I also think the current extensions to the duration of copyright is insane and too much power is being given to large corporations at the expense of future creativity and fair use. Nothing you won't find elsewhere, so I don't think this is an area I need to add my scintillating insight to that often. That said, I do find this story (and attached discussion) at Boing Boing titled "Copyright cops crack down on cooks over cakes" fascinating because it shows just how strange copyright has become and it's quite local to me too. The College Bakery is a nice little place in my neighborhood of Carroll Gardens where you can get lots of nice old-fashioned and wholesome pastry, cakes, pies, etc. To cut a long story short, Clay Shirky noticed the last time he was in there that they had posted the sign above. As he puts it:
Now the decor and ambience of College Bakery are echt Old Brooklyn, so it's an unlikely front in the copyfight, but the staff said they had to bust out the magic markers because they'd been roped in as the front line of defense against non-licit images of Dora the Explorer® and Thomas the Tank Engine®.
Yes, it's come to this. The large fortunes of Nickelodeon or other copyright holders are so threatened by the few custom cakes that the College Bakery makes a week that they had to take action. And certainly, I suppose there are other large bakeries that have paid licensing fees that are nominally harmed by the "infringing activity" here. But, honestly who cares? Are the property rights of copyright so paramount and absolute that it's worth denying some six year old who loves Dora (and probably has bought a fair share of Dora merchandise) one lousy cake? Do the copyright holders need to defend everything with an iron fist? More importantly, are we ceding ownership of our culture and our creativity to large corporations?

Tags , ,  | no comments

Sign Up Now for the Web1.0 Conference!

Posted by harrisj Tue, 07 Jun 2005 21:32:18 GMT

For an amusing joke (if you're a web geek that is), look at the announcement for Web 1.0 at House of Shields (Wednesday, October 5, 2005) in San Francisco. From the agenda:
We will meet to discuss line breaks, spacer gifs, and the ability to launch links in a new browser window.
Cutting edge stuff. Although I am surprised there is no seminar like "The <blink> tag: how much is too much?"

Posted in ,  | no comments

Intel Inside the Mac

Posted by harrisj Tue, 07 Jun 2005 17:50:00 GMT

Because the world doesn't have enough blog analysis on this topic already, I figure I'll wade into the fray with a few points I haven't noticed elsewhere yet (in pithy bullet form!). Update: I've decided to add some links to the best analysis I've seen so far.
  • Like most mac-heads, I must admit I feel a bit dismayed at Steve Jobs having a love fest with Intel. At the very least, I was hoping for AMD or someone else instead, but Intel? Ah well, I guess we'll get used to it.
  • Don't panic yet though. Although there will be an x86 inside, Apple is not abandoning making Macs. I imagine you won't be able to install OS X on any PC. Apple is still a hardware company and still wants to control every aspect of the machine. It will still have a Mac ROM, still boot with OpenFirmware and still have a paltry choice of video cards. It's limited I know, but I think the control that Apple has over the underlying hardware really helps to make OS X a better experience.
  • Processing power matters. Apple has managed to do more with less by doing things gracefully, but that only goes so far. To really get into the media center business, particularly the HDTV home consumer market, the Mini needs more horsepower. Heat dissipation is going to be a challenge though.
  • Some have said the crucial point behind the switch is the Pentium D's Digital Rights Management capabilities. Leander Kahney thinks this will be key for Apple wooing Hollywood studios for digital movie content, but I must lament that it will someday be impossible to find hardware without DRM built in.
  • I guess Windows emulation will be a whole lot faster now.
  • Although Apple has gained an advantage in processing power, I feel they've lost a fair amount of clout in controlling some aspects of computer design. A lot of work went into the holistic design of the G5 Powermacs, but I somehow wonder if Intel would want to do the same for Apple (holistic design isn't really part of the PC market's style). Apple is no longer a big fish in a small pond (like it was with the PowerPC), but is instead a smaller fish in a big pond. If Intel ever has to choose between making Apple happy or Dell happy, I think the lovefest will be over.
  • Intel has had a terrible track record in delivering 64-bit chips (unlike competitor AMD). I wonder how this move is going to affect people who needed the 64-bit capabilities of the G5.
  • This move also marks the continuing ascendency of the appliance part of Apple over the computer business. I think Steve Jobs is just tired of the chip design part of the business and would rather farm that aspect off to someone else (even though it means some loss of control). More importantly, unlike IBM, Intel makes more than one class of chip and I imagine the real ramifications of this move will not be high-end Pentiums in computers but lower end chips in tablets, cheaper Macs and a proliferation of appliances.
  • So, how exactly are they planning to sell any computers in the next year? Better hope iPod sales stay strong.
  • I hope I never have to see one of those Intel Inside stickers on my Mac. Those things are ugly.
Well, these certainly are interesting times. For further reading on the subject and speculation, I recommend one of the following links:

Posted in  | Tags ,  | no comments

Of Course, Episode V Is Still The Best

Posted by harrisj Tue, 24 May 2005 19:16:48 GMT

For a nice laugh (with spoilers I suppose), see the excellent top ten list of the reasons why Star Wars IV is better than the latest film at O'Reilly Radar. My particular favorite:
2. Travelling through hyperspace ain't like dustin' crops, boy.
which illustrates in a nutshell the biggest problems with the current batch of films. He should've left it at three.

Posted in  | no comments

Older posts: 1 ... 4 5 6 7 8 9