Pushing Data, Not Pages is the New Model for Application Development

Big Kite by Syntopia A crop of tools for building applications in a new way has emerged over the past month. The way developers build applications has changed over the past few years. The old client-server model involved doing most of the work on the server side and then piping the results down to a dumb client. That might be a command line application crunching numbers and then displaying the results to a terminal application, or it might be a complex Ruby on Rails application transforming data queried from MySQL and rendering it as an HTML page that is sent to a browser. But to create Web applications that are both feature rich and responsive, like Gmail and Facebook, developers have put more and more of the application in the browser. Add the inconsistent network connections of mobile devices to the mix, and you’ve got a host of new development challenges.

That’s where these new tools come in. Meteor, Mojito and Firebase have joined CouchApps in supporting the modern paradigm of data-centric model of development. What they all have in common is a design philosophy of sending data to apps, not rendered pages. Applications can then process the data and send the changes back to the server. The cloud is still important – extremely heavy lifting can still be done server side, but the many of the smaller processing actions will be done on the client side. Data will be stored in the cloud, but cached locally. Clients get smarter and servers become less visible.

Frameworks

The Meteor team thinks they’ve foundt a solution in providing a framework for using JavaScript for both client side and server side development. Developers can use Meteor to build real-time applications that live mostly in the browser, and even push code changes to users while the application is in use. Meteor doesn’t replace AJAX libraries like jQuery, instead it helps AJAX applications get smarter.

The founders of Meteor have all been dealing with the new realities of Web application development for years. Prior to Meteor, co-founder Geoff Schmidt worked at Asana building a similar JavaScript framework called Lunascript (he’s written a comparison of Meteor and Lunascript). Another co-founder, David Greenspan, co-founded AppJet, the company that invented Etherpad. Etherpad, a real-time collaborative text editor, was originally a demo of the AppJet’s server side JavaScript platform. But it became the company’s flagship project before Google acquired the company and funneled the AppJet team into Google Wave.

Most of the Meteor team met at MIT. “We’re all old friends from back in Boston,” Meteor co-founder Matt Debergalis told me in an interview this week. Greenspan didn’t go to school with the rest of the founders, but his co-founder at AppJet, Aaron Iba, did go to MIT with Schmidt.

“I was at David’s holiday party in SF, and after a drink or two we were talking about what we’d each been working on, and I told him a little bit about Meteor, which was a few months old at that point,” Schmidt explained on the questions and answers site Quora this month. Greenspan was skeptical since he’d already been down the server side JavaScript road, but Schmidt convinced him that AppJet had simply been too early.

Asana is an example of something that’s hard to build without something like Meteor or Lunascript. Companies like Google and Facebook, and well funded startups like Asana, can hire lots of extremely talented engineers to build this sort of stuff, but most startups don’t have the resources to build a framework from scratch. So far Asana is keeping Lunascript in house, so Debergalis thinks Meteor, which is open source under the MIT license, will help level the playing field. But it’s not ready yet.

“We released the prototype to show people our thinking and introduce them to the paradigm,” said Debergalis. But the unfinished prototype confused some people. For example, many developers weren’t sure how Meteor would handle authentication with a database – some thought the clients would connect directly to a database without any sort of intermediary. Debergalis explains that’s not the case. Meteor apps running in the browser will still connect to a Meteor app running on a server. The server application will handle authentication with a database. The client will cache data from the server in order to make apps more responsive. The server provides copies of the objects to the client, which simulates server responses.

Meteor has also been criticized for the way it uses Node.js, an increasingly popular server side JavaScript platform. Typically Node.js uses non-blocking callbacks, which is part of the platforms appeal. Meteor abstracts callbacks away using node-fibers to allow developers to use a more traditional linear execution model. Although Node.js contributor Mikeal Rogers claims that developers using “blocking style” APIs are a vocal minority in the Node.js community, Debergalis defends Meteor’s use of them saying that linear programming is easier. “Node.js’s event driven model is good for some stuff, but not for everything. As apps grow in size and complexity you might want to program more traditionally,” Debergalis says. “For most programmers the asynchronous model is difficult.”

Another possible area for concern is that Meteor doesn’t use the MVC (model–view–controller) model typical to other Web application frameworks like Ruby on Rails. Meteor is comparable to another new JavaScript framework, Mojito, which was recently open sourced by Yahoo. Unlike Meteor, however, Mojito brings the MVC paradigm to JavaScript/Node.js application development. Yahoo’s Chief Platform Architect Bruno Fernandez-Ruiz tells me that the lack of an MVC in Meteor might cause developers issues with structure and maintainability.

Debergalis disagrees that the MVC model makes the most sense for the modern application development environment where much of the application logic lives in the client and mobile devices may have unreliable Internet connections. “MVC makes sense in the server central model,” he says. “The problem is that in the MVC model the data lives too far from the app. It assumes you have a fast, reliable connection to a data source, which isn’t always true.” In the JS world, MVC isn’t quite right,” he says. “It’s too early to know what’s right, but I think something new will emerge and it will borrow heavily from MVC, but won’t be MVC.”

Databases

Mojito and Meteor just take care of the application framework. You’ll still need a data store of some sort. There’s been rapid innovation in databases over the past 5 years or so, spurred on by the NoSQL movement. But the launch of Firebase proves that there’s still more room for more.

Like Meteor, Firebase is especially well suited for real-time applications. Firebase is a non-relational/NoSQL database hosted in the cloud. Firebase takes care of data storage and replication, synchronization between clients and scalability.

Developers can then access the database directly through an API. It reminds me a bit of “CouchApps” – HTML/JavaScript apps that connect directly to Apache CouchDB with no middleware. In fact, Firebase can probably best be classified as a document database, although Firebase co-founder Andrew Lee explains that it’s quite a bit different from other document databases like CouchDB and MongoDB. While a normal document database contains a number of theoretically discrete documents, Firebase provides one large hierarchical JSON document. Lee explains that if Microsoft were to build its site on Firebase, it would have one single document/database titled “Microsoft” with subsections for each of its divisions, with each of those divisions containing subdirectories for different products (for example Microsoft -> Office -> Excel). A Firebase app can synchronize only the data that it needs, so you can use the same Firebase for multiple applications. Most importantly, since Firebase data looks just like a typical JavaScript object it’s easy for JavaScript applications to interact with, further blurring the line between client and server development.

Firebase talks about making servers optional, but you will still need to host your HTML and JavaScript files somewhere. When writing a CouchApp, however, CouchDB acts as both the HTTP server and as the database server. But the big downside is that users will need to have CouchDB installed locally (one open source project, PouchDB is attempting to solve this problem enabling synchronization between CouchDB and IndexedDatabase, a local data store supported as part of the HTML5 spec).

Although its founders have described Firebase as “It’s like Dropbox for your app’s data,” it serves another useful purpose: it’s a CDN (content delivery network) for data. In a model where an application server is connecting to a database server and then rendering pages to serve to a client, it makes sense to keep the application server and database server as close together as possible. But if you’re running your app in the browser and connecting directly to the database, then you may want to have that data available from multiple geographic locations. What a CDN does for static content, Firebase can do for dynamic content.

One potential downside is that although Firebase uses the open JSON format, and you can export your JSON data at any time, there’s really nothing else like it on the market. That means that if you build an application for use with Firebase and later decide to use something else, you’ll have to completely rewrite your application. But that’s a risk that comes with using any sufficiently new database. Lee says the company might open source some of Firebase’s technology in the future and that the company sees its managed hosting as its real value.

As with Meteor, authentication is still an issue. For now, all Firebase applications are publicly writable, which limits the database’s utility.

Overall, with the possible exception of Mojito, all of these tools are still immature. There are still many unsettled arguments about how to architect the new breed of application. But each of these tools points in the direction that Web-based, mobile-centric application development is going.

Image by Syntopia

About Klint Finley

Klint Finley is a Senior Writer at SiliconAngle. His specialties include IT services, enterprise technology and software development. Prior to SiliconAngle he was a writer for ReadWriteWeb. He's also a former IT practicioner, and has written about technology for over a decade. He can be contacted at angle@klintfinley.com.