In the movie “The Ten Commandments,” there’s an incredible sequence of scenes in which Moses goes to Goshen (where the Israelites are toiling away on a massive monument to the pharaoh) decides that the Israelites would be better workers if they were better fed, and then orders the opening of the city’s sacred grain silos. The silos are opened and grain comes streaming forth like manna from heaven, and the Israelites practically bathe in it (skip to 37:25 in the video here).
The metaphor might be a wee bit overblown, but this is precisely the image that comes into my mind when I think about open APIs. The silos in the film, naturally, represent the Twitters and Facebooks and Googles of the world: massive storehouses of data. Open APIs amount to a major (although still far from total) de-siloing of this data, and thus to a crucial cornerstone in the internet as we have come to know its. As a consequence, understanding the current state of web development demands understanding the significance of open APIs–where they are available, how to use them, and why you should demand them where they don’t exist.
Where the APIs are
There are currently thousands of open APIs floating around the web. Something like 3,500 have been added in the last 365 days. If you’re looking to immerse yourself in this space, there are fortunately a wide variety of wonderful sources available to you. If you simply want lots of basic information on APIs, I haven’t found anything more useful than Programmable Web, which offers information on which APIs are JSON- and/or XML-based (there are also numerous other data format options), which APIs have most recently emerged, which are the most popular based on the number of mashups created from them, and so on. I would recommend using Programmable Web as your first toehold into this surprisingly vast domain.
The size and depth of the API space might surprise even those who are already steeped in the usage of APIs. There are certainly a fair number of usual suspects on the table: GoogleMaps (by all accounts the most used in the world), Twitter, Twilio, Last.fm, YouTube, Flickr, Facebook, etc. But there are many that might surprise you. The United States federal government has an API strategy. You can get information on Department of Education grants. The Library of Congress has an API. So does NASA. The “social network for scientists” Mendeley recently celebrated its 100 millionth API call. There will likely be APIs created in the coming months and years that will be even more surprising than these.
Once you’re past the information-gathering stage and are ready to put APIs to work, the old-fashioned way of doing so is by simply going to the relevant docs and learning the API on your own. But if you’re interested in using APIs in IDE-style settings, tools like Apigee, Mashery, and Temboo can be incredibly powerful. They’re designed both for those using APIs as third parties as well as for organizations that want to produce their own developer-friendly APIs.
Hey, information hogs: start sharing
According to virtually anyone writing about APIs these days–and it’s difficult not to agree–if you’re operating a highly-trafficked node somewhere in the WWW and that node begins turning into a silo of information of high value to others, then it might be high time to do some de-siloing of your own. One could make the argument that certain kinds of information should by nature be shared. I often find those arguments compelling, but there are plenty of convincing self-interest arguments for setting up APIs and releasing precious grain into the world.
For companies like Twitter and Facebook, having open APIs is essential to their bottom line. Twitter gains from developers making applications like TweetDeck and thousands of others. Facebook benefits from the extensive use of its API by companies like Zynga. But in order to build a massive developer community around your API, you have to make your API easy to use and you should apply only the most necessary of restrictions to it. Why do you need third-party developers on your side? According to Apigee’s blog, there are simply “too many devices, too many platforms, and too many opportunities for any company, even the largest in the world, to cover all the bases alone. If developers have the tools to innovate, a way to get recognition for their work, and a means to get paid, an API can spark amazing innovation.”
Face it: you will never be as creative with your data as a veritable army of creative developers can be. If you think that you can squeeze all the conceivable value out of your data and do so across all conceivable devices and user interfaces, you are dead wrong, no matter who you are. If you’re BestBuy, you will suffer if people have to go directly to your website/app to buy things rather than purchasing them in settings produced by third parties. That’s why BestBuy offers monetary incentives in the form of sales commissions to those who enable users to purchase from BestBuy. eBay and others have followed a similar strategy. The percentage of companies that follow analogous API strategies will continue to grow and could approach 90% or more within just a few years.
The intrinsic dangers involved with using APIs
As we know from several hiccups and snafus over the last few months, not all is golden in the land of APIs. The case of Twitter and Netflix restricting API usage has demonstrated the danger that developers face when the information spigot is turned off. The case of the LinkedIn-Twitter controversy is also deeply instructive. Although LinkedIn’s use of the Twitter API wasn’t necessarily an essential part of their broader business logic, it still struck many as an unnecessary set of restrictions. It’s easy to see why someone could refer to scenarios like this as cutting off oxygen to broad swathes of developers–not to mention businesses.
If you’re a developer, you might spend months of your life constructing a SaaS offering that presupposes the availability of certain types of information from specific endpoints. If the manager of that endpoint decides to impose unforeseen restrictions or to turn off the spigot entirely, this could mean the death of your current business logic.
While Twitter has rightfully received flak at multiple points in time, Google has proven to be something of a model citizen in the API polity. They provide a straightforward API console and a whole slew of differentiated APIs under the broader Google umbrella, including APIs for Google Analytics, the Drive SDK, and more. Perhaps more importantly, they haven’t engaged in any of the questionable API restriction that we have seen from the likes of Twitter. While there are restrictions on the volume of API access that they allow free of charge, that’s fundamentally different from a content-based restriction.
Another fundamental danger surrounding APIs involves you, not “them.” If you set up an API (mad props to you already), you must do so with scalability in mind. Imagine the following scenario: your site starts getting big, your API starts getting tons of love, and all of a sudden it can’t handle all of the API calls. This leads to heavier and heavier latency and degraded performance. The take-home lesson: if you’re going to set up an API, make sure you do so with a back-up plan in case it begins receiving heavy traffic. Scalability comes in many forms nowadays, from your own hardware to VPS to Platform-as-a-Service, and you should have an appropriate game plan in place well before any potential bottlenecks. We can only expect that the future will be bright for companies like Mashery who offer consulting and a wide variety of other API management-related services.
But wait: didn’t I just try to convince you to produce an open API if you begin–perhaps unwittingly–accumulating a silo of data? So, if that happens, I’m supposed to both put together an elegant and well-documented API and expend resources on scaling it? Yes. That’s exactly what I meant. Welcome to the WWW in 2012.
What this all means for you
In spite of explosive growth in the API domain, there’s still plenty of web development to be done without the use of open APIs. But I would still implore developers of all stripes to do some committed exploration. Take a look at, for example, the list of APIs supported by Apigee and ask yourself: is there yet-to-be-explored value to be had in using these APIs? Are there services waiting to spring forth from the woodwork? Is there an Eventbrite-Foursquare-GroupOn mashup waiting to happen? GoogleMaps-Pivotal Tracker? World of Warcraft-LinkedIn?
Or to put it in historical perspective: in 2005, Programmable Web had only 32 APIs in its directory. Today, in August 2012, it has over 7,000, and the growth has been accelerating. In October 2011, less than a year ago, Programmable Web listed only 4,000 APIs (according to this source).
In response to these developments, New Relic’s Patrick Moran has argued that “today, companies can’t succeed without a data API for hackers. Ask Box.net. Ask Salesforce. Ask any relevant software company. Social. Mobile. Global. Big Data. APIs. These software strategies are more critical than ever before.” His advice: “Build tools for developers. Then win.”
To put it even more succinctly, I’ll borrow the words of Moses from the film: “Bring the push-pole men and some women with baskets.” There’s already a lot of grain to be had and there’s exponentially more on the way.