APIs as Economic Game Changers – InApps Technology 2022

The year 2015 has been a defining year in the growth of the API economy. The recent API Strategy and Practice conference in Austin, Texas, demonstrated the widening group of business and civic players reorienting towards an API play, with ever-more stakeholders now offering services in the API ecosystem.

And last week, the global APIdays event — held annually in Paris and organized by FABERnovel and OAuth.io — provided an opportunity to summarize some of the key themes of the year and to peek into what lies ahead for 2016.

It was clear from key conversations around using APIs for automation, and the role of APIs in enabling new business economic growth, that the API economy must straddle a widening market landscape that meets the needs of new entrants alongside more advanced users.

At one end of the spectrum, businesses are sipping the kool-aid for the first time and learning how APIs can be a key enabler in their digital transformation. At the other end of the spectrum, businesses and enterprises are leveraging APIs to automate whole processes and IT systems, build new products and reorient their business models.

Meanwhile, on a technical level, APIdays demonstrated how quickly the space is evolving. Just last month at API Strategy and Practice, it was clear that the strongest sign of maturity was the widespread adoption of API definition formats to enable business and technical collaboration, and to automate and extend the API lifecycle.

At APIdays, API definition formats were almost a given and instead, the technical envelope being pushed forward centered around event notifications (one of the technical session tracks that was stirred up at API Strategy and Practice but that by the time of APIdays had now swelled into a wave of new practice).

Automation and Machine Learning

As the maturing market is widening to encompass business players operating at all levels, from novice to sophisticate, the uptake of APIs for automation is also widening.

Automation may start as simple integrations (like the ‘recipes’ available through If-This-Then-That, IFTTT), or the emergence of Slackbots (like the new Ottspott Slackbot that enables Twilio-like telco functionality from within the team collaboration tool). The growth of API-enabled automation is evident, said Cyril Vart from digital transformation consultancy FABERnovel (one of the organizers of the conference), in API aggregation service IFTTT’s raising of $38.5 million, and their current usage rate of some 30-35 million API triggers occurring daily through the service.

Beyond simple API integrations, and in a similar manner to the exploding infrastructure automation stack for containers, integration-as-a-service tooling is evolving. At the same time as the APIdays conference was being held, Built.io Flow announced a partnership with the Cisco Spark, offering the same type of workflows within the one platform that IFTTT and Ottspott are offering individually. Meanwhile, API-wrapper service Blockspring announced a partnership with Bubble to enable the creation of apps that integrate data from APIs without users needing to know any code.

The next frontier for API automation, however, was mentioned in keynotes by CA Technologies’ Peter Matthews and from IBM’s Andy Thurai. Both pointed to techniques aimed at providing an automated service discovery element for APIs. When you look at how Built.io Flow operates, or even how Slack is integrating tools, you can see how API discoverability needs to be automated into a platform’s ecosystem.

This is where API definition formats can be a key enabler in the same way that they are being used to automate lifecycle development. If there is enough metadata about an API, a platform like Slack or Built.io Flow could identify what process is about to be undertaken and then scan their internal integration directories to see which APIs are best suited for doing the task at hand.

Peter Matthews

What Matthews and Thurai were proposing were automated discovery systems that would identify task needs and then make an assessment of which API to use based not just on functionality but perhaps also on security, performance, cost and reliability.

Matthews described his work with ModaClouds, a research project that “built an IDE that would allow you to compose applications that let you use a decision support system that would take inputs from requirements analysis and recommend services based on functional or non-functional requirements,” he said.

ModaClouds is “now a whole set of open source software: the methodology and the set of tools to allow you to create apps quickly using services that you have evaluated for cost, performance and risk, and even a monitoring tool for monitoring SLAs,” Matthews described.

“Now we are taking that concept further in terms of not an IDE approach but service detection and monitoring services in a security matrix (evaluated by performance availability and security vulnerability risk).”

Thurai pointed to IBM’s Harmony project — which he described in a similar way to how Bruno Pedro from API Changelog mapped out API automation at a previous APIdays — as a proof of concept for switching out APIs in the middle of real-time workflow processes based on parameters such as computing cost or reliability.

Andy Thurai

But here’s where the discussion on API automation — like AI and machine learning often does — ends up in the land of possibilities rather than actualities. If you look for further details on IBM’s API Harmony, you see that really this is just a proof of concept idea that has not yet been realized. You can sign up for the beta but the forums are empty: there is no product as yet.

The same with ModaClouds, which Matthews was quick to make clear from the beginning of his presentation: “This is the research end of it, we don’t even know if we can do it, yet.”

Given the examples available to us in the container system for service discovery, and the growth of API lifecycle ecosystems, projects like Harmony and ModaClouds are set to become the key next wave of API tooling to be created in 2016.

This mismatch between hype and reality is a common pattern in technology discussions, but perhaps particularly at the sort of stage of growth that the API economy is seeing: as the market expands to include new adopters alongside veterans who are now quite advanced in their usage, there becomes a disconnect between the horizon of possibility and the current capacity to deliver. It is almost like the API economy is trying to find a way to leap across the trough of disillusionment.

Building some bridge across the chasm between current technological prowess and the expectations around where we think we are is most evident in the machine learning and automation space. IBM Watson’s Ashley Hathaway made this clear in her talk, as did Louis Dorard, author of Bootstrapping Machine Learning, and Daniel Benoilid, CEO of Foule Factory.

All three shared an often forgotten truth behind many machine learning and artificial intelligence projects,  whether that be Foule Factory’s Mechanical Turk-like service, Facebook’s M or IBM’s Watson: automation and prediction are half compute-algorithm and half human fact-checking and confirmation. For now, at least, humans are the last mile of machine learning automation. Machine learning and predictive tools still need a lot of sophisticated human input to complete their tasks.

APIs as an Economic Game Changer

While automation may have been the public theme of the conference, a key concept that many speakers kept coming back to was how much APIs are the center of the digital transformation of business, and how important APIs are in creating an economic impact for business.

Kristin Moyer, from Gartner, sees APIs as the key unit of value that will enable many businesses to transform their business models and enable a platform play that encourages end users to compose their own value.

Kristin Moyer

Neha Sampat, CEO of Built.io — who outlined several of her predictions for the coming year — said that this year, when talking with businesses about integration with APIs, “a lightbulb goes off: I have never seen anything like it”. Sampat is seeing firsthand what 451 Research have pointed out will be a key market shift in 2016. Integration service providers like Built.io will need to meet API and cloud early adopters who are now looking for “different kinds of external services” (again, hinting at the need for automated service discovery in the API economy).

Neha Sampat

Repeatedly, speakers came back to the theme of how much the emerging market leaders are all using APIs to gain their strategic and business advantage:

• Matthews from CA Technologies says that in eight years time, three-quarters of Standard & Poors’ Top 500 businesses will be companies we have not even heard of yet. Many of those will be there because they are using APIs today, Matthews says, pointing to research that shows that companies already winning the digital transformation agenda are 2.8 times more likely to be using APIs.
• Vart demonstrated how the next wave of tech ‘unicorns’ are accelerating their growth by using APIs that plug into GAFA (Google, Apple, Facebook and Amazon) infrastructure. Uber, for example, have been able to grow quickly because their app is available in the iTunes and Google Play stores, uses Facebook to allow instant user registration, and is managed from Amazon’s S3 storage and global web service infrastructure.
• Moyer pointed to research that shows that enterprises using APIs as a business channel, see net revenue growth of 30% while those using APIs more pervasively internally and with third party developers see time (and cost) to-market reduce by up to 90%.

Amongst these opportunities, keynote Steven Willmott, CEO of 3scale, urged a model in which APIs are used to leverage greater equity of opportunity. Rather than businesses seeing a narrow API call monetization opportunity, Willmott challenged businesses to look at how to use APIs to create a new form of platform business that enables everyone to be creators.

Steven Willmott

While Willmott spoke more practically about designing monetization models that could foster a co-creation agenda, closing keynote philosopher Bernard Stiegler urged the importance of leveraging APIs as a means to connect citizens rather than lead us further into the current Facebook commodification of society, where the users themselves are the product in an automated, soulless future based purely on consumption.

Bernard Stiegler

Event Notifications: Are We There Yet?

Just one month ago, at API Strategy and Practice, one of the key themes obviously driving discussions was the API definition format. That is still a key technical enabler, as it draws an API into a whole product lifecycle that can help a business better understand the value of their API program, redefine the business’ API strategy as product, and link their APIs to a range of tools (including via automation) in a similar way to what we see in the container ecosystem.

But with that now accepted, the discussion has already moved on. Perhaps the clearest signpost to the next technical need was in discussions on event notifications. Highlighted in a keynote by Saul Caganoff, CTO of Australian digital consultancy Sixtree and co-organizer of APIdays Australia, the API ecosystem must solve the problem of how updated and real-time information can be shared without requiring a constant stream of call-response plays. Has the data been updated yet? No. Has the data been updated yet? No. Has the data…

Saul Caganoff

Caganoff points to three main techniques currently being used by API architects:

• Syndication: This involves some latency and polling. RSS is the key example of this type of event notification. API calls can ask for updated information, perhaps every ten minutes.
• Webhooks: This emerging standard, being advanced particularly by Zapier’s Resthooks model, is more difficult for end users to deal with, as they need to set up an API endpoint, but it reduces some of the latency and polling problems. The idea is built on establishing an endpoint or URL to which details of any update can be sent when they occur.
• A new generation of notification services: This is where Caganoff sees the API economy heading with players including IronMQ, Streamdata.io, Push Technology’s Reappt, and Pubnub all offering a new wave of notification technologies that enable more real-time data notification and processing.

Eric Horesnyi, CEO of Streamdata.io, explained how event notification and stream processing is being technically implemented. Horesnyi says industry best practice today is to have a UX-API end-to-end event driven stream architecture made up of Reactive + HTTP/2 + Server-sent event notifications. JSON Patch operations are implemented to make incremental updates closer to the server, minimizing bandwidth when sending calls and providing updates only on the parts of the data that have been updated.

Caganoff says that an enterprise’s unique business value today depends on how they integrate and operate infrastructure, commodity, function, and industrial services in a way that creates an end-to-end business process. At present, as businesses stitch together internal and external assets via API, they deal with techniques like rate limiting to manage the number of times an API can ask “has the data been updated yet?” As more enterprises require real-time access to data-flows, rate limiting fails to scale, requiring a new breed of event notification services such as those that Horesnyi describes.

In the container age, APIs have been taking a backseat in terms of technology discussions. But they are an essential component in the engine that makes container technology a possibility: encouraging the composability of microservices and the creation of an endless variety of value chains. As cloud technologies create greater expectations that all businesses will be digital, APIs will be at the core of making that transformation possible. As the market has matured in 2015, there is now a wider variety of services and possibilities for both newcomers and those with some expertise. How automation, service discovery and event notification are made accessible will be the industry’s defining work in the first half of 2016.

Cisco, Docker and IBM are sponsors of InApps Technology.

Mark Boyd helped organize APIdays Paris and has written for the APIdays team.

Feature image “APIdays Paris” by Benjamin Boccas, used by permission from the APIdays Flickr collection.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker, Bubble.