The troubles with APIs: security, discovery, bulk loading
Public APIs are hot, and getting hotter. They offer access to an amazing range of services and functions that are being built into countless commercial and enterprise applications. They are bringing us closer to the ideal of rapid assembly of apps that meet any and all business situations as they arise. However, working with APIs is not without its pain points, especially for developers.
A recent report, published by Cloud Elements, reviewed 107 public endpoints, 58 beta endpoints, 28,000 individual instances and more than 1.6 billion API calls between September 2016 and March 2017, to gauge the challenges arising with API integration. Areas such as security, discovery, bulk loading, and rate limits came up as major concerns. The study's authors, Mark Geene and Ross Garrett of Cloud Elements and API evangelist Kin Lane, observe that increasing reliance on public APIs are increasing the urgency of addressing a range of integration challenges.
The need to bridge application data from similar types of products -- such as CRM -- "is driving significant growth," Geene and his team found. "This is occurring both within and without corporate IT - which is impacting the requirements placed on integration technologies and even the APIs themselves." For example, they observe "increased API consumption and data transformation for services in the CRM, marketing automation, cloud storage and e-commerce segments -- and a growing need to simplify, speedup and automate common integration workflows - such as creating new CRM records for a purchases from new customers."
Geene and his team discussed the following challenges with APIs:
APIs as islands. The proliferation of APIs is creating new challenges for developers, Geene and his co-authors state. Today's generation of applications tend to be composite applications that mix local services with vendors' APIs. "These new composite apps help to automate workflows among multiple services, based on events and triggers." The challenge is that APIs tend to designed and built as their own little islands of standalone services -- each "has its own resource definitions, data model schema, error handling, paging structure and more.... With each new API,developers are challenged with determining how it works and what value writing to the API brings."
Security. This has been cited as the leading concern in surveys, and there are efforts to address aspects such as authentication. OAuth is the most widely accepted standard, "but there are still many APIs out there today relying on Basic Auth (17%), or some custom implementation of API Key & Secret (33%)." The survey finds that the "unloved and complicated" OAuth 1.0 (and 1.0a) is used by only 8% of providers today, compared with the more ubiquitous OAuth 2.0 at over at 41%..
Metadata discovery. API discovery is another concern, and 58% of the endpoints Cloud Elements studied support this capability -- but the authors caution this may be overstated, since many are custom integration provided by the vendor. It's likely that only a minority of APIs enable metadata discovery at this point. "With such proliferation of applications and services used in enterprise or application development and the sheer number of APIs available today, developers must carry the burden of understanding the details of each integration." While efforts such as the Open API Initiative have taken steps to enhance metadata discovery, there's still a lot of work to be done. "Developers are still challenged with the task of reading and learning virtual reams of documentation in order to integrate with the APIs they need to use," the report states, advocating that "API product managers should consider how to enable metadata discovery for their APIs - even where the data model is static."
Bulk operations. This is another challenged area, the study finds. "Bulk upload and download of data is useful for many applications, and where available we see users are keen to leverage this functionality. Yet, only 42% of APIs that Cloud Elements works with actually support bulk. This is a surprising figure, as almost all data sets in enterprise SaaS applications are substantial and require some form of bulk operations. Thus, they continue, with 58% of APIs not offering bulk support, SaaS applications make it hard to migrate data into and out of their systems. This situation acts as a barrier for migration, or synchronization between any new applications that want to seamlessly work with existing systems of record."
API rate limits. "Rate limits can make it near impossible to deal with large volumes of data," the report states. "For example, a popular e-commerce app limits applications to two calls per second, with a 'leaky-bucket' mechanism allowing up to 40 calls in backlog. This means your application must deal with bulk data at rate of 2TPS which might result in hours or even days of continuous API calls."