These are the news items I've curated in my monitoring of the API space that have some relevance to the API showcase conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is testing their APIs, going beyond just monitoring and understand the details of each request and response.22 Aug 2018
I’m evolving the search for the Streamdata.io API Gallery I’ve been working on lately. I’m looking to move the basic keywords search that searches the API name and description, as well as the API path, summary, and description using a key word or phrase, to also be about searching parameters in a meaningful way. Each of the APIs in the Streamdata.io API have an OpenAPI definition. It is how I render each of the individual API paths using Jekyll and Github Pages. These parameters give me another dimension of data in which I can index, and use as a facet in my API gallery search.
I am developing different sets of vocabulary to help me search against the parameters used across APIs, with one of them being focused on company related information. I’m trying to find APIs that provide the ability to add, update, and search against company related data, content, and execute algorithms that help make sense of company resources. There is no perfect way to search for API parameters that touch on company resources, but right now I’m looking for a handful of fields: company, organization, business, enterprise, agency, ticker, corporate, and employer. Returning APIs that have a parameter with any of those words in the path or summary, and weighting differently if it is in the description or tags for each API path.
Next, I’m also tagging each API path that has a URL field, because this will allow me to connect the dot to a company, organization, or other entity via the domain. This is all I’m trying to do, is connect the dots using the parameter structure of an API. I find that there is an important story being told at the API design layer, and API search and discovery is how we are going to bring this story out. Connecting the dots at the corporate level is just one of many interesting stories out there, just waiting to be told. Pushing forward the conversation around how we understand the corporate digital landscape, and what resources they have available.
You can do a basic API search at the bottom of the Streamdata.io API Gallery main page. I do not have my parameter search available publicly yet. I want to spend more time refining my vocabularies, and also look at searching the request and response bodies for each path–I’m guessing this won’t be as straightforward, as parameters has been. Right now I’m immersed in understanding the words we use to design our APIs, and craft our API documentation. It is fascinating to see how people describe their resources, and how they think (or don’t think) about making these resources available to other people. OpenAPI definitions provide a fascinating way to look at how APIs are opening up access to company information, establishing the digital vocabulary for how we exchange data and content, and apply algorithms to help us better understand the business world around us.
I was working on a serverless app for Streamdata.io that takes posts to Hacker News and streams them into an Amazon S3 data lake, and I came across the Algolia powered Hacker News search API. After being somewhat frustrated with the simplicity of the official Hacker News API, I was pleased to find the search kindly provided by Algolia.
There is no search API available for the core Hacker News API, and the design leaves a lot to be desired, so the simplicity of Algolia’s API solution was refreshing. There is a lot of data flowing into Hacker News on a regular day, so providing a search API is pretty critical. Additionally, Algolia’s ability to deliver such a simple, usable, yet powerful API on top of a relevant data source like Hacker News demonstrates the utility of what Algolia offers as a search solution–something I wanted to take a moment to point out here on the blog.
I consider search to be an essential ingredient for any API. Every API should have a search element to their stack, allowing the indexing and searching of all API resources through a single path. Making Algolia a relevant API service provider in this area, enabling API providers to outsource the indexing and searching of their resources, and the delivery of a dead simple API for your consumers to tap into. This path forward is probably not for every API, as many weave specialized search throughout their API design, but for teams who are lacking in resources, and can afford to outsource this element–Algolia makes sense.
Seeing Algolia in action, for a specific API I was integrating with helped bring their service front and center for me. I tend to showcase Elastic for deploying API search solutions, but it is a good to receive a regular reminder that Algolia does the same thing as a service. Their work on the Hacker News Search API provides a good example of they can do for you–sure, we can all build our own search solutions, but honestly, do you have the time? I’ll make sure and regularly highlight what Algolia is doing as part of my search API research, and thanks Algolia! I really appreciate what you did for the Hacker News API, it made my work a lot easier.
When you are operating an API, you are always looking for new ways to be discovered. I study this aspect of operating APIs from the flip-side–how do I find new APIs, and stay in tune with what APIs are to? Historically we find APIs using ProgrammableWeb, Google, and Twitter, but increasingly Github is where I find the newest, coolest APIs. I do a lot of searching via Github for API related topics, but increasingly Github topics themselves are becoming more valuable within search engine indexes, making them an easy way to uncover interesting APIs.
I was profiling the market data API Alpha Vantage today, and one of the things I always do when I am profiling an API, is I conduct a Google, and then secondarily, a Github search for the APIs name. Interestingly, I found a list of Github Topics while Googling for Alpha Vantage API, uncovering some interesting SDKs, CLI, and other open source solutions that have been built on top of the financial data API. Showing the importance of operating your API on Github, but also working to define a set of standard Github Topic tags across all your projects, and helping encourage your API community to use the same set of tags, so that their projects will surface as well.
I consider Github to be the most important tool in an API providers toolbox these days. I know as an API analyst, it is where I learn the most about what is really going on. It is where I find the most meaningful signals that allow me to cut through the noise that exists on Google, Twitter, and other channels. Github isn’t just for code. As I mention regularly, 100% of my work as API Evangelist lives within hundreds of separate Github repositories. Sadly, I don’t spend as much time as I should tagging, and organizing projects into meaningful topic areas, but it is something I’m going to be investing in more. Conveniently, I’m doing a lot of profiling of APIs for my partner Streamdata.io, which involves establishing meaningful tags for use in defining real time data stream topics that consumers can subscribe to–making me think a little more about the role Github topics can play.
One of these days I will do a fresh roundup of the many ways in which Github can be used as part of API operations. I’m trying to curate and write stories about everything I come across while doing my work. The problem is there isn’t a single place I can send my readers to when it comes to applying this wealth of knowledge to their operations. The first step is probably to publish Github as its own research area on Github (mind blown), as I do with my other projects. It has definitely risen up in importance, and can stand on its own feet alongside the other areas of my work. Github plays a central role in almost every stop along the API life cycle, and deserves its own landing page when it comes to my API research, and priority when it comes to helping API providers understanding what they should be doing on the platform to help make their API operations more successful.
I was looking at the set of security APIs over at Elasticsearch as I was diving into my API security research recently. I thought the areas they provide security APIs for the search platform was worth noting and including in not just my API security research, but also search, deployment, and probably overlap with my authentication research.
- Authenticate API - The Authenticate API enables you to submit a request with a basic auth header to authenticate a user and retrieve information about the authenticated user.
- Clear Cache API - The Clear Cache API evicts users from the user cache. You can completely clear the cache or evict specific users.
- User Management APIs - The user API enables you to create, read, update, and delete users from the native realm. These users are commonly referred to as native users.
- Role Management APIs - The Roles API enables you to add, remove, and retrieve roles in the native realm. To use this API, you must have at least the manage_security cluster privilege.
- Role Mapping APIs - The Role Mapping API enables you to add, remove, and retrieve role-mappings. To use this API, you must have at least the manage_security cluster privilege.
- Privilege APIs - The has_privileges API allows you to determine whether the logged in user has a specified list of privileges.
- Token Management APIs - The token API enables you to create and invalidate bearer tokens for access without requiring basic authentication. The get token API takes the same parameters as a typical OAuth 2.0 token API except for the use of a JSON request body.
Come to think of it, I’ll add this to my API management research as well. Much of this overlaps with what should be a common set of API management services as well. Like much of my research, there are many different dimensions to my API security research. I’m looking to see how API providers are securing their APIs, as well as how service providers are selling security services to APIs providers. I’m also keen on aggregating common API design patterns for security APIs, and quantity how they overlap with other stops along the API lifecycle.
While the cache API is pretty closely aligned with delivering a search API, I think all of these APIs provide a potential building block to think about when you are deploying any API, and represents the Venn diagram that is API authentication, management, and security. I’m going through the rest of the Elasticsearch platform looking for interesting approaches to ensuring their search solutions are secure. I don’t feel like there are any search specific characteristics of API security that I will need to include in my final API security industry guide, but Elasticsearch’s approach has re-enforced some of the existing security building blocks I already had on my list.
I was learning about the microservices discovery specification Pivio, which is a schema for framing the conversation, but also an uploader, search, and web interface for managing a collection of microservices. I found their use of ElasticSearch as the search engine for their tooling worth thinking about more. When we first launched APIs.json, we created APIs.io as the search engine–providing a custom developed public API search engine. I hadn’t thought of using ElasticSearch as an engine for searching APIs.json treated as a JSON document.
Honestly, I have been relying on the Github API as the search engine for my API discovery. Using it to uncover not just APIs.json, but OpenAPI, API Blueprint, and other API specification formats. This works well for public discovery, but I could see ElasticSearch being a quick and dirty way to launch a private or public engine for an API discovery, catalog, directory, or type of collection. I will add ElasticSearch, and other platforms I track on as part of my API deployment research as a API discovery building block, evolving the approaches I’m tracking on.
It is easy to think of API discovery as directories like ProgrammableWeb, or marketplaces like Mashape, and public API search engines like APIs.io–someone else’s discovery vehicle, which you are allowed to drive when you need. However, when you begin to consider other types of API discovery search engines, you realize that a collection of API discovery documents like JSON Home, Pivio, and APIs.json can quickly become your own personal API discovery vehicle. I’m going to write a separate piece on how I use Github as my API discovery engine, then I think I’ll step back and look at other approaches to searching JSON or YAML documents to see if I can find any search engines that might be able to be fine tuned specifically for API discovery.
I am finally getting the time to invest more into the rest of my API industry guides, which involves deep dives into core areas of my research like API definitions, design, and now deployment. The outline for my API deployment research has begun to come into focus and looks like it will rival my API management research in size.
With this release, I am looking to help onboard some of my less technical readers with API deployment. Not the technical details, but the big picture, so I wanted to start with some simple questions, to help prime the discussion around API development.
- Where? - Where are APIs being deployed. On-premise, and in the clouds. Traditional website hosting, and even containerized and serverless API deployment.
- How? - What technologies are being used to deploy APIs? From using spreadsheets, document and file stores, or the central database. Also thinking smaller with microservices, containes, and serverless.
- Who? - Who will be doing the deployment? Of course, IT and developers groups will be leading the charge, but increasingly business users are leveraging new solutions to play a significant role in how APIs are deployed.
The Role Of API Definitions While not every deployment will be auto-generated using an API definition like OpenAPI, API definitions are increasingly playing a lead role as the contract that doesn’t just deploy an API, but sets the stage for API documentation, testing, monitoring, and a number of other stops along the API lifecycle. I want to make sure to point out in my API deployment research that API definitions aren’t just overlapping with deploying APIs, they are essential to connect API deployments with the rest of the API lifecycle.
Using Open Source Frameworks Early on in this research guide I am focusing on the most common way for developers to deploy an API, using an open source API framework. This is how I deploy my APIs, and there are an increasing number of open source API frameworks available out there, in a variety of programming languages. In this round I am taking the time to highlight at least six separate frameworks in the top programming languages where I am seeing sustained deployment of APIs using a framework. I don’t take a stance on any single API framework, but I do keep an eye on which ones are still active, and enjoying usag bey developers.
Deployment In The Cloud After frameworks, I am making sure to highlight some of the leading approaches to deploying APIs in the cloud, going beyond just a server and framework, and leveraging the next generation of API deployment service providers. I want to make sure that both developers and business users know that there are a growing number of service providers who are willing to assist with deployment, and with some of them, no coding is even necessary. While I still like hand-rolling my APIs using my peferred framework, when it comes to some simpler, more utility APIs, I prefer offloading the heavy lifting to a cloud service, and save me the time getting my hands dirty.
Essential Ingredients for Deployment Whether in the cloud, on-premise, or even on device and even the network, there are some essential ingredients to deploying APIs. In my API deployment guide I wanted to make sure and spend some time focusing on the essential ingredients every API provider will have to think about.
-Compute - The base ingredient for any API, providing the compute under the hood. Whether its baremetal, cloud instances, or serverless, you will need a consistent compute strategy to deploy APIs at any scale. -Storage - Next, I want to make sure my readers are thinking about a comprehensive storage strategy that spans all API operations, and hopefully multiple locations and providers. -DNS - Then I spend some time focusing on the frontline of API deployment–DNS. In todays online environment DNS is more than just addressing for APIs, it is also security. -Encryption - I also make sure encryption is baked in to all API deployment by default in both transit, and storage.
Some Of The Motivations Behind Deploying APIs In previous API deployment guides I usually just listed the services, tools, and other resources I had been aggregating as part of my monitoring of the API space. Slowly I have begun to organize these into a variety of buckets that help speak to many of the motivations I encounter when it comes to deploying APIs. While not a perfect way to look at API deployment, it helps me thinking about the many reasons people are deploying APIs, and craft a narrative, and provide a guide for others to follow, that is potentially aligned with their own motivations.
- Geographic - Thinking about the increasing pressure to deploy APIs in specific geographic regions, leveraging the expansion of the leading cloud providers.
- Virtualization - Considering the fact that not all APIs are meant for production and there is a lot to be learned when it comes to mocking and virtualizing APIs.
- Data - Looking at the simplest of Create, Read, Update, and Delete (CRUD) APIs, and how data is being made more accessible by deploying APIs.
- Database - Also looking at how APIs are beign deployed from relational, noSQL, and other data sources–providing the most common way for APIs to be deployed.
- Spreadsheet - I wanted to make sure and not overlook the ability to deploy APIs directly from a spreadsheet making APIs are within reach of business users.
- Search - Looking at how document and content stores are being indexed and made searchable, browsable, and accessible using APIs.
- Scraping - Another often overlooked way of deploying an API, from the scraped content of other sites–an approach that is alive and well.
- Proxy - Evolving beyond early gateways, using a proxy is still a valid way to deploy an API from existing services.
- Rogue - I also wanted to think more about some of the rogue API deployments I’ve seen out there, where passionate developers reverse engineer mobile apps to deploy a rogue API.
- Microservices - Microservices has provided an interesting motivation for deploying APIs–one that potentially can provide small, very useful and focused API deployments.
- Containers - One of the evolutions in compute that has helped drive the microservices conversation is the containerization of everything, something that compliments the world of APis very well.
- Serverless - Augmenting the microservices and container conversation, serverless is motivating many to think differently about how APIs are being deployed.
- Real Time - Thinking briefly about real time approaches to APIs, something I will be expanding on in future releases, and thinking more about HTTP/2 and evented approaches to API deployment.
- Devices - Considering how APis are beign deployed on device, when it comes to Internet of Things, industrial deployments, as well as even at the network level.
- Marketplaces - Thinking about the role API marketplaces like Mashape (now RapidAPI) play in the decision to deploy APIs, and how other cloud providers like AWS, Google, and Azure will play in this discussion.
- Webhooks - Thinking of API deployment as a two way street. Adding webhooks into the discussion and making sure we are thinking about how webhooks can alleviate the load on APIs, and push data and content to external locations.
- Orchestration - Considering the impact of continous integration and deployment on API deploy specifically, and looking at it through the lens of the API lifecycle.
I feel like API deployment is still all over the place. The mandate for API management was much better articulated by API service providers like Mashery, 3Scale, and Apigee. Nobody has taken the lead when it came to API deployment. Service providers like DreamFactory and Restlet have kicked ass when it comes to not just API management, but making sure API deployment was also part of the puzzle. Newer API service providers like Tyk are also pusing the envelope, but I still don’t have the number of API deployment providers I’d like, when it comes to referring my readers. It isn’t a coincidence that DreamFactory, Restlet, and Tyk are API Evangelist partners, it is because they have the services I want to be able to recommend to my readers.
This is the first time I have felt like my API deployment research has been in any sort of focus. I carved this layer of my research of my API management research some years ago, but I really couldn’t articulate it very well beyond just open source frameworks, and the emerging cloud service providers. After I publish this edition of my API deployment guide I’m going to spend some time in the 17 areas of my research listed above. All these areas are heavily focused on API deployment, but I also think they are all worth looking at individually, so that I can better understand where they also intersect with other areas like management, testing, monitoring, security, and other stops along the API lifecycle.
I am coming across more API providers who have carved off specific "skills" derived from their API, and offering up as part of the latest push to acquire new users on Slack or Facebook. Services like Github, Heroku, and Runscope that API providers and developers are putting to work increasingly have bots they employ, extending their API driven solutions to Slack and Facebook.
Alongside having an application gallery, and having an iPaaS solution showcase, maybe it's time to start having a dedicated page to showcase the bot solutions that are built on your API. Of course, these would start with your own bot solutions, but like application galleries, you could have bots that were built within your community as well.
I'm not going to add a dedicated bot showcase page until I've seen at least a handful in the wild, but I like documenting these things as I think of them. It gives me some dates to better understand at which point did certain things in the API universe begin expanding (or not). Also if you are doing a lot of bot development around your API, or maybe your community is, it might be the little nudge you need to be one of the first APIs out there with a dedicated bot showcase page.
I'm going to keep beating the patent API drumbeat, until I bring more awareness to the topic, and shine a light on what is going on. While I will still be my usual self and call out the worst behavior in the space, I am also going to try and be a little more friendlier around my views, and try and help bring more options to the table. This is a serious problem, nobody is talking about, and one that has many dimensions and nuances--if you want my raw stance on API patents, you can read it here.
One area I wanted to try and cover, in response to my friends trying to convince me their aren't bad people, in having patents. I know you aren't, and it isn't my goal to make you look bad in this, it is only to shine light on the entire process, how broken it is, and call out the worst offenders. If you truly believe in patents, protecting the work you've done, and that your intentions are good, share your patent portfolio with the world, and showcase it like you do the other aspects of the work you do. You will craft a press release about everything else you do, do the same for your patents.
I do not think patents are bad. I think ill-conceived patent ideas, that haven't been properly vetted by the under resourced USPTO, that are used in back door dealings as leverage, and litigated in a court of law are bad. I'll take your word that your patents are good, and you aren't operating in any of these areas, if you are public, transparent, and openly proud of them, as you state in private conversations.
Part of the purpose of my research is to encourage good behavior in the sector, by highlight the common building blocks of the space. I think I will add a patent portfolio building to my research. While I have ZERO examples to highlight, I encourage API companies to do this, and would love to highlight in a positive way, any company that is straight up enough to showcase their patents. If you are proud of your API patents, and do not have bad intentions in having them, please publish your portfolio, show case them as you would anything else you are doing--help bring API patents out of the shadows.
Time Tracking API platform Harvest has embraced Github as part of their API ecosystem. I'm always on the hunt for examples of API providers using Github, so I figured I'd showcase Harvest's creative use of the social coding platform.
Starting with their documentation, the Harvest team has moved the API documentation to a Github repository, allowing developers to "watch" the API, get updates when changes are made, asks questions or even contribute to the API docs by submitting a pull request.
Harvest is also using the wiki portion of their Github repo for a developer application gallery they are calling Community Creations and Hacks, where they showcase innovative uses of the Harvest API--currently displaying 20 integrations by Harvest users.
I'm currently tracking on 11 separate uses of Github for API management, and always on the hunt for new ways to use Github to support API ecosystems. Nice move Harvest!
I stumbled across the Twitter Counter API in my monitoring for the API Stack this morning. The Twitter Counter API allows you to retrieve key metrics on any Twitter account like username, url and avatar. All data you can get via the Twitter API, but with Twitter Counter API you get additional information like account growth statistics and ranking, that Twitter doesn't provide at all.
I find it fascinating that someone can build an API to augment an existing API, which is why I keep talking about it, I guess :) We are seeing a more standardized version of this with API aggregation providers like Singly and Adigami, where they not only aggregate APIs from a variety of sources, they also build entirely new APIs based the added value that is created after they are brought together.
Thinking about if further, it would be cool if you could submit your API to be listed in your parent API providers API area. Think of APIhub and Mashape, but every API area would have its own 3rd API marketplace. API providers often allow 3rd party developers to submit code libraries and samples to be listed as resources, as well as applications for listing in an application showcase. So it makes sense to potetially allow for your developers to submit APIs for validation and publishing into a designated area.
Poster boy for how to properly run your API ecosystem properly, Twilio, recently updated their DOer Gallery to highlight developers in the Twilio ecosystem that build cool stuff on the popular voice and SMS API.
Twilio has the best record I’ve seen of any API, when it comes to showcasing and being loved by their developer community, and I'm sure the DOer Gallery plays an important role in that.
The Twilio DOer Gallery has the following features:
- Personal Details
- Short Bio
- Other Profiles
Devloper Galleries like Twilios might not be for every API platform. But if you have a passionate base of developers you might want to consider giving them their own profile and a gallery where they can not just discover and interact with each other, it can let other companies find potential developers to execute projects via your API.
A Developer Gallery can be a great way to give your API developers some love and attention. Twilio even features developers from their DOer Gallery on their blog in a "DOer of the Month".
Would showcasing your “API DOers” benefit your API community?
I pushed an update for the CityGrid Wordpress Directory Plugin earlier today. I finished porting over the changes to the CityGrid Wordpress Search Plugin as well.
Both my Wordpress plugins had a pretty basic look and feel, I wanted them to look better. So using the HTML and CSS template I borrowed from CityGrid I added it to both of the CityGrid Wordpress Plugins to make them look sharper.
Next I’m going to add another template or two, then start extending the functionality of each template, allowing Wordpress site owners to have as much flexibility when deploying business search or directories using CityGrid Places API.
LinkedIn has just open-sourced the technology behind IndexTank under the Apache 2.0 License.
The IndexTank technology has three separate toolsets:
- IndexEngine - A real-time fulltext search-and-indexing system designed to separate relevance signals from document text
- API - A RESTful interface that handles authentication, validation, and communication with the IndexEngine(s)
- Nebulizer - Multitenant framework to host and manage an unlimited number of indexes running over a layer of Infrastructure-as-a-Service
Factual has launch a new application gallery to showcase the diverse number of applications built using data provided by Factual.
You can search for apps, browse by category, and filter by open source, paid or free apps. Looks like there are about 18 apps in the directory currently ranging from augmented reality to daily deals.
The Factual App Gallery isn’t a particularly unique launch, we are seeing app showcases popup within many APIs, but it shows that Factual is gaining steam, and I think it shows the appetite for building apps around datasets is growing.
- 1st prize - Makerbot Thing-O-Matic 3D Printer
- 2nd prize - 1TB USB hard drive enclosed in a vintage nintendo game (Zelda, Metroid, etc)
- 3rd prize - Set of BuckyBalls magnetic building spheres
- NewIn - This application shows new members joining LinkedIn from around the world.
- ChromeIn - Integrate LinkedIn directly into Google Chrome. Easy access to your LinkedIn updates, anytime.
- Signal - Signal is aimed at making it easy for all professionals to glean the most relevant insights from the never-ending stream of status updates and news. An API Labs is a great way to showcase experimental and innovative projects that utilize your API.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.