This post explains how to build an effective API. Once it’s ready, you need to make it accessible to your users. NGINX and NGINX Plus are frequently used as an API gateway. For more information, check out these resources:
- The announcement for NGINX Plus R10, which includes support for the use of JSON Web Tokens (JWTs, or “jots”) for authentication
- A description of how to implement API authentication with JWTs from our own Liam Crilly
- A video describing how Adobe powers its API gateway with NGINX
This post is adapted from a presentation delivered at nginx.conf 2016 by Mike Stowe of MuleSoft. This first part focuses on spec‑driven API development. The second part discusses best practices. You can view a recording of the complete presentation on YouTube.
| 0:00 | Introduction |
| 1:52 | APIs are Changing the World |
| 2:32 | APIs Are Connecting Everything |
| 3:36 | APIs Should Be Persistent |
| 4:01 | 5 Steps to Building an API that Lasts |
| 4:38 | Think Long-Term |
| 6:03 | Your API Is a Contract |
| 6:39 | Think Things Through |
| 7:36 | Who Will Be Using Your API? |
| 8:40 | What’s the Purpose of Your API? |
| 9:41 | List What Your Users Need to Be Able to Do |
| 10:53 | What Type of API Are You Building? |
| 11:43 | SOAP vs. RPC vs. REST |
| 12:34 | Do You Understand REST Constraints? |
| 13:42 | Most APIs Are Not RESTful |
| 14:55 | Build for The Long Term |
| 15:20 | Versioning Is a Necessary Evil |
| 17:51 | When You Need to Version |
| 18:35 | When Not to Version |
| 19:54 | Versioning Doesn’t Excuse Poor Design |
| 21:12 | Use Spec-Driven Development |
| 22:00 | Hybrid Approach |
| 23:11 | The Agile Design Cycle |
| 24:36 | Some of Today’s API Specs |
| 25:05 | RAML Benefits |
| 27:25 | What Does RAML Look Like? |
| 28:33 | The Spec Is the Blueprint |
| Part 2 |
0:00 Introduction
Mike Stowe: Today we’ll be talking about building your API for longevity. Over the last ten years, I’ve had the chance to both build and use APIs, and I’ve discovered a couple different things.
The first is that building the API is actually quite easy. It’s really not that hard to write the code, or use a framework like Grape, or Rails API, or Jersey, or Apigility, or the numerous other ones out there. The second thing I realized is that people aren’t really good at making long‑lasting APIs. Let me ask this: how many people here have used an API that has broken? So we all share the same pain point.
With this talk the focus is on how we can make an API that’s going to last, why should it last, what are the problems with versioning, and how can we make sure that what we build is something our users are going to love, so that it’s not going to cost us more money for development or support in the long run.
We’re not going to dive into a lot of code. Instead, we’re going to focus on principles, best practices, and how we can create a more efficient process.
1:52 APIs are Changing the World
I think it’s fair to say APIs are changing the world. If you’ve ever used a smartphone, or even if you’ve ever driven a car, you’ve probably used an API.Today there are over 15,000 public APIs listed on ProgrammableWeb, 15,000 public APIs connected in all sorts of devices. But that’s really just the tip of the iceberg.
If you think about it, there are hundreds of thousands, if not millions, of APIs connecting mobile apps, as well as internal APIs for business use, connecting departments.
2:32 APIs Are Connecting Everything
Today, APIs are connecting a lot of different things. Obviously they’re connecting our mobile apps, they’re connecting enterprise architectures, but they’re also connecting phones. They’re connecting watches such as the Apple Watch and the Android watches. Glasses, cars, refrigerators, thermostats, in‑home robots, and a lot more.
Think about that for a second. Think about all the things that APIs are connecting and how important that is.
Imagine for a second you’re at home, and you have your home security system and the worst thing possible happens, someone breaks into your house. The alarm starts going off but then says “Update required” – it’s unable to notify the police because the API is out of date. That’s not a good thing.
If we look at microservices, the last thing we want to do is deploy 700 microservices out there that are dependent on an API, and then find out that API broke – that we have to go back and update the 700 microservices that may be composed upon it.
3:36 APIs Should Be Persistent
In order for APIs to work, in order for the IoT to work, in order for microservices to work, they require that APIs be persistent.
Not only that, but versioning or making changes to your API is really, really expensive for everyone involved.
It’s not just you that has to go on and say, “We’re going to rebuild our API or refactor our code”, but also all your users, all your company’s departments that rely on that API.
4:01 5 Steps to Building an API that Lasts
Thankfully with five simple steps, you can build an API that’s designed to last.
The first step is to go in with a long‑term mindset. This sounds like common sense, but a lot of people go in and say, “We’re going to build this API, we’re going to get it out there, we’ll get feedback, and then we’ll version it.” That never works.
Understand what you’re building. Again, it seems like common sense, but most APIs that get developed are developed on the fly, without true understanding of what they’re actually building.
Utilize something called spec‑driven development, which we’ll talk about later.
Incorporate best practices.
And as you do this, repeat Steps 1 through 4.
4:38 Think Long‑Term
Your API is a contract. We forget that unlike an agile web application – where we can say, “We’re going to launch it, and then we can make changes as we go” – you can’t make changes in your API without affecting your users, without causing their systems to go down. Versioning is also not a solution.
My favorite reason to think long‑term is that you’re probably really bad at long‑term design. You may think you’re good at design, but let me ask you this: have you ever looked at your code a year later and said, “This is the most amazing thing ever, I wouldn’t have changed anything”?
We’re really good at solving the problem in the short term, but not for the long term. Much like with security, with API design you can pay a little bit more now or a lot more later. So you need to think things through.
Mindset is everything. You wouldn’t want a team walking to a basketball game going, “We’re going to lose, we’re going to lose.” Because chances are, they’re not going to win. Same thing with API design.
6:03 Your API Is a Contract
Your API is a contract. We often forget that when we make a change [that breaks old functionality], we are taking away the ability of our consumers to make money. When we change our API, our customers or other departments in our company have to say, “Wait, I have to stop everything I’m doing and go back and fix things, because you broke it.”
That means they’re not adding new features, they’re not making more money, and they’re certainly not going to pay you more money, because they’re trying to fix things instead getting new customers.
6:39 Think Things Through
You have to think through every aspect of your API before building it. These aspects may seem like basic questions and common sense, but they get missed quite a bit.
Who is your API for? What type of API are you building?
How are you going to maintain your API? A lot of companies out there say, “Look, we’re going to build an API, push it out there, and that’s it.” I promise you, that’s not it. You’ll have to maintain it, they’ll be security patches.
How are you going to document your API? Are you going to have a full documentation team? Are you going to use the Swagger console? Are you using RAML?
How are you going to let users interact with your API? How are you going to handle little things like authentication, provisioning, throttling, security risks, DDoS attacks, developers using infinite loops by accident? How are you going manage support?
These don’t have to be really tough discussions that you have to have. Your approach to support might be to have a dedicated API support team, or [have] engineers will provide support, or have a community forum, or [you might even say] “We’re really not that invested in APIs, so good luck; if they need support, they can figure it out.”
7:36 Who Will Be Using Your API?
Who are your end users? Is it going to be current customers? Is it going to be business partners? Third‑party services or developers?
This is important to answer because before you can build, you have to understand who they are and what they need. What actions do they need access to? This means involving your users from Day One.
When I started off as a web developer, I was a rookie, and I got this photography client. She told me, “I’m looking for a beautiful photography website, I want this type of imagery, this type of font.” She gave me the specs, and I went out there, and I built the most beautiful photography website on the planet, in my opinion.
I sent it to her, and she looked at it and said, “What is this? You’re using dark colors. I do wedding photography. I want light colors. You used this image, I want that image. You used this font, I want that font.”
I poured my heart into this site, but it turned out she hated it because I hadn’t involved her in the design process. I didn’t ask her what she was looking for, and we do the same thing with APIs.
8:40 What’s the Purpose of Your API?
List out why you’re making the API.
At MuleSoft, we deal with a lot of enterprise customers, and they say, “Hey, we want an API.” We reply, “Well, that’s great, but what’s your API going to do?” They say, “We’re going to share our services and data with the world.” So we ask them, “Okay, but what data do they want? What services are they going to have access to? How are they going to interact with your existing services or other APIs? How do they interact with other APIs or services from third parties that they might need to use with your service? What actions do they need to be able to handle?”
One of the traps we run into is to say, “We’re going to build the REST API, we’re going to do CRUD, we’re going to do the HTTP methods.” We start thinking POST, PUT, PATCH, DELETE right away.
Take a step back and say, “They [our customers] need to be able to add a user, edit a user, retrieve a password possibly, send a message.”
This next point is for the engineers in the room, because let’s be honest, we’re horrible at this: do only what is necessary. Don’t get fancy with your API. You’re creating a foundation and like any foundation, you’ll be able to build onto it, and build onto it more.
9:41 List What Your Users Need to Be Able to Do
List out what actions your users need to be able to do.
This is a really simple way to organize an API. We simply say, “We’re going to have users, and then you’ll be able to create a user, edit a user, etc. We’re going to have a messaging system; here’s our messages, and the actions that we need. We’re going to have products, so we need to be able to review a product, add a product to the cart. We’re going to have a cart, so we need to build a checkout.”
As we go through this process, a couple of things start happening. The first thing you notice is that we’ve already defined our resources without even trying. We have all the stuff for users, which is the Users resource. All this stuff for messages [is] the Messages resource.
The other thing you’ll notice is that we have duplicates on there. We have Message user and Send a message. Well, it might not make sense to actually have a resource specifically under Users for sending a message. That seems redundant.
What we have done is had a chance to look at the question, “Where does it belong?” We can say, “Send a message and Message user belong under Messages,” but also develop the relationship link between those two. So rather than going back and saying, “Let’s create a hypermedia map and link all these things together” after the fact , we’ve already created a hypermedia map. We already know that when we pull up a user, we need the ability to send them a message, and that’s going to be a link on there.
10:53 What Type of API Are You Building?
What type of API are you building? Are you going to build a REST API, a partial REST API, SOAP, RPC?
Let me ask you this: what’s the right type of API to build? The answer is: it depends.
A lot of people would say “REST!” Well if you’re a large enterprise, and all your customers are legacy customers, and they aren’t ready for REST API but they need SOAP because they have the SOAP libraries, it probably makes sense to use SOAP. That’s why Salesforce for the longest time maintained both a SOAP API and a REST API, because that’s what their clients demanded.
Why are you building that API in that format? Do you understand the pros and cons of the format that you’re choosing? What does it mean for your development time? What does it mean for usability for your users? And what does it mean for longevity? How is it going to affect you long‑term?
11:43 SOAP vs. RPC vs. REST
Understand that each of these API types has pros and cons.
For SOAP, we could say, “Oh SOAP, it’s archaic. It’s this huge SOAP package that needs SOAP libraries, and it’s only XML.” But SOAP has WSDL, which some people prefer, and it can be stateful or stateless, which is kind of nice.
RPC, on the other hand, is really, really easy to use. It returns back one format – JSON‑RPC or XML‑RPC – but there’s really no onramp to get to it. I call this URL, I get this back, and do a GET or POST and that’s it.
Finally there’s REST, which has all these great benefits, but has some caveats with it as well. It’s also a little bit more difficult for developers to use, especially when you start incorporating things like hypermedia, because it’s always easier to hardcode a link than it is to actually use it dynamically.
12:34 Do You Understand REST Constraints?
Let’s say you’re going to build a REST API because you’re aiming for for longevity, flexibility, and agility. Do you really understand what REST entails?
Do you understand that the client and server have to be able to separately evolve, and that changes you make to your mobile device shouldn’t affect the server at all, or that the client shouldn’t affect the server?
Do you understand that REST is stateless? You do not store state on the server side. Of course, clients can store state, but you don’t keep a record of the calls they’re making. So if I make two calls in a row here, you don’t remember the first call. You don’t care about the first call. You take each call as it is, on its own.
It’s cacheable, meaning that you not only provide caching on your end, but you tell your users, “Here’s how you cache this page” or “Here’s what you can do to cache this page, and here’s the time limit” or “This isn’t cacheable” or “This is going to be cached across all layers.”
Uniform contract and layered system, which in the context of microservices is just basically having different components in your architecture to keep the API up.
And then the last one is Code on Demand. Code on Demand is an optional constraint, but that’s sending code over the API to your users, such as a servlet or functional programming code.
13:42 Most APIs Are Not RESTful
Most APIs are not RESTful.
Roy Fielding, who created the theory of REST, noticed one day that nobody’s using a hypermedia. And he went on to ask, “Is there a broken manual? Is the documentation wrong? What did I do wrong to make it not clear that if it doesn’t have hypermedia, it’s not REST?”
Does that mean that you have to build a full, completely RESTful API? Absolutely not. You have to build the API that meets your needs, but you need to understand why.
There’s an example of a company that was building an API, and they said, “We’re going to build REST.” They started off with it and they went down and down and down, and pretty soon they said, “Well, we’re going to have them send us HTML and then we’ll send them JSON.” And then it was, “We’re going to do POST but they’re going to send data through the query string.”
They started tweaking things, because they thought, “This’ll make things easier and better for our users,” without understanding what they were building. They actually created more problems, required more iterations, and spent a lot of money trying to build this API.
So again, understand why you’re building one type of API over the other. If you don’t, you’re going to pay a lot more down the road, and end up having an API that’s kind of a mix of all things and isn’t going to work.
14:55 Build for The Long Term
It also means building your API for beyond today. That’s the challenge we run into.
Today we say, “This is the API, these are the needs we have, this is our platform.” That’s great, but what’s your platform going to be like in two years? Three years? Five years?
We’re really, really good at short‑term design. But as Roy Fielding says, we’re awful at long‑term design.
15:20 Versioning Is a Necessary Evil
That brings us to versioning. Versioning is a necessary evil.
In the short term it’s easy to say, “We’re going to version, and it’s going to be great.”
Well, versioning has drawbacks.
First and foremost, versioning creates backward incompatibilities, which means that when you upgrade your API, the apps relying on your API are going to break. And when they break, you’re going to hear complaints.
The second is that there’s multiple services to maintain and multiple systems to support. You may say, “No, no, no – that’s not how it works. We’re going to deprecate this API. We’re not going to support it anymore. We’re going to release a new API.”
Well, that sounds great in theory until there’s a security risk, and you have to patch that security bug. Or you get that million‑dollar customer who says, “You know what? Actually, we’re stuck on the old API. We can’t upgrade yet and we need this feature.” Or another department in your company says, “We need it.”
Before MuleSoft, I had the chance to work as a developer evangelist for Constant Contact, and my entire job there was to get people to upgrade from version 1 to version 2. During the year I was there, I was able to get, I think, four people to upgrade their APIs. Nobody wanted to upgrade their API. They thought, “This one works. Why should we go back and spend money to fix something that’s not broken?”
Another problem is versioning creates confusion among developers.
At Constant Contact, one day I had a call and the gentleman on the line said, “I have a problem: the API is not working.” I asked him, “Okay, what version API are you using?” He says, “Yours.”
Okay, no problem, sometimes people get confused about which version they’re using. So I dug in deeper, “How does it work?” He replied, “Well, I send a POST and I get data back.” I said, “Okay, and are you getting XML or JSON?” His answer: “Yes.”
It turns out that he was taking over for another developer who had just said, “This didn’t work.” He called to figure out why it wasn’t working, but he hadn’t done his homework, he didn’t know what version API we were using. It took about 20 minutes just to figure out what version he was on. It turns out he was on version 2, but the feature he needed was on version 1, which really ticked him off because they had just upgraded from version 1 to version 2.
So developer adoption is nearly impossible; you end up creating confusion and having to support both newer and older APIs.
17:51 When You Need to Version
But there are times when you do need to version.
You do need to version your API when you have backward‑incompatible platform changes. If anyone watches the show Silicon Valley, the company Pied Piper goes from being an algorithm to a media company. In that case, it might make sense to change your API to reflect the new things you’re doing now. Or if you were previously an auto shop and now you’re a bakery, it makes sense to change your API.
The second instance is when your API is no longer extendable. In other words, you messed up. You thought you had the perfect API, [but] you kept changing and extending it, kept changing and improving it, and then you got stuck and now you’re in trouble.
Lastly, if your spec is out of date. Maybe [your customers] now want JSON but you’re still using SOAP with XML, so you need to update your API.
18:35 When Not to Version
You shouldn’t version your API just because you added new endpoints or you added additional data in the response. There’s the argument about strict schemas. Unless you’re doing something that requires strict schemas, I’d avoid saying, “This is the strict schema for a data response.” Be flexible in that.
You shouldn’t version because you changed technologies (you went from Java to Ruby).
Or if you changed your application services. You should be able to change the code. You should be able to change your services. You should be able to change your dependencies without affecting the interface of the API.
This is another challenge we see, by the way, is companies that say, “You can generate an API based on your database. We’ll create the API based on your database names.” That’s actually okay if you create a unique interface – if you go back and you start changing all of those and you decouple it from the database.
The problem is that once you say, “Okay, we use this database. We’re going to move to a CRM.” All those names are strictly aligned to the database; you’re tightly coupled.
Same thing if you’re saying, “Look, we’re going to have a CRM and we’re going to represent a CRM through our systems.” If you’re using a specific API or specific system and your API on top of that is tightly coupled, the second you change that, you’re breaking the interface and that’s what breaks backward compatibility.
19:54 Versioning Doesn’t Excuse Poor Design
Versioning does not excuse poor design. That’s very, very important to point out: versioning is not a solution. It can actually create more problems down the road.
A poorly designed API is going to cost you far, far more in the long run than any benefits it may give you in the short run.
One of the stories I’m not proud of is I had the chance before I joined MuleSoft to work as a contractor, and we built this API. We spent six months writing the code, and we built this perfect API. It followed all the best practices and newest standards; the code was perfect.
Then after six months of work, we released this API, and three weeks later, we realized no one was using it. It wasn’t because of the code – the code, again, was good. It wasn’t because of best practices – we had a great REST‑based API. It was simply because it didn’t meet their needs.
We spent the next three months trying to salvage this API, trying to fix it. But we ended up finally scrapping it and throwing it away. That’s nine to ten months’ worth of work completely thrown away because our API couldn’t meet our customer’s needs. All because we didn’t spend two weeks at the very beginning asking our users, “Does this work for you?” We didn’t involve our users.
21:12 Use Spec‑Driven Development
The way that we can avoid that is by using something called spec‑driven development.
The way spec‑driven development works is the first thing we’re going to do is define our API before we write any code. We’re also going to bring in design patterns and code reuse. We do this with all our applications, so why don’t we do it with our API designs?
We’re going to mock up our API and get user feedback from our users before we write a single line of code. We make the necessary changes. Then we start coding to the specification and not deviating. Our specifications become the source of truth.
What we end up with is a two‑stage agile process. We have an agile design process to develop the contract and then an agile development process where you can use test‑driven development to actually develop the code. The reason to do it this way is, this way nothing goes into the code or into the design that we haven’t tested and we’re not sure about.
22:00 Hybrid Approach
So first we say we’re going to design our API, mock it up, get user feedback, make any changes, test it, make sure it’s perfect, then we’ll go into development.
What happens if you run into an issue in development where you [realize] this was a great idea in theory but it doesn’t work? You simply go back to the design process, fix it in your test, make sure it works, test it with your users, and then move forward.
What you’re doing now is you’re creating an API which not only your users are going to love, which will meet their needs, and which you get to test against real‑world use cases, but also one without design flaws.
What happens if you don’t do that? There’s a certain social media giant’s API where they have a resource that if you want to search it by age, you have to put age in one way in the JSON. If you want to search by location, it’s a completely different way. If you want to search by age and location, it’s a third way altogether. This wasn’t done because of certain dependencies. It was done because they had different developers working on it and they didn’t realize that they have this set up.
23:11 The Agile Design Cycle
What does the agile design cycle look like? Design the API. Get user feedback after you mock it up. Validate: “Does this meet our needs? Does it meet our user’s needs? Is it a good design?” If not, keep working on it until you get the design perfected, and then when you’re ready, move into the coding stage.
The goal here is to let our developers develop fearlessly. We don’t want to have our developers look at the API and say, “We’re building this resource, and have a question for you. How did you set that data? Did you use camel case or snake case? And are using a PUT or a PATCH to handle that? And is that data encased in a data object? How do we separate that?” You don’t want to run into those issues. You don’t want those inconsistencies, because that’s what’s going to cause your API to break.
The problem is, up until now, that’s been pretty darn expensive. We say, “Hey, we’re going to build out this API. We have to do an MVC, go and change the code,” and [find out] it doesn’t work.
The good news is today, there are several new specifications that are going to make this very, very easy. The other nice thing about this is these same specifications handle a lot more for you. So, you’re not just designing your API and then throwing that away, like we do with waterfall [development].
Now you have a spec that can be used to generate your documentation, that can be used to generate your test‑driven development test. You have a spec that can be used to create SDKs and client generators, interactive scenarios, all these different things.
24:36 Some of Today’s API Specs
Some of today’s specs out there:
The first is RAML which is the RESTful API Modeling Language.
The second is I/O Docs which is by Mashery®. Here’s my disclaimer on I/O Docs: I wouldn’t recommend using it simply because TIBCO Mashery has moved toward Swagger, so they’re really not maintaining I/O Docs as much anymore.
You have Swagger, or the OpenAPI Initiative, which is another great and very popular specification out there.
Then you have API Blueprint, which is fantastic when it comes to documentation. It has some great tools out there as well.
25:05 RAML Benefits
I’m a fan of RAML for couple of reasons. With RAML, you can define your API in just a few lines of text. You can see what your API looks like as you go, with the API Designer. You can quickly prototype your API for developers to try out. That means, with a click of a button you can send this API around the world and let your developers try it out in real time.
You can quickly make tweaks and changes when you find design flaws. You can easily document your API. You can let developers try your API online.
Another great feature with RAML is something called the API Notebook. Rather than have your developers say [to your customers], “Hey, here’s our API. What do you think?” you can send [your customers] an API Notebook and, without having to write any code outside of minor JavaScript, they can actually test your API using real‑world use cases to see if it meets their needs.
You can have developers try your API online. You can have developers interact with other APIs through the API Notebook. Then if you want SDKs, client generators, code wrappers, libraries, even your frameworks for building the API, you can generate those through the open source community. If you’re creating SDKs for public consumption, however, I would really lean towards using a professional company like APIMatic.io or REST United just because whether you’re using RAML or Swagger, a lot of the open source client generators are kind of sketchy.
Most importantly – and this is the real reason why I prefer RAML over Swagger – you can use data models and design patterns, which means we’re saying “this is what our API looks like.” It stays consistent.
You can reuse code; you can pull in libraries, which means if you have a user object that’s shared across multiple APIs, you can pull that in a namespace manner. You can create overlays, so you can have one specification that you can then overlay for your dev environment, your QA environment, your stage and production [environments], etc.
In this example we want to have a resource type of collection. For our collections, we’re going to get multiple items at once. Our get has an example description. Rather than having to write all of this again and again, we simply have to say exampleCollection, and that pulls the rest in for us.
27:25 What Does RAML Look Like?
What does RAML look like? This is the other reason why I love RAML, by the way. RAML is very verbose and very complex [not!].
If I want to create a resource and I want to call a resource playlist, how would I define that in the RAML file? /playlists, just like I would on the API.
If I want to add a GET method to playlist, how would I add that? get.
And if I want to have a 200 response, what status code do I use to put in a 200 response? 200.
So, you can see it’s very easy to get started with. The nice thing about RAML is that it doesn’t matter if you’re a developer, a tech writer, or a CEO. You can even let your CEO modify your API documentation; that’s up to you.
I’m not going to go too much deeper into RAML here, but you can find out more at raml.org.
So in summary, you’ll want to use a specification, whether it’s RAML or Swagger, as you build out your API.
28:33 The Spec Is the Blueprint
But remember it’s not a “one and done”.
Once you start designing your API, every time you make a change, you’re going to run through these tests. And the reason for this is, you can design the perfect foundation, but if you start building things incorrectly on top of it, you’re shortening your lifespan.
If you build the foundation for a house, you don’t add ten more rooms to the house without having a blueprint. Otherwise, pretty soon you’re going to have a brick wall that has a window facing it. You don’t want to run into that.
This post is adapted from a presentation delivered at nginx.conf 2016 by Mike Stowe of MuleSoft. This first part focuses on spec‑driven API development. The second part discusses best practices. You can view a recording of the complete presentation on YouTube.