One Cloud API to Rule Them All – InApps Technology 2022

Abstractions have clearly been on the mind of the top engineering executives of cloud giant Amazon Web Services, to judge from the company’s AWS Re:Invent conference held earlier this month in Las Vegas. In his signature keynote, Amazon Chief Technology Officer Werner Vogels had shared news of the AWS Cloud Control API, which unified the company’s sets of services under a single API format, or “abstraction” for developer ease.

The company had also abstracted away from the messy development processes with the newly released Amplify Studio, as well as with SageMaker Canvas, two tools to ease the cognitive overhead of building web applications and machine learning models, respectively.

Later that day, after Vogels’ keynote, InApps Technology sat down with Ken Exner, who is AWS’ head of developer tools, a title that barely conveys the wide range of AWS development projects under his purview, including the software development kits that help developers build on AWS, the command-line interface tools, mobile and frontend services like Amplify Studio, AppSync, and Device Farm.

Also under his domain are the infrastructures-as-code tools such as CloudFormation, the Cloud Development Kit (CDK) and the Cloud Control API;  IDE plug-ins like CloudNine; the application lifecycle management tools like the code suite, CodeDeploy and CodeBuild; as well as all the company’s investments in open source languages and runtimes, such as those around Java and Rust. Whew!

Later that day, Exner himself hosted a delightfully entertaining developer talk, along with AWS DevOps Product Marketing Head Emily Freeman, about abstractions, where he unveiled a next-generation open source Java garbage collector. In our interview, we discussed abstractions, Amplify Studio, the meaning of “low code,” the CDK and the Cloud Control API, the newly-released Rust SDK, and back around to abstractions a bit more, with a discussion of leaky abstractions.

InApps Technology: What are some takeaways from AWS Re:Invent 2021?

Exner: Sure, as you saw in Werner’s keynote, there were a couple of things that were captured there that are part of a bigger trend. One is abstractions. One of the things that we’re investing in now, and we’ll continue to invest in, is layering abstractions on top of AWS. AWS built out this broad portfolio of all these services. And now one of the things that we’re focused on is building abstractions that make it easier to consume these services.

Things like Amplify and Amplify Studio, are essentially a set of abstractions on top of AWS services. So if you’re building a website, you don’t have to figure out how to put together DynamoDB and Cognito, and s3 and Lambda. We do that for you.

So Amplify Studio is a frontend for the Amplify services?

Traditional Amplify is a set of tools for helping you build the backend to your web and mobile applications. It focuses on the business logic and backend. So if my website or web app needed to have a login, and an identity pool, it helps set that up. If it needs to have some data tables, it will help you set that up.

What it did not do is help you build the frontend. So, Amplify Studio connects it. So it allows you to build the frontend, connect that to a backend, and actually build the backend.

Would you call that a “low-code” solution?

It’s less code, not low code. I think the challenge with the term ‘low code’ and ‘no code’ is oftentimes what people are referring to is something that tries to hide code from you and tries to tries to just allow you to visually or graphically to do something. It tries to not expose you to code.

Amplify is not that. This is for developers. A lot of low-code and no-code products are targeted more at business analysts or non-developers. And there’s an important market for that. We want to democratize development and expose programming to non-developers.

Amplify studio is for developers. It is just making them much more productive. It is very specifically for building web and mobile apps.

Do you have to run Amplify on AWS?

Well, theoretically, you could host it somewhere else. But a lot of this is built up in layers over foundational AWS services. There’s the core AWS services underneath like Lambda and s3, Cognito, and DynamoDB. And we build libraries on top of that to provide a simplified experience based on what you’re trying to do in building a web app.

And then on top of that, there are tools like CLI or Amplify Studio, the UI layer. So it allows people to engage with a set of tools at any level. And it has extensibility. So you could actually integrate with the CDK. Now, so if Amplify doesn’t provide something for you, you can use CDK to extend that to anything that the CDK supports, which includes things that are not in AWS. So, yes, you could theoretically use it for non-AWS things.

Could you explain the CDK?

I love the CDK. So CloudFormation has been around 10 years. CloudFormation allows you to declaratively describe your infrastructure in JSON or YAML. And then provision that infrastructure.

What CDK does is a couple of things on top of that. One is it allows you to use infrastructure-as-code in whatever language you want. So you can use Java or JavaScript or GO, or C# to imperatively describe your infrastructure, which then compiles down to CloudFormation.

“The problem with all these expressive, verb-based APIs, or operations-oriented APIs, is that there’s a lot to learn.”

By allowing you to use the language of your choice, we make it more familiar to developers. They can use the things that they’re used to. They can have an object-oriented design. They can have inheritance, they can have for and if statements, all the things that you normally do in a language, but it still compiles down to CloudFormation.

The other thing that CDK does, is it has an abstraction model called constructs. And there are different layers of constructs. These are essentially layered-up abstractions.

So a ‘layer 1,’ or L1 construct, is simply a representation of a resource. That resource can be an AWS resource or non-AWS resource.

There’s an L2 construct, layer two, which is an opinionated implementation of an L1. Like if you’re setting up a virtual private cloud (VPC), there are literally dozens of options for how to set up one. Here’s probably what you want. So it’s an opinionated simple annotation, based on best practices.

And then there’s layer three, which is an architectural pattern. So here’s a website with a VPC and everything else. You describe that as a layer three construct. Or a Pub-Sub implementation, which strings together SMS and SQS.

So what we’ve done here is essentially create a composition model for how you can put together an application using these constructs. You can use low-level resources, opinionated resources, or these patterns. And then do all of this in your language of choice, which compiles down to CloudFormation.

The entire framework is quite a powerful thing. We use it pervasively across Amazon. It is how we do Amazon.

And I think the community is getting pretty excited about it. The Terraform [community] just created a fork of it and developed a CDK for Terraform. There’s now a CDK for Kubernetes.

It’s moving up the stack a little bit and in making sure that we’re bringing higher-level interfaces, interfaces that are more targeted to particular use cases. In the case of Amplify, it’s a web app. We’re going to give you APIs are specific to that, so it’s easy to create a leaderboard or whatever, because those things are in the tool. You don’t have to work with low-level resources.

Can you talk a bit more about the Cloud Control API? 

So most of the AWS APIs use a remote procedure call (RPC). It is our operations-oriented interface, which are very expressive. They’re self-describing: Launch ‘Instance,’ and guess what it does? It launches an instance. Attach volume? It attaches a volume.

So they’re very expressive. The problem with all these expressive, verb-based APIs, or operations-oriented APIs, is that there’s a lot to learn.

There’s another, resource-oriented, style to design APIs. You have an object, and then there are a consistent set of verbs. You can create, read, update and delete [CRUD].

s3 actually, just to be technically honest, is more of a resource-oriented API. But most of us have APIs that tend to be more operations-oriented.

Cloud Control API was developed by my team to create a consistent API for all of AWS, based on a resource orientation. So you can use it for any resource in AWS. And in fact, any resource in the CloudFormation registry. You can have a consistent API to create read, update, delete, list. It’s a model-based interface.

This was based on the work that we had done in CloudFormation. One of the things we did with CloudFormation is create that mapping. So that we could have this CRUD-based interface, rather than an operations-oriented interface, for the context of infrastructure-as-code.

“The entire history of software is building abstractions.”

And what we realized was there was an opportunity here to create a consistent control plane for all of AWS. And our partners were super-excited about this, HashiCorp, and Pulumi, because they were trying to constantly keep up with AWS. So we gave them a consistent API that they’re now using for all their resource providers for within Terraform and Pulumi. They can stay up to date with us, because it’s one API. It’s consistent.

It also works with non-AWS resources. So anything on any CloudFormation registry can get exposed through this. It’s one consistent API for the cloud, to describe any resource.

And I would imagine that helps developers in the fact that like UNIX, you don’t have to learn each API  you can kind of guess at it…

Yeah, a lot of developers will like this, because it’s consistent. I will say with some developers, there’s a preference. Some people prefer the style of RPC style. But they don’t have to choose. They get both here. They get the existing service APIs. And they now have this consistent, quality-based interface.

Let’s talk a bit about the new Rust SDK. AWS is seeing developers gravitate towards the Rust programming language?

Quite a bit, internally and externally. So a few years ago, we started doing a lot more Rust development inside of Amazon. And today it’s quite popular. So a lot of the backend of EC2 to s3 Cloud front is written in rust.

So we started investing in Rust a couple of years ago in my team and hired some of the contributors and people who were involved in the community. We helped put Rust into a foundation, to make sure that it was gonna be set for growth and a positive future.

So we’re excited about the growth and what’s happening there, and externally we have we’ve seen customers increase their usage as well. So the Rust SDK was produced to support us as well as our customers.

With the release of these new SDKs, we’re continuing to focus on trying to meet developers where they are, making sure that we are always supporting them, and whatever tools, whatever languages, whatever technologies that they want to work in. It’s important to us that our customers understand that we will meet them where they are, we will help them create a great experience on AWS within whatever technology and tools that they use.

I’m looking forward to your talk. What’s your definition of leaky abstractions?

Joel Spolsky coined that term. Leaky abstractions are when lower-level implementations bleed through into higher abstractions.

So an example I use is AWS Elastic Beanstalk. It helps you build web apps on AWS. To do that, it creates some resources in your account, so it can access EC2. So that implementation leaks through to the high-level account, for security reasons.

What abstraction fundamentally does is simplify something. In our talk, we compare this to art. If you look at modernist art, you can see it reduces something to its essential elements.

If the underlying details bleed through, it’s negating that promise. It’s no longer simple. The details are something you have to be exposed to.

So how do you make sure that the abstraction is pure to itself pure to its intent, which is, you don’t have to know about the underlying details.

Abstractions are really about how do we as an industry make developers more and more productive. We do this by building abstractions on top of abstractions. The entire history of software is building abstractions, so you don’t have to think about resource management on your device the way you used to. You don’t have to manage memory manually the way developers used to.

You start with an operating system, which is an abstraction of the hardware. You don’t have to think about the hardware anymore because the operating system abstracts it. Programming languages are abstractions. The hypervisor is an abstraction of an operating system. EC2 is an abstraction of a hypervisor. Lambda is an abstraction of EC2. An object is an abstraction, an object-oriented function is an abstraction in functional programming. All these things are different abstractions.

So abstractions are about productivity, about eliminating things that used to take up time.