Using Languages to Program Across Clouds – InApps 2022

Seattle startup Pulumi has unveiled its platform designed to eliminate the complexities of cloud software development and the hassles of managing all those YAML files.

Co-founders Joe Duffy and Eric Rudder hail from Microsoft, while Chief Technology Officer Luke Hoban comes from Amazon.

In a blog post, Duffy, who led software development teams during 12 years at Microsoft and was part of the open-sourcing of the .NET developer environment, highlights the pain of cloud software development, especially for container and serverless workloads:

“For every serverless function, I had dozens of lines of JSON or YAML configuration. To connect that to an API endpoint, I needed to learn obscure concepts and perform copy-and-paste grunge work. Docker was great when I was running a little cluster on my machine, but running it in production required manually managing etcd clusters, setting up networks and iptable routes, and a whole host of things that were leagues away from my application domain. Kubernetes at least let me do this once and reuse it across clouds, but felt alien and distracting.

“I thought I was a reasonably experienced engineer, having worked 20 years in this industry, but trying to get my code into the cloud made me feel dumb. And frustrated!”

Building on the idea of infrastructure as code, Pulumi offers one consistent approach involving containers, serverless functions, APIs, and infrastructure, and using programming languages DevOps teams already know, such as JavaScript, TypeScript, Python, and Go. Other languages including Java and C# are in the works.

Kubernetes co-creator Brendan Burns, now working on Microsoft Azure, presented a similar idea at KubeCon last December with his Metaparticle project.

Cloud Object Model

Explained Rudder: “We wanted to get away from infrastructure configuration, the siloed approach to making containers and serverless easy, and the vendor lock-in.”

Everyone on the team saw how contorted the process was becoming for building and managing YAML configuration files.

“YAML has its own abstractions and it’s a little bit crazy to deploy your service and about 10 minutes into your deployment, it fails because you have an extra space in a YAML file. That was a shocking thing to us. People tried to extend YAML files for semantics, but then you had crazy escape mechanisms to sort of put code in YAML files, then people where copying code between their source, then they had a macro thing that would jam it into a config file – it just wasn’t a good fit,” Rudder said.

“These days with all these YAML tools, you crack open an IDE (integrated development environment) and you’re not getting help from the IDE because it’s just this opaque blob of YAML. You don’t get any refactoring support, it doesn’t give you interactive errors when you make a mistake, you don’t have testing support. You don’t have all those things as developers we’ve come to expect in our tools,” he said.

“Real programming languages solve many of the problems we wanted to solve, and we wanted to embrace the trend of developers and DevOps guys moving closer together,” said Rudder, who was once considered a possible successor to Bill Gates.

Pulumi uses a cloud object model, coupled with a cloud-neutral evaluation runtime that understands programs written in any language and the cloud resources required to run them, then plans and manages those resources in a robust way.

Pulumi is not a PaaS — your programs always run directly on your chosen cloud with all that cloud’s capabilities still fully available to you, Duffy explained. Its Hawaiian theme includes embracing what it calls “CoLaDa architecture” —applications comprised of containers, lambda functions and data services.

Power of Languages

The company, founded last year, also announced a $5 million series seed investment from Madrona Venture Group and Tola Capital.

Learning Machine was one of its first customers, shaving 25,000 lines of code to just 500.  One of its co-founders, Dan Hughes, now president and chief operating officer explained its experience this way:

“We needed to bring our enterprise blockchain toolset into the data centers of major corporate and government clients, They had specific on-prem or cloud vendor requirements that would have meant that we had to rewrite thousands of lines of code in each new environment we supported. This would have made our time to market and total lines of code expand significantly.”

After meeting with the Pulumi team, it decided to try it.

“This decision made all the difference for us. During the one-time port to the Pulumi cloud programming model, we were able to retire 25,000 lines of code in exchange for a few hundred, and we eliminated our DevOps single point of failure. In a month our solution was able to be deployed exactly as our customers expected. Pulumi unified our practices across disparate platforms and made DevOps a first-class feature that the whole team is now a part of.”

There are two parts to Pulumi:

• Pulumi core is the tools and programming model, languages and frameworks. That is open source and available on GitHub.
• Pulumi service. A SaaS product that’s available as public preview. The team describes it as “What git is to GitHub.”

“It’s really about getting your Pulumi code into the cloud —  connecting to wherever your source code is and getting into Azure, Google Cloud or wherever it’s going. The two are designed to work together. For developers, it will just work, be convenient and they won’t necessarily perceive that there’s a boundary between the two,” Rudder said.

Pulumi keeps a record of who updated what, when and why, with a unified log of multiple services. It links back to the source code. And it enables users to test changes on a copy of the infrastructure.

While you can write programs specifically for AWS, Azure, Kubernetes and so on, you also can program at higher levels and use your package manager of choice.

“One of things we found is that we can encapsulate best practices in packages that we can publish to share with your own team, with a community…you can leverage other people’s contributions. This is a game-changer for the way people manage their software,” Rudder said.

A company can package up its security best practices as well.

Pulumi has created packages for AWS, Azure, and Kubernetes, but also provides a  cloud package is geared toward higher-level programming that can run in multiple clouds. You can use it, for instance, to create Docker containers, a feature that integrates with continuous integration systems like Travis or Jenkins.

“It’s a piece of computation that just runs a Docker image. Most people trying to use Docker in production have to manage crazy build pipelines and somehow synchronize that with their application deployment. This is one unified way of doing it where you just have your Docker file, you reference it from within your code, and Pulumi works its magic. It builds that, synchronizes it, makes sure it gets deployed along with your code. You don’t have to think about it.”

Rudder stressed interactivity as a premier feature of the technology.

“Usually you have another window open for documentation, and you’re trying to figure out if it’s right or not right. No tool will tell you if it’s right. We can build on that. We have plans for better interactive analysis — static analysis to find bugs, linters, we can find security issues and possibly cost issues. We’re betting on languages is getting us all these [capabilities].”

Microsoft is a sponsor of InApps.

Feature Image: “As tasty as colorful” by Carlos Madrigal, licensed under CC BY-SA 2.0.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker, Real.