The Exploding Infrastructure Automation Stack and Its Ecosystem – InApps Technology 2022

Matt Baldwin

Matt Baldwin, head architect, is responsible for ProfitBricks DevOps Central community, along with the libraries, SDKs and documentation that are accelerating the use of ProfitBricks through its REST and SOAP APIs. When he isn’t hacking on containers and Ansible, he enjoys collecting obscure Exotica and Latin jazz records.

You’ve heard this a thousand times: software is eating the world. Or should I say “Infrastructure as Code” or “Infrastructure Automation” is taking over the world?

Software is replacing infrastructure, and now containers are allowing us to migrate into a world of applications that are seamlessly integrated across the Internet via web services. Solutions, like Kubernetes, help you keep your application online and in your desired state, while infrastructure providers offer the compute resources for these clusters to scale and perform at increasingly lower prices.

We’re moving into an era of democratized compute; infrastructure has become the application. In this era, we can easily deploy and re-deploy the same footprint across various providers, moving our apps from one to the other. We can choose by price, performance, or some metric in between. It’s only a matter of application architecture and the integrations in between — such as CI/CD solutions, configuration management, and other tools — all enabling the exploding infrastructure automation stack and its ecosystem.

Modern Applications Require Infrastructure Automation

Applications are going global. What used to be a reliance on hardware when delivering solutions for mobile, social and web applications, is now a reliance on software delivered through cloud compute. This can be platform or infrastructure as a service. The line is only going to get greyer as we move into the era of container adoption.

Never before in my 15 years of Internet infrastructure development have I seen so much interest in the backend. And, apparently, nor have the VCs.

I won’t get into why this is happening — sure, cloud compute, containers and DevOps are driving quite a bit of it. And the business cases are here to fund the need: big data, sophisticated modern SaaS applications, highly available everything or applications going global, mobile clients, etc. Business is seeing a strong trend in continuously delivering improvements into IT. This can be CI/CD for the app the company offers, or simply the consumption and integration of various business-line applications through the cloud.

Read on to learn how the stack is evolving and how it fits into the modern development and delivery of applications as workloads become more complex. We’ll start with infrastructure automation and touch on a few categories of software and services (you can find in the nifty little infographic here) that you would use to deliver various types of workloads into cloud computing.

Infrastructure already presents a list of choices. Cloud architects need to choose amongst companies that deliver automation and deployment options for public cloud, hybrid clouds or containers. The Internet has become a landscape of disparate APIs and startups that have been able to leverage them to build the modern application delivery model and modern infrastructure. The following list highlights broad categories of companies helping build this new future of application and operations delivery.

Out of this list, I think some categories are mandatory for a team to be successful.Things like collaboration and issue tracking tools are now essential requirements.

You cannot begin to deliver software into production until you’ve adopted a means to have a conversation around that which you’re developing and a method of testing and tracking issues.

Ideally, at this point, containers are smart to move towards as you begin to build out the stack you want to work with, where the stack is a mixture of the tools and ideas outlined in this article — but they’re not necessary right now.

Collaboration Tools are Table Stakes for a Modern Development Team

The most effective teams leverage and use the most powerful communication tools. Nowadays, development teams have multiple apps in their toolbox that allow them to stay in touch with teammates and colleagues who may reside in different parts of the country or globe. These not only keep them in touch with each other, but sometimes with the community around their application.

All successful applications and platforms are engaged with their community in some meaningful way. In turn, developers are engaged in a conversation about the application.

Currently, the developer collaboration space is being consumed by a handful of companies — all of whom are built-out on cloud infrastructure which services distributed teams, who themselves deploy into cloud infrastructure.

Some companies that fall into this space would include:

• Flowdock
• HipChat
• JIRA Software
• Slack
• Pivotal Tracker
• Trello

Teams are becoming more distributed with developers and engineers living wherever they want and companies, especially startups, being receptive to that requirement. This is evident in the use of various collaboration tools, my favorite of which is Slack. We use Slack internally and are committed users. It’s nice to see alerts dropped into a conversation, posts, sharing of docs, and Slack’s various ways to manage conversations, which all adds up to a great experience.

Collaboration tools have become an important way for teams to communicate about various aspects of their development and operations environments. These tools are becoming more automated, better integrated into the systems the teams are using.

You also, absolutely, need a software ticket tracker of some sort.

You can use Jira, Assembla or GitHub. There are various benefits and pitfalls to each. Regardless of your flavor, they’re required for any modern development team. Why aren’t you using Git yet? It’s also neat to connect one of these to Slack so you get all the alerts coming in — read the commit messages, so on. These can all become part of the conversation now. We use it to organize a global team.

Conversation is the central tenet of the collaboration space. Each of these applications helps teams establish a conversation around what’s being built and delivered.

Microservices are All the Rage, and For Good Reason

Microservices is a software architectural model in which you componentize your application and run each component as its own container across one or many physical or virtual machines. These containers can be built by developers, then shipped into production via continuous delivery. Containers should be fungible. You should be able to run one or many identical containers, depending on how much you need to scale your application.

Companies continue to pour into this space, with the most well known being Docker. It seems if you work within the container ecosystem it wouldn’t be too difficult to fund your startup and team.
There are a ton of opportunities in the container space — from networking to data persistency — and numerous companies are trying to solve these problems.

Here is a list of various companies and projects working with microservices:

• ClusterHQ with their volume management solution, Flocker.
• CoreOS, an operating system purpose-built for containers.
• Docker.
• Kubernetes, by Google, helps you do cluster orchestration and scheduling of containers into the cluster.
• Mesosphere helps your organize your infrastructure.
• Rancher with their container control plane for private container clusters.

Containers will “eat the world” over the next few years. Any company in this list that isn’t working in the container space or addressing it in some manner will be doing so within the next two to three years. For me, it’s simple:

Any tool a developer and engineer choose to adopt without being a forced tool will be a tool that wins.

There are already adopters. Companies like eBay are moving to containers scheduled with Kubernetes. As developers become more savvy and aware of the tools out there, delivery of software will continue to become more automated. Operations teams will begin to focus on managing the underlying infrastructure as an application in itself.

IaaS Platforms, the Essential Foundation of Modern Software

Under everything, even containers, sits the cloud infrastructure component. The whole concept of “cloud” wouldn’t exist if it weren’t for the providers in this category, like Amazon with its AWS. It’s obvious that AWS is more than just cloud compute instances. AWS is a platform that provides various tools, such as Lambda, that unfortunately lock you in to Amazon and force your teams to learn a provider. Think IBM and its long history of lock-in.

This is why we’re starting to see a transition from running code in VMs to going up one more layer and running code, when appropriate, within containers. They can easily move from the developer’s desktop to production in a easy and hopefully seamless workflow — they span providers and use a higher level solution, like Kubernetes, to manage scheduling of the containers into a cluster of compute nodes.

Other providers on this list, like Google, are tending to focus on bringing in workloads for which they are known; e.g., it’s not uncommon to see big data companies shifting to Google Cloud to take advantage of the platform Google has built for dealing with large data sets. Google is also doing a great job in helping to democratize running container workloads on their platform and everyone else’s.

By no means is this list comprehensive. Its intention is to highlight some of the common providers along with others who focus on delivering better performance or on-premises solutions.

Our list of cloud providers includes:

• Amazon with EC2, EBS, ELB and the rest of their platform. There’s an assortment of solutions to choose from at Amazon. I’ll leave it up to another post to dissect everything in the catalogue.
• Google with GCE as their compute solution.
• Microsoft Azure: Another platform-as-a-service where you can consume traditional compute or build your application around Microsoft’s hosted services.
• ProfitBricks: Provides infrastructure as a pure utility with a focus on performance and flexibility.
• Rackspace: A managed cloud computing company
• SoftLayer: Owned by IBM, a provider of virtual machines, bare metal and other services.
• VMware: With the largest install base, VMware is the leader in on-premises cloud infrastructure.

Developers can choose among different infrastructure providers. But, as these platforms become more platform-centric, smaller providers are beginning to differentiate on things, like performance. For example, ProfitBricks focuses on high performance, which makes them ideal for workloads that require good I/O, while others, like Packet, are focusing on delivering rapid, bare metal solutions. As developers shift from virtual machines to containers running on virtual machines or metal, performance and differentiators like it will become important.

Database Decisions: Become More of a Conundrum

It seems that you can’t crack open a tech blog without seeing yet another database startup hitting the scene with their own take on how to verticalize database services. We’re seeing cloud databases popping up, database solutions that address vertical needs, more NoSQL than we know what to do with — it seems the list just keeps growing.

For a developer, there’s a lot with which you can play, experiment, work with. Instead of entering a project with a preconceived idea of which DB solution to use, teams are beginning to fit the database to the problem they’re attempting to solve; e.g., instead of choosing Microsoft SQL for a metrics app being built, you would now choose some type of time series optimized database, such as InfluxDB.

We’re at the point where you can map the database to the problem versus being forced to map the problem into the database.

Common database solutions being used today include:

• Apache HBase: a column-oriented database management solution.
• Cassandra: A hybrid key-value, column-oriented database designed to handle large amounts of data distributed across commodity servers.
• CouchBase: Another document-database.
• Firebase: A real-time database service delivered through the cloud, focused on keeping your applications in sync — offline or online.
• MariaDB: A fork of MySQL to ensure it stays free after the Oracle acquisition.
• MongoDB: The most common document-database you’ll find.
• MySQL: Traditional, RDBM.

The beauty with SQL and NoSQL solutions is that, in general, they’re all free. Developers and startup teams do not need to rely on an enterprise agreement to quickly build and deliver a database-driven application.

Configuration Provisioning: Automation Adds Speed and Consistency

It is important to ensure your environments — development, test, staging, production and so on — are identical. It’s also important to be able to orchestrate changes across your environments in bulk or in small portions, depending on what your update needs are. This has been a problem in tech for quite a while, and why we’re seeing the rise of a variety of IT automation tools and containers.

Configuration management is mostly used with virtual machine infrastructure, at this point, with containers being managed in a slightly different way. This is actually leading to a growing debate around the continuing need for tools like Chef to manage environments. As Dockerfiles become richer, we simply make changes in the container and then the container gets pushed into production or multiple copies of the container are pushed.

The most common solutions in this space are:

• Ansible: A simple, agent-less solution built in Python. Ansible has been catching on, for various reasons I won’t get into here, and it is becoming one of the more common tools you’ll see in use.
• OpsCode Chef: One of the most popular solutions, built in Ruby, with broad support for Linux and Windows environments.
• Puppet: Another very popular solution — again, built in Ruby.
• SaltStack: Another easy-to-use Python-based solution.

Some of the less common tools you might find would be things like CFEngine and Rudder. Developers will commonly use tools like Vagrant to build out their own virtual machine environments when testing and working. Hashicorp has also given us neat things, like Terraform, that help us ensure we have a repeatable infrastructure build and configuration management.

All of these tools help you embrace the concept of infrastructure-as-code. With any of the above tools, you can orchestrate repeatable build-outs and changes to your environments. Need to make a change? Make the change in your given tool, then instruct that change to be reflected across all environments. This allows you to treat the environments as boxes that can only be modified through configuration management.

The day of administrators logging into individual systems is long past due.

Continuous Integration Tools: Refining Development and Operations

As teams adopt continuous integration, tools like Shippable, Jenkins and TravisCI will become increasingly more important. They allow the developer to easily ship code that is then verified upon check-in. This code can then be, say, merged into master, if you’re using Git, automatically — once everything (unit tests and integration tests) passes. Developers can be alerted to issues in the build ahead of time and fix it before it hits the production repo. Once they merge, the system inspects the change and alerts them.

CI is one step towards its natural conclusion: continuous deployment.

An operations team — or the developer themselves, if they have access — can leverage many of these same systems to push into production: once a merge into the share repo happens, deploy into production. CI/CD allows the delivery of applications from development into production at a higher quality and faster pace.

Continuous integration allows your team to catch issues early by doing continuous builds that get merged into master. You can even plug this into a continuous delivery system that deploys it out into production.

Companies in this space include:

• Buildbot
• CircleCI
• CloudBees
• Codeship
• Continuum
• Cruise Control
• Jenkins
• JetBrains
• Shippable
• Snap
• TeamCity
• TinderBox
• TravisCI

It’s obvious the space is already crowded — with many more rising — as Docker and containers continue to rise as a target. Some of these companies focus on leveraging containers for their platform, while others let you test your code then ship the containers into the registry of your choice.

Infrastructure Automation Ecosystem Summary

Infrastructure automation is going to continue to grow tremendously. Containers are heating up. This will drive more investment into the space. I would expect to see quite a few categories highlighted as the infographic continues to grow. The rate of adoption of DevOps teams, and processes and other types of tooling, is trending upward significantly. Just drop Ansible into Google Trends to see why Red Hat bought them. This is all leading to explosive growth in the space.

Sure, there are still gaps and rough spots to work out in the future, but that’s only an opportunity for more startups to emerge. Follow this list, as we hope to update it on an ongoing basis as the ecosystem evolves.

Review the infographic and recommend new companies get added or moved. We’ll be updating the graphic throughout 2016.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.

Feature image: “looks like cubes exploding outwards” by TORLEY is licensed under CC BY-SA 2.0.