The Configure-Once Kubernetes Platform for Seamless Dev/Prod Integration – InApps Technology 2022

Many of the challenges developers run into with Continuous Integration (CI) pipelines stem from the fact that CI is siloed from the rest of the development process.

A typical development-to-CI workflow in the cloud native era consists of coding and running unit tests locally before pushing to GitHub or GitLab, while integration and end-to-end tests only run in CI. Development environments use a completely separate (and often pared-down) configuration compared to CI, which builds, tests, and deploys in a more production-like setting.

Let’s think through how this approach impacts developer experience.

First, the discrepancy between development and CI environments leads to hard-to-predict errors in CI. No matter how much testing happens during development, there’s always a chance of running into errors and test failures in CI — so long as there’s a discrepancy between dev and CI environments. Classic “but it worked on my machine” issues.

Second, developers have no idea if integration tests will pass when they push to CI. Developers miss errors they would have been able to catch with a nice, tight, pre-commit feedback loop that includes integration testing. And fixing these errors forces a dev to return to work they’d thought was “done.” This is the sort of context switching that might be tolerable every once in a while but can take a big toll on productivity (and happiness!) over time.

Third, the process of troubleshooting CI is slow and tedious. Imagine a developers encounters an error in CI that they didn’t come across during development — which we expect to happen fairly often, since there’s no integration testing before pushing to CI. With every attempted fix, a dev has to wait for the images to be rebuilt and for the CI pipeline to run. The timescale here is usually at least getting up to make a cup of coffee, if not longer. It’s a slow feedback loop that results in a lot of time spent sitting around and/or frequent context switching.

And fourth, even just writing integration tests takes a lot of time and effort. When debugging a newly written integration test, a dev needs to go through the same slow, unpleasant CI feedback loop we just described above—not much of an incentive for developers to ensure solid integration test coverage.

We can boil all of this down into a couple root issues:

1. Inconsistencies between development and CI environments
2. Slow and insufficient integration and end-to-end testing

The good news is, we believe that both are solvable!

We’ll lay out one possible path forward using a project called Garden, and we think it’s most useful to share our vision alongside concrete steps you can take to implement it. Full disclosure, we’re Garden’s creators and maintainers. It’s open source, so anyone can follow along and try it out. And feel free to ask us questions along the way.

Configure Once, Run Everywhere

The developer experience we described above can be meaningfully improved if we provide three things:

1. Production-like, “full-stack” environments that are consistent across development and CI
2. An easy way for developers to spin up these environments for coding and testing
3. The ability to run the same integration tests during development that eventually will run in CI

But before we talk more about the developer experience with Garden, here’s a quick primer on how Garden works.

To use Garden, you write a brief configuration file next to each of your services, which lives next to a Docker file and your source code. These config files provide a consistent way to describe to Garden how each service is built, tested, and deployed, without messy scripts or a massive, monolithic configuration.

Each of these steps (building, testing, and deployment) can also describe dependencies. For example, it might be the case that an integration test for Service A has a dependency on Service B (a different service in the same project). Garden will know that in order to run integration tests for Service A, Service B also has to be running and up-to-date.

Garden scans for all of these declarations (even across multiple repositories), validates them and compiles them into a DAG that describes all the steps involved in building, deploying, and testing your application. This is called the Stack Graph, and it sits at the heart of everything Garden does.

The “distributed” approach to configuration makes it a suitable choice for complex microservices applications made up of many different components. Each team can manage their Garden config files independently, and no one has to grapple with a dense, “centralized” configuration file.

Garden also makes it easier for developers to reason about the application as a whole, with the Stack Graph providing “living documentation” to show the relationship between all of the different moving pieces. This is the sort of contextual awareness that’s really difficult to convey otherwise.

The result is the kind of dependency-aware pipelining that you usually only have access to in your CI system, but Garden gets you all of the same capabilities in your pre-commit workflows.

Production-like Environments for Dev, Testing, and CI

The Stack Graph then serves as a foundation for every pre-production environment, be it development, testing, CI, or even an application preview that PMs and QA engineers use for acceptance testing.

With a single command, any application developer can spin up their own namespaced, production-like environment in a Kubernetes cluster. But a dev environment is only as useful as what a developer can do with it. Because all tests — including integration tests — are part of the Garden configuration and live alongside the application code, a developer can easily run integration tests in these environments, during the dev process.

It’s important to note that these are the exact same integration tests that will run in CI, in an environment with the exact same configuration as CI. If a test fails during the dev process, a developer gets that feedback right away and can fix the problem while they’re in context and writing code.

By the time you’ve pushed to CI, you’ve already successfully run your integration tests during development. And because your pre-commit environment is the same as your CI environment, you can be pretty confident your tests will pass in CI.

It’s also straightforward to make Garden a part of your CI pipeline. The only thing you need to install in CI is the Garden CLI and its dependencies or use a ready-made container image.

A CI pipeline run with Garden is basically just a matter of checking out a repo, configuring a connection to a K8s cluster, and running garden test (a command that runs all tests defined in the project, builds modules and dependencies, and deploys service dependencies if needed).

And once you’re using Garden, there’s no need to change your CI configuration when you change your stack since Garden holds the entire Stack Graph.

What About Non-Technical Stakeholders?

It’s also easy for Garden to spin up a working preview environment of your application with every CI run, which can be used during the QA and review process. Whoever’s carrying out a code review, (or manual QA, or pre-release acceptance testing) can see not only that automated tests have passed, but can also click through a working version of the application and get a sense of how it behaves.

These preview environments are especially helpful when collaborating beyond the dev team. They’re ideal for product managers or other non-engineering stakeholders who need to do acceptance testing before a release.

And with that, we’ve covered every step of our pre-production development and testing pipeline.

Wrapping Up

Thank you for reading, and we hope you found it helpful. Here’s a quick recap of what we covered today:

• Most of the problems that developers run into with CI are caused by a) discrepancies between dev and CI environments and b) insufficient, slow integration testing.
• One possible approach to solve these problems is to use a consistent configuration for every pre-production environment, from development to testing to CI.
• Garden is an open source project that makes it possible to describe your entire stack — all services, tests, and dependencies — then spin up on-demand, full-stack environments at every step of the dev pipeline.
• Consistent environments reduce friction in the dev process and enable developers to ship more and to ship faster.

We’re always interested in learning about how organizations manage their dev and testing pipelines, so if you have any questions (or you’d like to give us feedback about our approach), feel free to drop us a line in our community forum.

If you’d like to get started with Garden, we recommend checking out the docs, and we’re happy to help along the way.

Feature image by Ignacio Correia on Unsplash.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker, Camunda.