How We Built Preview Environments on Kubernetes and AWS – InApps 2022

For two years at Qovery, we built a Preview Environment system to help developers get a full replica of their production environment for every pull request. Meaning when you create a pull request for an application, Qovery duplicates all the other applications and databases.

Building our Preview Environment system was challenging because we needed to adequately support applications, databases, networks, secrets and many other services. This article will explain how some components work for our Preview Environment system.

Some people call preview environments “review apps” due to their background with Heroku. The main difference with the Qovery Preview Environment system is that all the applications and databases run inside an environment. Each environment is attached to a Kubernetes cluster. I will go into deeper detail below.

If you want to see in detail how Qovery Preview Environment works, check out our open source Qovery Engine.

Environment

On Qovery, every application and database belong to an environment. It is a logical entity that links all resources together. When turning on the Preview Environment feature, Qovery will duplicate an existing environment to create a new one.

A new environment will be created with all the resources for every pull request created. Then you can try out the new feature in a new isolated environment. There is no risk of breaking your original environment; that can be the production environment.

Let’s dig into the details of each service created, from the simplest to the most complex.

Cloning an Application

On Qovery, an application is a container running in a Kubernetes pod. Cloning this application when a Preview Environment is created results in creating a new Kubernetes namespace and duplicating the Kubernetes deployment configuration by changing the namespace ID.

The simplest part is if you do not consider the network and the environment variables and secrets yet. Building a POC (proof of concept) supporting only containers is straightforward and can be done in a few days. This is what most of the preview environment systems out there provide. But things start to get more complicated for real-world use.

Cloning Environment Variables and Secrets

You might not want to use your production environment variables and secrets for a preview environment, right? That is why we built an inheritance and scope system to manage your environment variables and secrets across your different environments.

Inheritance

Let’s say you use the environment variable NODE_ENV=production. You might want to use the same environment variable with the value development. This is what you can do with Qovery, and it means you need to map the values depending on the environment dynamically.

Scope

Let’s say you have 20 environments and you want to set an environment variable for all your environments. With no scope feature in place, you will have to set it individually for each environment. That is why we associated a scope to every environment variable and secret. You set an environment variable with the “project” scope, and it will be available for all the environments.

Secret Encryption

A secret is similar to an environment variable, except you can’t reveal the value, meaning it is write-only. Behind the scene, Qovery encrypts and salts the value, and stores it in the Kubernetes secrets service.

Cloning the Network

Cloning the network results in cloning the configuration for the internal (Kubernetes services and ingresses) and external network (load balancers, gateway, etc.). The most complex is the external network because you have to consider generating a new external domain and TLS. We also let our users set their own domain and wildcard domain, which is additional work to make sure everything is working correctly.

Cloning Database

Qovery provides two types of databases, the managed one and the container one. AWS provides the managed databases, for instance, AWS RDS Postgres. It is great for production, and AWS guarantees backups and scaling with Postgres’s container mode providing a container instance with attached block storage (AWS EBS) for persistence.

It’s cheaper and faster to spin up but with no backups, which is excellent for development purposes. Depending on the database type chosen, we will clone the database and the appropriate network services. On RDS, we also need to consider creating a second VPC (virtual private cloud). This is where everything starts to become more complex.

Cloning a database also means seeding the Preview Environment database, or even clone the production database, which is not always possible for legal or technical reasons. One thing we had to consider at every stage is, “What happens if X operation goes wrong?” This is why we have built a transactional deployment engine. I would be happy to explain how it works in a future article.

Resources Isolation

Depending on their usage and how critical their business is, our users choose to use physical isolation combined with a logical one between their Preview Environments. It is a premium feature that requires a huge amount of development time from us. Physical isolation is safer than logical isolation, but it is also more complex and expensive to maintain. For instance, SaaS in the health care industry must comply with HIPAA security requirements. They need to provide physical isolation between their users, while a SaaS in the gaming industry can stick to the logical isolation model.

Logical Isolation

Kubernetes namespaces provide logical isolation in our context. Each environment has its own namespace. Deleting one namespace results in deleting all the Kubernetes resources of an environment. At Qovery, we do not consider Kubernetes namespaces, even with the appropriate RBAC rules, suitable for multitenant applications.

Physical Isolation

For physical isolation, Qovery provides different architectural choices.

From VPC to Kubernetes cluster and cloud account isolation, the Qovery platform uses a high level of abstraction to manage those architectures.

Testing

Building a Preview Environment system robust enough to be used by thousands of developers must be well tested and where we spend at least 50% of our development time. We have thousands of tests running to validate that we fit all the normal and edge cases, from unit tests and functional tests to end-to-end tests (E2E). It is a huge part of our development.

Conclusion

I explained how we built our Preview Environment system on Qovery with Kubernetes on AWS. Building a stable, secure and performant Preview Environment system is not easy and takes a huge amount of time, especially when managing stateful resources like databases and networks. However, the reward is massive for companies using the Preview Environment features.

For example, here is a company that releases more than 100 times faster than before with the Preview Environment feature in less than a week.

Any question? Feel free to contact us on Qovery or check out our Qovery Preview Environment page.

Feature image via Pixabay