Applied GitOps with ArgoCD – InApps 2022

The traditional way of deploying applications is either with manual work, or by using a “Push” process — where a continuous integration (CI) system is sending updates to a Kubernetes cluster (or other deployment targets) with new software releases. GitOps is an alternative deployment paradigm, where the cluster itself is “pulling” updates from manifests that reside in source control (making “Git” an integral part of the name). There are already several articles that explain how GitOps works in an abstract way, but very few that show how all GitOps directives map into actual tools.

Argo CD and Flux are the two major solutions for implementing a GitOps workflow with Kubernetes. At some point, they were even going to merge into a single solution, but this is no longer the case and the two tools will remain independent.

In this article, we have chosen Argo CD to illustrate how GitOps looks in practice. Argo CD is an open source project that is also part of Cloud Native Computing Foundation (currently as an incubating project).

Using git as the Source of Truth

This is arguably the most important tenet of GitOps (hence the name). All components of the deployed application should be stored under source control. This means not only the source code itself, but also the Kubernetes manifests that describe how the application is run inside the cluster.

Usually, the manifests are placed on a separate git repository than the application code, but this is not strictly necessary.

Creating an application deployment in ArgoCD starts from the Repository settings. Here you can add your public or private git repository that contains the application manifests. ArgoCD will use the connected repositories for performing application deployments.

ArgoCD has also recently added native support for Helm repositories. This means that you can now directly connect the path of a packaged Helm chart (as created by your CI solution) and ArgoCD will automatically monitor it for new versions.

If you don’t use Helm or plain Kubernetes manifests, you can also use Kustomize with ArgoCD.

Declarative Deployments

After placing everything in Git, the next step is the actual deployment. Here you can see the magic of GitOps, because unlike other CI tools that follow an imperative approach (which usually revolves around custom kubectl scripts), ArgoCD performs deployment by “syncing” an application to the cluster.

The sync process is essentially a consolidation of the current cluster state (which is initially empty) to the desired state (described by the git manifests). The ArgoCD engine is deploying and monitoring all application components until both states are identical.

After the deployment has finished, we know that the state of the cluster is exactly the same as described in git. The git state essentially reflects the cluster state at all possible times.

Having an equal state between the cluster and git sounds simple in theory, but is actually one of the core tenets of GitOps. It’s shocking but true that most companies perform “cowboy” deployments, where developers or operators are manually performing changes to the cluster. These ad hoc changes are never recorded anywhere and are incredibly fragile. With Argo CD this behavior is no longer possible — manual changes that are not recorded in git can be easily discarded, as we will see later.

Adopting GitOps is much easier in companies that already follow infrastructure-as-code practices, as this means that teams will already be familiar with dealing with git (and Pull requests) as the principal way to perform deployments.

Repeatable and Auditable Deployments

Two of the classic questions that software teams always ask are:

1. What version is right now deployed in environment X?
2. Can we quickly rollback to the previous version in environment Y?

These questions are sometimes very difficult to answer. Knowing what version of an application is deployed in a cluster is not always easy, because sometimes several “hotfixes” or “patches” have taken place after the initial deployment and in an ad-hoc manner.

With GitOps, things are a bit easier. The contents of the git repository tell you what should be deployed in the cluster. If the deployment is also synced successfully — either by ArgoCD itself or another external solution — then the git state is indeed the same as the cluster state.

Git commit history can essentially function as the cluster deployment history. Reasoning about the cluster state is straightforward, if you can guarantee that it mirrors the contents of the last git commit.

Performing rollbacks in Argo CD projects is also very easy. In traditional CI applications, you need to find and trigger the respective deployment pipeline with different parameters.

With Argo CD there is a clear distinction between Continuous Integration and Continuous Delivery. You can simply choose another git release that will be used for the sync process and be certain that after a while the cluster will revert to that commit hash.

Alternatively, you can use the standard git reset/revert actions (if you want to roll forward) and the cluster will eventually follow your changes — either by ArgoCD auto-syncing or using another solution to perform the sync.

Handling Configuration Drift

We briefly explained that in traditional deployments, several times teams might perform changes to the cluster which are not recorded anywhere. This is a very common occurrence for one-off changes, quick patches, or other emergency updates. While kubectl is a very powerful way to manage a cluster, teams can also abuse kubectl and create configuration drift problems.

Configuration drift is a notorious problem, as the longer it persists, the more critical it becomes. Teams that don’t know what exactly is deployed on a cluster, or multiple clusters, can have many issues with failed deployments.

A classic scenario for failed deployments is the discrepancy between a staging and a production environment. Developers test their application in the staging environment and assume that it will also work in production.

A developer will try to perform a production deployment and it will fail because somebody had made ad hoc changes to production that were no longer reflected in staging.

Argo CD has a very smart way to solve the configuration drift problem. After the initial deployment takes place, Argo CD continuously monitors the cluster and compares it to git.

If at any point somebody performs a manual change in the cluster, ArgoCD detects it and marks the deployment as “out of sync”.

This instant visibility of out-of-sync deployments cannot be overstated. ArgoCD not only knows that the application no longer reflects the state of git manifests, but it even offers you a handy diff view to explain what was changed.

In the example below, somebody tampered with the ports exposed by the application service.

When you face this situation you now have a choice of actions. In most cases, you can simply sync again the deployment. This will discard the manual changes and bring back the cluster to the same state as the git commit.

If you believe that the manual change is indeed essential (maybe it was an emergency patch) you can simply commit it to Git. This way ArgoCD will compare again the cluster and the git state, and realize they are now the same.

The first scenario (discarding manual changes) can also be fully automated so that Argo CD is always fetching the cluster state from Git.

Argo CD essentially cures the configuration drift problem once and for all. Companies that have a big number of clusters that are currently suffering from configuration drift will be able to revolutionize their deployment process by adopting Argo CD.

Apart from the basic GitOps features that we have described in this article, Argo CD has several other features such as sync windows, deployment hooks, application grouping, etc.

Completing the Continuous Delivery Puzzle

Argo CD is just one component of a larger deployment story. Handling secrets, triggers, pull requests, builds, etc, makes it important to pair ArgoCD with a robust automation platform that can provide a full view of the software lifecycle — and also handle the CI part, as ArgoCD expects all deployment artifacts to have already been created.

At Codefresh, we understand that Argo CD cannot be used on its own for the whole software lifecycle. Argo is great for deploying an application, as the last step of the delivery process, but it must be coupled with another platform that will need to perform all the steps that come before it (e.g. running unit tests).

This is why at Codefresh we just launched the first component of what we’re calling GitOps 2.0, which is not restricted to just deployments but combines the best of both worlds (Continuous Integration/Delivery and Observability).

Having a solution that can cover the full software lifecycle is essential — especially for companies that wish to practice Continuous Deployment and even perform commits automatically, in the Manifest Git repository monitored by Argo.

List of Keywords users find our article on Google