Blueprint for Securing Your CI/CD Pipelines – InApps 2022

Recently there have been an increasing number of cybersecurity incidents that have affected organizations of different kinds and sizes. These incidents have prompted urgent and deliberate attention to hardening the overall cybersecurity posture of our software IT infrastructure. One critical area that has a lot of momentum, from the highest of government agencies to major cloud vendors and open source foundations, is around the security of our software supply chain.

The typical scope of the software supply chain starts with a developer writing the code, to when the application is deployed and continuously monitored on a cloud. Within this spectrum there is a crucial section of the continuous integration/ continuous delivery (CI/CD) pipeline where we implement all our DevSecOps controls, perform builds, publish artifacts and finally deploy on the cloud. These are arguably the most security-sensitive operations throughout the application life cycle. There has been tremendous momentum to innovate and bring unique security controls inside CI/CD pipelines, including static code scanning, dependency auditing in software bills of materials (SBOMs), sign verifications and much more.

As we beef up our pipelines with these security controls, I want to point out that our CI/CD pipelines themselves are susceptible to security breaches. And there is evidence from recent events that proved the weakness in the configuration and operations of our CI/CD pipelines. Therefore, it is essential that we give equal consideration to securing our pipelines, just as we do for our production workloads. In this article, I will discuss a blueprint through which we can try to accomplish this.

Life Cycle of CI/CD Pipelines

Just like applications, our pipelines also have their own life cycle, as shown below. We will use this as reference to drive our discussion around the desired security properties for our pipelines at various stages.

Pipeline Composition: This is the stage where we code and compose our pipelines. There is a growing open source ecosystem around pipelines where a number of common functions like git-clone, vulnerability-scan, container-build, etc., are made available for use off the shelf from catalogs like GitHub Action Marketplace or Tekton Catalog. Therefore, it’s natural that during composition we use these functions in our pipelines. But, again, open source does not mean that it’s free! We need integrity and security verifications before using these functions. That can be accomplished through sign verification and standard vulnerability scanning. There’s an open source tool we have built, called “tapestry-pipelines,” that would allow us to sign and verify pipeline resources.

Pipeline Configuration/Install: Once composed, next we install and configure the pipelines. This typically includes tasks like setting up worker clusters to run our pipelines, configuring pipeline parameters and setting up user credentials. In principle, just like we have Docker and Kubernetes CIS Benchmarks for containers and Kubernetes services to guide us in their respectively secure configuration, we need to design a secure configuration checklist for CI/CD pipelines as well.

Pipeline Trigger: Next, the pipeline is executed by some triggers, which could be event-triggers like GitHub Events on pull_request/push, manual triggers or even timed triggers like cron jobs. This is the last resort at which we need to ensure only blessed pipelines are allowed to execute. The criteria for “blessedness” could differ by user, which typically would include checking whether pipeline definitions are signed and that images referenced for executing the functions are scanned. At the same time, we need to check whether the event triggers are coming from authorized sources. For instance, in the case of GitHub Events, we observe the event payload to verify the identity of the user. Once cleared, our CI/CD pipeline would then start executing on the cluster.

Pipeline Monitoring: I think monitoring in general is an underappreciated tool in the security domain. Although it might not help in preventing any security-related issue, it is probably the oldest and most mature technique for discovering them. It can effectively be employed to monitor all pipeline runs and to identify any resource bottlenecks and anomalous behaviors across multiple pipeline runs. Plus there are existing cloud native monitors available; we just need to enable the right instrumentation in our pipelines to collect this telemetry data.

Pipeline Attestation: As the pipeline execution completes, we need to capture signed attestation records for every function executed in the pipeline. For instance, for the Tekton CD pipeline, we can leverage tektonCD/chains that introspect individual task execution states and records its signed attestation. It also automatically signs any container image built and published in the pipeline. Going forward, there is an intent to automatically sign every artifact produced in the pipeline. Also, I had submitted an extension proposal to the Tekton community that would allow us to record end-to-end provenance, all the way from event source to signed artifacts in our CI/CD pipelines.

Pipeline Auditing: After the pipeline execution finishes and is recorded in the history, we need an ability to audit the pipeline and any artifacts produced by that pipeline, at any point in the future. For instance, say I have a CI pipeline that builds and publishes a very popular database application image called “whySQL,” and I frequently release new versions for this image. At any point, under supply chain security practice 101, before using that image, if you want to determine its provenance in terms of the attestation record(s) for the pipeline used to build that image, and get confirmation that the pipeline was signed and verified, then we need to facilitate an automated auditing framework for users to do that.

In summary, I believe this blueprint is a good start for us to begin securing our CI/CD pipelines. We are looking to build these technologies with the support of our amazing open source community. So if you have any feedback or are willing to help and contribute, please reach out to me on GitHub, Twitter or email.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.

Photo by Flo Dnd from Pexels.