3 Ways to Use Automation in CI/CD Pipelines – InApps Technology 2022

Successful DevOps, continuous integration (CI) and continuous delivery (CD) largely hinge on automation in a number of ways. This is because scripts, plugins and other tools for automation help to standardize production and deployment processes, leaving less room for software delivery error.

By relying on automated processes to complete repetitive and time-consuming tasks, DevOps teams also have more time to add value to application delivery in a way only humans can deliver. The end result: applications are delivered and managed much more rapidly and securely while DevOps teams have more free resources to devote to delivering better value to the end user.

Without automation, a developer might spend a substantial amount of time working on configuration files instead of creating brilliant code that is quickly approved and moved into production. A deployment of a new tenant across a multicloud infrastructure takes days when multiple manual steps are involved, instead of just with a few mouse clicks. An audit requires days of chasing down files and data and other forensic steps that should otherwise be built into the production pipeline so the required information-retrieval process is automatic.

“You have one slowdown, one choke point, one single manual road test point of failure and it sets off a ripple effect on productivity you simply can’t afford,” Shawn Ahmed, senior vice president and general manager of software delivery automation, CloudBees, said during the DevOps World 2020 keynote. “So, if you’re a company that still relies on manual processes for a set of disconnected tools to build and deliver software, you simply can’t survive and scale without a fully automated software delivery automation backbone. And this isn’t just me talking here — the industry trends bear that out.”

Automation was, understandably, a key theme in this conference. In numerous talks and keynotes, many substantive tips and tools were communicated to help DevOps teams automate their production pipelines.

In this post, we gathered a few examples that show how organizations can fill in automation gaps in their CI/CD processes, in order to ultimately deliver better value to the end user. Among the many tools and processes discussed, three creative approaches stood out for their ability to meet both engineering and business goals.

Post-Deployment Checks

The first key takeaway is that automation must span the entire CI/CD concepts life cycle to include the deployment and post-deployment stages of software delivery. However, teams consistently fail to automate the post-deployment stage of their pipelines, Amit R. Bhandarkar, director of engineering, American Express Global Business Travel, said during a panel discussion about DevOps, held for media and analysts.

“Very few teams have implemented any sophisticated post-deployment automation into their CI/CD,” Bhandarkar said, “which is usually handed off to a separate team that may use automation to validate the deployment, but it’s not part of the post-CI/ CD pipeline.”

At American Express, Bhandarkar said development teams are analyzing how they can better validate post-deployment changes to code. During the code validation process, for example, analysis and actions that are subsequently required should be automated as well. The process includes when an application fails and the appropriate rollback is automated.

“So there’s a lot of sophisticated automation for CI/CD,” Bhandarkar said.  “And usually, with post-deployment actions, there are gaps to fill.”

In addition to automating the post-deployment processes of application delivery, American Express has been investing in automation for several years, Bhandarkar said. During the course of the last year, for example, the organization has increased its number of security scans, code quality gates and post-deployment checks that have served to improve CI/CD efficiency and DevOps productivity, he said.

“We’ve also added support for a number of additional internal as well as cloud deployment targets through automation,” Bhandarkar said. “So, we do feel a deep sense of responsibility to continue innovating in the space.”

Automated Audit Controls for DevSecOps

DevSecOps also plays an obvious role in CI/CD. While automation is required to ensure security controls, monitoring and remediation are tightly integrated during the entire process, from initial code uploads on Git to once applications are deployed and distributed. For compliance, and when audits occur, automated “audit-ready” pipelines represent one way to ensure that retrieving audit-related information remains constantly available on an as-needed basis.

With audit-ready pipelines, for example, relevant audit-related data can thus be traced and tagged throughout the entire CI/CD process.

CloudBees’ Avan Mathur said audit-ready pipelines are critical for today’s DevSecOps. Forget manual and painful data searches — the pipeline itself “becomes the audit trail,” thanks to automation. #DevOpsWorld https://t.co/ePE2tjGGtx @thenewstack @CloudBees pic.twitter.com/3F23A2ggVn

— BC Gain (@bcamerongain) September 23, 2020

“We recommend that there is automation across the entire process with one release pipeline that builds the audit data right into it,” Avantika Mathur, senior product manager, for CloudBees said, during a talk. “The pipeline itself becomes that audit trail.”

Audit data is collected as part of the pipeline as it runs in one central location, Mathur said. “Access controls and compliance are then baked right into the pipeline to ensure that processes are being followed,” Mathur said. “And only authorized users are taking actions. So, all of the data, all of the tools and everything are in one place rather than hunting down what happened.”

Opinionated CI/CD

Automation is also required for relying on CI/CD to form a so-called “opinionated” application delivery process — which, among other things, means simplifying the user experience — when applications are deployed.

“The last thing we’re doing as part of our cloud automation process is providing opinionated solutions for infrastructure. This extends to automated compliance, governance, security” and financial operations, Gerard McMahon, head of cloud center of excellence for Fidelity Investments said during a press conference. “The end result: We can get either architecture patterns or common blueprints to standardize how teams and groups deploy into the cloud for their particular applications.”

Live Panel: There are automation gaps to fill. American Express Global Business’ Amit Bhandarkar: “The CI/CD pipeline, in most teams that I’ve seen, concludes at the deployment of code in the target environment. @devopsworldconf @thenewstack @CloudBees pic.twitter.com/j0sI7P9gQP

— BC Gain (@bcamerongain) September 23, 2020

Into the DevOps Future

Ultimately, automation, and especially its implementation, has laid the groundwork for DevOps. For more than a decade, the concept of DevOps and its importance in the organization and for CI/CD has been widely adopted. Automation has further accelerated a number of positive trends, including the removal of silos between the various DevOps stakeholders, including developers, operations, security and business teams. In many ways, the adoption of software delivery automation and tools is just the beginning.

“When it comes to the different pieces of the software delivery automation that are in place today, everybody is kind of in the middle of a rat race,” Daniel Ritchie, a distinguished engineer for Broadridge, said in a “Live with InApps Technology Makers” livestream during DevOps World. “This move into software delivery management, this idea that we’re going to have a number of ways to deliver software that goes beyond each of these automated components — I think right now that expansion is what we’re getting into and that’s really pushing the edge in a way that is defining the future of DevOps.”