Building a Collaborative DevSecOps Culture that Works – InApps 2022

The cultural and technical change of DevSecOps can be tough to grasp. But Tanya Janca, a security trainer, offered InApps readers a succinct definition: “DevSecOps is what the security people do to enable developers and operations to make awesome, rugged, safe and secure systems.”

DevSecOps puts security at the center of the DevOps pipeline. It’s another technical and cultural shift left, which sees security engineers embedded on DevOps teams throughout the software development lifecycle, from ideation to sunsetting.

In this model, security is no longer the last hurdle right before production. DevSecOps means cybersecurity teams understand how to codify their policies, and developers and sysadmins learn firsthand how security can be used to actually speed up release cycles.

DevSecOps culture and processes aren’t easy to foster, but it’s the only way forward as high-profile cyberattacks have become the norm. And with cybersecurity experiencing a terrifying talent gap, everyone has to pitch in to keep systems secure. Here’s what you need to get teams into the right unified mindset necessary for DevSecOps success.

Setting up a DevOps/Security Mind Meld

“The whole DevOps movement to me was always about security, because I thought security was part of operations. But then I realized, we do want to call it out, because security is an equal partner,” Ashley Ward, senior product marketing manager for Prisma Cloud at Palo Alto Networks’ enterprise cybersecurity platform, told InApps.

Security has always been an operations consideration, Ward noted — but only when code heads to production. A report by Palo Alto Networks’ Unit 42 threat researchers found a surge in cloud adoption from 2019 to 2021, all while security was being left behind. In addition, a 2020 survey from the Linux Foundation found that only 3% of developers were interested in being responsible for security.

It adds up to not a lot of optimism for DevSecOps adoption. But it’s still desperately needed, according to experts like Ward.

There’s already so much developers need to know about their technical stacks and the cloud even before writing a line of code, Ward told us. DevOps teams are overwhelmed with faster delivery. Having to then learn the intricacies of vulnerability and compliance management would just be adding to their perpetual burnout.

“If we focus all our time on only the production environment, if security has blocked that, then that team is already onto another thing,” Ward said. That’s why teams need to “shift left” and learn to develop and operationalize security. While they don’t often realize that it’s available, security teams need to embrace codification of security or policy as code to translate their documentation so it can be consumed by DevOps teams and pipelines.

It’s not just about technology. It’s about security understanding how the application is used and what DevOps teams are doing. If developers are using specific toolsets, security needs to operate within those toolsets.

“Nobody knows everything. It’s about being humble and communicating that so we can figure out how to operate within that organization,” Ward said.

For instance: Things like the European Union’s General Data Protection Regulation (GDPR), which dictates how data is moved and stored, affects the creation and deployment of code. It’s up to developers and sysadmins to ask about how GDPR works, and it’s up to security to learn DevOps pipeline automation.

“Security is about enabling the business to do exceptional things in safe ways.”

— Ashley Ward, senior product marketing manager for Prisma Cloud, Palo Alto Networks

The problem is that DevOps teams often experiment like innovation sandpits, where creativity often flows in isolation.

“A lot of the time in DevOps, we try things out to see if they work and then you get budget for it. And then people get down that path — we need to consume more cloud resources, but then security lags behind that,” Ward said. “Who is going to engage security with what we’re doing, when we don’t even know if we’re going to do it?”

It’s not just the DevOps teams that are getting creative. Threat actors are adjusting all the time. The number of cryptomining accounts has dropped, Ward notes, but the activity in each of these accounts has gone up, as they attack harder than ever, including making use of Kubernetes’ API to deploy attack containers into clusters.

Up against this level of sophistication, “The only way of doing it is to try and automate as much as possible,” he said. “With cloud native tooling, we can actually approach doing stuff in a different way.”

Until recently, Ward reflected, the security checks remained very manual, with people actually walking around data centers checking servers. Now cloud native tooling enables everything to be automated. Manual processes are just too slow, especially considering the current scale that has teams managing thousands of ephemeral containers and serverless, deploying to 5,000 virtual machines, several times a day. Instead, you can implement security within existing processes via APIs.

It’s also about increasing developer psychological safety, instead of encouraging an “If I don’t mention it, it’s someone else’s problem” mindset among team members. Security should be part of the whole team’s backlog.

Of course, as always, you have to measure the things that matter. For security, that means a risk reduction dashboard, showing the number of critical vulnerabilities dropping. For DevOps teams, it’s showing how security is increasing the pace of delivery, while decreasing the time they’re actually dedicating to security.

A Parallel Pipeline for DevOps Speed and Learning

Mohammed Imran, CEO of Practical DevSecOps certification program, says DevSecOps is about applying the CALMS DevOps Model to security: culture, automation, lean, management and sharing.

But security isn’t very good at the first two, he said. And if security doesn’t embrace communication and automation, they will just slow down the pipeline.

For the security trainer Janca, CEO and founder of We Hack Purple, DevSecOps is about bringing the information security department into the three axioms of DevOps, as discussed in both “The Phoenix Project” and “The DevOps Handbook”:

• Maximizing efficiency for speed
• Fast feedback
• Continuous learning

To achieve this, security team members have to automate quick scans into the pipeline, like scanning for secrets or running a continuous SCA to evaluate open source security and license compliance. Then they have to run necessarily tedious tests like static application security testing (SAST), in what Janca calls a parallel pipeline outside of production.

“I think the mission of DevSecOps is to say: I love the code you’re writing, and I think you’re rock stars, and I think we need to put security into this.”

— Amanda Nock, DevSecOps engineer,  NU Borders

Security team members have to be choosy with what goes into the pipeline, so as not to overburden the systems. For example, they can run a limited-scope version of their dynamic application security testing tool (DAST) continuously for the most important vulnerabilities, which vary by team. They can also record a test in HTTP Archive format, which will then only test those things specified, like checking if an application contains XSS or injection vulnerabilities. This can cut testing time by 70%, Janca said, while dramatically improving accuracy.

“I can target things like no one is using security headers, or people have a problem of cross-site scripting — which, by the way, most teams do. You can have your scanners just scan for that in a release pipeline because you know that it’s a problem and you want to stamp it out,” she said. Then, run a full DAST without scope and fuzzing in that parallel pipeline.

“If you’re going to put something in the pipeline, I believe it needs to give you fast, accurate feedback,” Janca said, which includes getting feedback to the right people through a data-driven approach. All scanning tools should create reports that feed into the same folder, so that a vulnerability management tool — such as DefectDojo, ThreadFix or ZeroNorth — can ingest and create metrics and trends over time.

Then a security champion can review and validate the results to remove false positives before adding the issues to a bug tracker. This step is necessary because, according to Janca, some SAST tools garner up to 90% false positives — and yet, if you adjust to “report only if sure,” you miss 80% of true faults.

Make sure to include development teams in understanding security flaws, so they can help prevent them in the future. “It makes them feel engaged and involved and actually give a crap so they do go and fix those bugs,” Janca said. “It helps make them care and helps show them that we are having a problem.”

To get developers engaged in security, Janca prefers the carrot over the stick. Before the Covid-19 pandemic, that took cookies, pizza and donuts. Now it takes coveted stickers, security books, and even “security champion” virtual backgrounds. And, always, extra help and advice.

What Does a DevSecOps Engineer Do, Anyway?

At the last three startups Amanda Nock worked for, “somebody Zuckerberged it,” which she defines as a couple developers building an application and releasing it, without really considering the consequences to security or anything else. This includes her current role as senior DevSecOps engineer at NU Borders, a trade and borders data analytics company that grew from five to 50 to 500 users. (At which point, the company realized it needed to consider systems and security, so it hired her.)

For each role, she was hired to head DevOps and then adopted the Sec part because no one else was doing it. “I was elbows up in AWS, and I see all those EC2 instances, and I can run Amazon Inspector,” she said. “We host our applications in Amazon Cloud, so it kicks it off every week, scheduled.”

Nock looks at her role as a DevSecOps engineer as “seeing a need and adopting it,” pushing security culture change and going where needed. For NU Borders, she got her Certified Information Security Manager certification to be able to talk as the information security manager to governments and to “say our DevOps and our security is a holistic team. Nothing is built without bringing security to the forefront.”

She can easily run an automated scan to find vulnerabilities and make smaller updates, but when there are bigger security issues they need to rearchitect for, she needs to create an issue ticket — “I know it’s not a feature, but this is something we need to do and to prioritize.”

After a year as the team’s DevSecOps point of reference, she reflected that it “really has to be involved in a person-to-person level, especially at companies this small.” (NU Borders employs about 50 people.)

“You are almost doing a PR job for security. Instead of thinking there’s this big faceless department around security, it should be ‘I should run this by Amanda.’”

When embedded within dev teams and Scrum processes, security becomes a cultural force, Nock said. Most recently, she’s worked closely with developers in creating initial architectural diagrams. “I was doing infrastructure in the AWS cloud, and they were doing diagrams for how data will flow.”

Cybersecurity specialists are hard to find, and the field is evolving so rapidly that a lot of learning happens on the job. Many people doing this work don’t hold computer science degrees.

DevSecOps doesn’t come down to formal titles or departments, Nock argued. It’s reminding people about something they might not think about much, and getting developers into hacking, including with the HackTheBox playground.

“Especially at these smaller startup companies, the technical stuff is hard work, of course, but the real challenge is making those culture changes to have a culture of DevOps and a culture of security,” she said. “We need to make this all automated, repeatable, secure.”

DevSecOps: Learning on the Job

Cybersecurity specialists are hard to find, and the field is evolving so rapidly that a lot of learning happens on the job. Many people doing this work — including Ward, Nock and Taurean McDade — don’t hold computer science degrees.

When McDade joined IHS Markit’s cloud security team a year ago, right when the information provider was merging with S&P Global, he was among five junior entry-level engineers working alongside three with much more experience.

“My boss wanted to create a team with individuals wanting to do great in the cloud but, unlike a lot of people in the industry, he wanted to bet on people who didn’t have the experience,” including no degree requirement nor work experience at a unicorn, McDade said. (His degrees are in marketing.)

It was an exciting baptism by fire, moving sprint to sprint, automating pen tests, running cloud security audits, shoring up the ample data sets, authenticating, networking, red-teaming, implementing SOCKS protocols, and assisting the organization’s cloud migration and AWS permissions, while pair programming and documenting everything together.

“To be a security person, you have to know what you’re securing,” McDade said. He warned that onboarding is needed, especially when moving data science teams — and widespread customer data — to the cloud for the first time.

IHS Markit is the sort of company with ongoing acquisitions, which he said “comes with acquiring the fluff. My task was to remove the stale DNS inquiries,” which involved talking to teams, learning if they were still using websites and explaining the security reasons it should be kept or not.

In order for DevSecOps to be a success, there has to be a cross-organizational buy-in that security matters and it’s everyone’s responsibility. As McDade said, “​​No one can be an expert because security keeps changing.”

Want to learn more about DevSecOps? Download InApps’s free ebook:

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Unit.

List of Keywords users find our article on Google: