Red Hat’s Perspective – InApps Technology 2022

Vincent Danen, Vice President of Red Hat Product Security.

I’ve known Vincent Danen for a long time. He was one of the first writers I hired when I served as a senior editor in the industry. From the very beginning, I knew there was something special about Danen, and now that he is serving as Vice President of Product Security for Red Hat, that assumption has been proven absolutely true.

I reached out to Danen to have a Q&A about cloud native security. Danen’s replies did not disappoint. Let’s dive in and see what he has to say about this hot topic.

Danen’s answers are his personal views and not necessarily those of Red Hat, though they provide great insight into the enterprise open source software provider and its strategy for securing Linux-based cloud native workloads.

What’s the biggest security issue cloud native developers face?

Speed and collaboration. Developers, especially cloud native developers, tend to move quickly. The technology they’re using moves quickly. Traditionally in the IT space, there have been silos, think back to “dev” and “ops.” Those silos have been broken down somewhat with “DevOps.” As buzz-wordy as it sounds, “DevSecOps” and “SecDevOps” are the next evolution of silo-busting, where these traditionally separate silos need to come together and really collaborate and learn from each other.

Otherwise, we’ll continue to see these silos as a drag on momentum, slowing things down and stifling innovation. From a security perspective, this is critical to maintaining an organization’s security posture while still striving for operational efficiency and scale.

If you could give one piece of advice to businesses wanting to deploy containers as securely as possible, what would that be?

Embrace DevSecOps. Note that DevSecOps includes shifting security left, by adding security gates to your CI/CD pipeline (DevSec), but also includes using defense-in-depth security capabilities in your production environment (SecOps).

DevSec gates should include image vulnerability analysis to look for known vulnerabilities, application configuration analysis to look for excess privileges, and using signatures to verify the provenance of external content. SecOps capabilities should include admission controllers to prevent deployment of privileged containers, SELinux to prevent or mitigate container escapes to the filesystem, runtime analysis to look for anomalous behavior, exec within a container, crypto mining and to take action on the discovery of such behavior, as well as full-stack auditing.

It’s also really important to have stacks that work well together. Because a container is a subset of an operating system running on top of another operating system (the container platform), having a mismatch in kernel expectations or things like glibc could be problematic, where code inside the container was built against a particular kernel version, and so expects things in a certain way, but the host kernel is entirely different. We view running the same major version of an operating system inside a container as the outside operating system as the most reliable and secure way to run containers.

What is Red Hat doing unlike any other server operating system for cloud/container security?

Red Hat takes the full-stack approach. We don’t just focus on a container management platform, we focus on the surrounding ecosystem as well. We do this with Red Hat Enterprise Linux (RHEL) CoreOS as our container-optimized operating system, as an integrated part of Red Hat OpenShift, our container-as-a-service / application platform. This affords our customers the assurance that these separate parts were meant to work together as one cohesive whole, and the security technologies we’ve built into RHEL for years are a substantial part of the overall security of how containers are managed and deployed.

What’s the future of cloud native development look like?

Just as more and more infrastructure is moving to the cloud, software developers building cloud native solutions will benefit from adopting software-as-a-service development tools. Development environments that can be accessed from a web browser are a better fit for the emerging work-from-anywhere world. These environments also provide great visibility at a lower cost for organizations.

Applying policy to a centralized service that still allows for customization is easier than applying policy to a wide range of independent pipelines managed by individual teams. That said, developers still need the flexibility to use the tools best suited to the work they are doing.

What’s the first thing you do to a server operating system to harden it?

Apply any outstanding patches and remove all unnecessary packages to reduce the potential attack surface and turn off any unneeded services. Too many things get enabled out of the box, and by the time you install, there are already packages waiting to be updated. Additionally, make sure you go after how it is configured and lock things down. More often than not, you will be compromised through poor configuration than an actual vulnerability in software.

Consider using benchmarks like CIS or other good practices as a baseline for how to sensibly and securely configure your operating system. Most organizations don’t have a single server they’re worried about, so consider automation tools like Red Hat Ansible Automation Platform to orchestrate these configurations across multiple servers and, perhaps as importantly, use these tools to ensure the configurations stay in a good, secure state.

How can small to medium-sized businesses gain the levels of security found in the enterprise?

Focus on securing the data, so look for controls around the data. Then focus on access control, both to the data and to super admins. A compromise in either of these can lead to loss of knowledge and market, so there is a high return on investment for small to medium businesses in focusing on these areas.

What’s the best thing container developers can do to ensure they’re building off a solid and secure foundation?

One of the values of containerized software is that the system dependencies are built into the container, making the container more portable. The question then becomes: Where do the system dependencies come from, and how do I know those libraries can be trusted? For this reason, it’s important to use trusted software for your base image, software with a clear provenance that is maintained and regularly updated. And be sure you keep up with the updates.

From your perspective, what’s the answer to securing the supply chain?

From a practical perspective, monitoring your development and CI/CD systems. With good monitoring in place, you can determine whether inappropriate things are happening. Without it, no matter how much you patch, if someone gets in, you need to be able to detect it in order to respond.

All the other security hardening and proactive measures are obviously needed, but without the ability to detect, you hamper your ability to respond quickly, fix any issues you didn’t know you had or eject anyone in your network that doesn’t belong there. There are many other considerations, like proving the provenance and pedigree of the code being used, but monitoring and detecting potential threats to the supply chain is foundational.

What is the coolest piece of cloud native technology coming out of Red Hat these days?

We’re really excited by the speed at which the sigstore project is evolving. Provenance is an important element of supply chain security. Signed content and verification of signatures have long been promoted as a way to validate provenance and ensure that content has not been tampered with in transit.

But adding signing to the pipeline process is not at all easy to do right, especially considering the challenge of securing the use of private signing keys. The sigstore project, launched under the Linux Foundation, is working to address these challenges, and we’re really happy to be contributing to that effort.

Should businesses be striving for full-blown automation, or should they keep a layer of human intervention involved in the DevOps process?

Businesses should be investing in automation whenever security policies can be automated. Humans still need to define and review policies and adjust them when necessary, but automation is needed to establish and maintain a strong security posture. Automation not only enables consistent configuration, but it also helps with hardening and applying other security policies.

Managing everything as code with a GitOps-like approach means your infrastructure, application platform, applications and security policies are more resilient. You should be designing your IT infrastructure so that recovery from an incident can, ideally, be managed through asset redeployment. This is becoming a new best practice because it acts as a forcing function to make sure that strong security policies to the code are deployed early.