Lessons from a DevOps Failure – InApps 2022

“Five years ago I joined a company that was having problems,” said Patrick Debois, kicking off his talk at QConLondon earlier this month. Debois is now director of DevOps relations for security vendor Snyk.  In the previous job, at a now-defunct company, his role became overcoming bottlenecks, breaking down slips, and asking why.

MTTR is usually short for mean time to recovery but what about the mean time to reflect? This year’s QCon featured a whole track on when things go wrong, including Debois’ talk. It included topics about mental, environmental and privacy health. Debois’ own DevOps talk had him looking back and reflecting on the more known and lesser-known bottlenecks of DevOps — technology, suppliers, recruitment, docs, support, legal — and how to overcome them.

Is It a Tech Problem?

Debois started with the code and the people and processes surrounding it.

He said the company was doing git checkouts on production servers. There were no build pipelines. No testing. He said they could at that time get away with manual testing.

The DevOps team took ownership of the production environment. They changed for an ethos of you build it, you run it, you fix it if it breaks. And they started to set up and disseminate best practices.

“During a live television show, we deployed in production in two minutes because a producer on Slack said ‘Please fix this’.”

It was a simple fix of two conflicting percentages, making sure that on-screen the total can never be over 100 percent. But this rapid response helped build trust between the business and tech sides.

Is It a Suppliers Problem?

DevOps is not limited to internal operations.

“Instead of just looking inwards, we had to look outside for bottlenecks,” Debois said. For instance, the company was working with suppliers that created an external data source “so we had to make friends with our supplier.”

The DevOps team quickly learned to send suppliers feedback starting with the first experience with the product. Debois says suppliers are just as desperate for feedback as internal teams. And it’s really hard for them to get unbiased feedback.

“It isn’t about being serverless, it’s about being service-ful. We are relying on so many services that we need to be friends with them,” he said.

Is It a Recruitment Problem?

DevOps, like all of agile, is people over processes. So when you are transitioning to a DevOps way of working, the who matters better than the what.

Everyone on the growing startup was in charge of recruitment. So they attended a lot of meetups. And the company published articles on its website about “how life was as we felt it.”

“It wasn’t this is what we are hiring for — this is what life is the good and bad,” Debois clarified.

They tried to give a realistic view of what it’s like to be part of their team.

And then they looked for people who didn’t have the exact same technical skills as their current team. They looked for people with high potential. And candidates were required to spent time with the engineering team.

Even after all this, the recruitment process wasn’t perfect and not everyone was a perfect hire, but it made it more likely.

Is It a Sales Problem?

In many tech companies, sales is on another floor. But sales are the first like of customer defense and have made promises to your shared customers.

Just as much as a developer is focused on prioritizing their backlog, so are their clients. Sales needs to understand when features will arrive and be able to understand not only how much something costs but its inherent value for that cost.

Debois also said at a smaller company like his, sales has to be basically on-call, ready to respond to leads immediately.

So the company made sure sales felt authorized to customize contracts based on these building prioritizes.

Sales feedback is crucial for pricing too. Debois admitted that a lot of technical people share the habit of over-engineering, which raises the price.

“Because of the pricing structure, we had to build the simplest things possible,” he said.

The company needed to figure out CPC (cost per customer) and then map that to the minimal thing to achieve what the customer needs without raising prices.

Is It a Docs Problem?

Documentation is a serious selling point, Debois noted. Before asking for money, the company quickly realized it had to make it easy for prospects to test out and use the product first. This requires good documentation.

“It’s all about getting that feedback as soon as possible,” he said.

Documentation should happen after the sales as well. He pointed to how companies are more and more starting to write postmortem blogs and share their failures publicly. This is a good sign of a trustworthy company.  He realized that if the DevOps teams started writing blogposts, it could turn into marketing material and open up a lot of doors.

“Together we can set a correct tone in the company,” he said.

Is It a Legal Problem?

One benefit of the General Data Protection Regulation [GDPR] and California Consumer Privacy Act [CCPA] is that engineers are more mindful about security. IT now makes a habit of checking in with legal to make sure they are meeting regulations. This, in turn, opens up a conversation between the two departments and legal learns more about the tech they are backing.

Debois said this silo-smashing led to legal teams writing more custom contracts that showed they trusted the engineers. There were no longer templated contracts that promised: “you have to be 100% secure.”

Is It a Finance Problem?

The finance department is often working on antiquated systems. And it can be the last to embrace agile. But finance has to be part of product decision-making since otherwise, it becomes an impossible bottleneck to overcome. A transition to DevOps takes an ample investment in new tooling and potentially new staff. If the silo between IT and finance isn’t broken down, the latter can’t see the reasoning for the first.

Also opening up the lines of communication with finance means better reporting to show the bridge between business and tech. And faster billing, which benefits everyone.

What if It’s Not Enough?

Debois says DevOps is always about thinking about your next bottleneck. But sometimes there just isn’t a product-market fit.

On stage, you can hear the ghost of burnout as he gets to this part of the talk. He says the company was high on “Hopeium” — “five customers next quarter,” became the next and the next.

Debois now applies what he’s learned to DevSecOps at developer-focused open-source security tooling Snyk. Debois is also a founder of the DevOpsDays conference series.

And with a whole new section of the business, he found a whole new bottleneck.

“When you see Dev and Ops becoming DevOps and now with security, there’s this same level of distrust, the notion of control,” he said.

He ventured to say that, in a way, the most established tech and processes within the company have the most freedom, like “You can’t do this because it doesn’t work in our CI pipeline,” or has the freedom to say the other department can’t do that.

Security personnel thinks developers don’t know anything about security. Developers think sec doesn’t know code. But this distrust is built on nonsense.

“That’s not their job. But we judge them at the level of competence of our job not theirs,” Debois said.

Debois also learned from his failed venture that as an engineer he found himself risk-averse. In his role at Snyk, he is quick to reflect: “Are we doing this best practice because we don’t trust others or we don’t trust ourselves?”

He says it’s essential to be sincere in your actions.

He says DevSecOps has to be about those things working together. You aren’t looking to control any one of these pillars, but to educate.