Continuous Testing for (a) DevOps World – InApps Technology 2022

Tricentis sponsored this post.

Interest in continuous testing has been growing for five years now — yet the more we talk about it, the more polarized the discussion becomes. Complicating the conversation is the fact that Agile and DevOps are both driving the need for continuous testing, but both require distinctly different things from a quality perspective. For Agile, we need to test faster and earlier. But DevOps demands a more deep-seated shift.

Janye Groll, CEO of the DevOps Institute, and Wayne Ariola, chief marketing officer for Tricentis, discussed this topic at DevOps World last year. And the discussion is still relevant as we enter DevOps World 2019 this week. Exploring the role of continuous testing in a DevOps world, they discussed:

• Where testing is at odds with the core tenets of DevOps.
• Why we’re beyond “shift left.”
• The challenges of DevOps testing at the enterprise level.

Here is the video, followed by a summary of some of the highlights from the discussion:

DevOps Speed Makes the Testing ‘Speed Bump’ Hurt More

Ariola: If you increase the speed of anything, the speed bumps you hit hurt more. I couldn’t think of a better analogy for testing because testing usually stops the process or the quality cadence that you require in order to reduce the risk to your organization. It’s something that is a start/stop process today, and that has to evolve. If you’re a Global 2000 organization with some web or mobile frontend interfaces that drive engagement, as well as a set of backend systems that actually process those transactions, you need to smooth out that cadence to allow the testing process to be continuous — what we call Continuous Testing.

Groll:   That’s really interesting because “no handoffs” is one of the mantras of DevOps—and certainly “shift left” is the other. When we start to look at traditional IT, testing was always a downstream activity. The agile teams would produce something, it would go into some type of release process, and QA would be engaged as part of that. However, there was a barrier because it could be a two-week sprint that had a two-month testing cycle ahead of it downstream.

It’s Not All About ‘Shift Left’

Ariola: Testing, as you mentioned, was a time-boxed event that happened at the end of the cycle. As we matured it became its own sprint and then we started to scratch our heads and say, “Hey, should we really be having a testing sprint? And if we do, are we really done, done? Are we agile?” And people began to say, “No, it doesn’t feel right. Let’s move the process back.” But it’s not necessarily all about “shift left.” It involves shift left, shift right, shift everywhere. And I think this is where the actual idea of quality is beginning to morph, and we’re not there yet, by the way.

The Wikipedia definition of Continuous Testing says that it’s all about getting the right feedback, at the right time, to the right person — and that feedback needs to be actionable. The biggest problem we face with testing is that we went out and built these huge regression suites — whether they were unit tests or functional tests — and we ran them, and we ran them, and we ran them. And what did we ask? We asked if we were done testing. However, it’s the wrong question. Are you done? What does that matter?

The real question is: Does the release candidate have an acceptable level of risk? We’re pretty far away from that right now. And this is where the idea of just shift left isn’t going to solve the problem. We need a comprehensive delivery definition for quality, with every stakeholder is playing a role. As you know, in every organization, the makeup of who actually contributes to the release shifts a little bit.

Groll: I think you’re absolutely right. We’ve been expounding shift left, but it is shift everywhere. With everything is code… and quality…and value being part of this systems thinking, we really have to start to look at that. Testing has to be pervasive.

DevOps Testing at the Enterprise Level

Groll: One of the barriers to DevOps at the enterprise level is risk. When we start to look at risk at the enterprise level, they’re like, “Whoa, wait a minute. We have compliance issues. We have governance issues. We can’t take the chance of using a bunch of open source tools and suddenly start sending things into production minute, by minute, by minute.” I think that when we start to look at testing as an organizational capability more than just an individual competency, hopefully that solves some of that.

Ariola:  A couple of interesting points. I think it really shouldn’t matter what you’re using to achieve quality. If you’re able to get the right feedback — which is actionable —to the right person, you’re doing a really good job. Now the coordination of that is a totally different subject, right? We’re going to be using different tools that provide different speeds for different levels of automation, depending on the architectures that we have within the organization. Coming from the DevOps Institute, I’m sure that you see Global 2000 organizations whose CIO is coming in and telling you they have 12,500 applications that are running within the various incarnations of the system via merger, via growth, via legacy, right?

Groll: [nods]

Ariola:  And bringing that all together so you can actually remove what is known as this “process cadence mismatch” in testing is critical, and there’s ways to do that today. Tricentis focuses on this. We focus on trying to bring those big legacy systems in line with the more nimble systems so you can actually have a merged cadence to hit that release cycle. And this is huge, huge for digital transformation.

Groll: And huge particularly for those very large, very legacy complex organizations that were not born five years ago, 10 years ago, but have a long history there.

Ariola:  That’s another great point. Some of these organizations were five [people] in a garage, and they’re now 40 [people] in a garage. They were born with testing because they had to do it, right?

Groll: It’s in their DNA, right?

Ariola:  It’s in their DNA. Their infrastructure is componentized. They have a base of microservices on top of cloud native infrastructures, right? This gives them the flexibility to do that. But the Global 2000 organizations that are trying to move through this innovative cycle really need to reinvent software testing.

CloudBees is a sponsor of InApps Technology.

Feature image via Pixabay.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Tricentis.