“State is hard. That’s the bottom line,” explained Sacha Labourey, chief executive officer of CloudBees. “I don’t think managing state in Kubernetes is necessarily harder than in other environments.”
Is handling state in Kubernetes really difficult? Or is it, as we’ve already seen with developer experience and Kubernetes, more of a skills gap than a technological problem?
“Kubernetes is a very engineering-led platform,” said Karena Angell, principal product marketing manager for OpenShift Container Storage at Red Hat. “It looks difficult from the outside because it’s very technical. So there is this perception gap that storage is really hard in Kubernetes.”
It’s All So New
“Enterprise IT has existed in more or less its modern form for the past 50 years,” said Michael Ferranti, vice president of marketing at Portworx. “For 48 of those years, we have not run databases on Kubernetes.”
In a legacy world, database administrators would spend years — decades — becoming experts in one specific database. They would fly out to the Oracle or Microsoft conference every year. A senior database administrator would likely have 15 years of experience or more. Kubernetes was first released in June of 2014. No one has 15 years of experience running databases on Kubernetes, particularly in an enterprise setting with customer-facing applications with security, compliance and SLA requirements.
Organizationally, the legacy system also made it harder for individuals or organizations to switch technologies. Oracle database admins have a heap of certifications that are valid for Oracle databases only — while there’s obviously some skills transfer when moving to MongoDB or PostgreSQL, but much of the knowledge required to become an expert database admin doesn’t transfer between vendors.
Nonetheless, Ferranti says, there are ways Kubernetes makes things easier. Instead of building deep expertise in MongoDB, Postrgres, Kafka and other databases, you just need to learn how to use Kubernetes and it will allow you to administer a variety of database platforms.
“Kubernetes is intended to provide a Goldilocks level of abstraction,” said Craig McLuckie, one of the original creators of Kubernetes, who is now vice president of products and strategy, modern apps platform business unit at VMware. “The goal was to have something that is low level enough that you can run anything but high level enough that it hides away the specific details of your infrastructure.”
This doesn’t mean that all the tools are in place that one might expect from an enterprise database platform. “The whole ecosystem is growing and maturing,” Angell said. “So maybe if you’re coming from an enterprise platform and you don’t see something that you expect to see.”
The Challenges
Aside from the experience gap, there are some real challenges when handling state in Kubernetes. First of all, if a stateful application isn’t architected in the right way from the very beginning, success can be elusive — everyone responsible for each layer of the stack has to make the right choices to ensure that state persists.
There’s also the fact that state in some ways goes against the way Kubernetes sees the world. “The intent behind Kubernetes is to be able to stitch together a large number of commodity machines and treat them as a destination for distributed apps, where you’re no longer bound to a specific instance,” McLuckie said. “If you have state that’s locally persistent, that represents a challenge to the metaphor.”
There’s also operations to worry about. “A lot of stateful workloads have very challenging operational behaviors,” McLuckie said. “If you’re upgrading a database that might be in multiple instances, you have to manage coordination during the upgrade process to make sure you don’t have interruption of service.”
Another issue is scaling, which can be essentially challenging in a multi or hybrid cloud environment.
Avoiding the Pitfalls
Here’s one thing everyone interview agrees on: It’s entirely possible to successfully handle state in Kubernetes, but doing so skillfully requires understanding how Kubernetes views the world and working with it, not against it. According to Angell, the key is to make sure to think through the architecture correctly at the beginning — and be aware that the best architecture for a stateful application on Kubernetes is not the same as running a legacy stateful app.
“A lot of people will try to change as few things as possible when they bring a service on to Kubernetes,” agreed Ferranti. “I think that is one of the biggest mistakes people make, taking something that works outside of Kubernetes and assuming it will work the same in Kubernetes.”
“I’ve seen a lot of success with customers running NoSQL and MySQL databases in Kubernetes; I feel confident recommending customers use Kubernetes for a pretty broad cross-section of stateful workloads,” McLuckie said. “The thing I do implore organizations to start thinking about is, ‘what additional storage assets should I bring in? How do I make them accessible to my developers in a natural way? Will doing so over time reduce the cost and complexity of my storage solutions?”
It’s also important to remain realistic about what Kubernetes can and cannot do. Just because Kubernetes is a very powerful platform that is great for many types of stateful workloads doesn’t mean it is appropriate in every case. If companies tie developers’ bonuses to getting all their workloads into Kubernetes, the developers might containerize services that should have been left alone — leading to unpleasant consequences latter.
The Kubernetes Skills Gap
Maybe it’s not state that’s hard — it’s Kubernetes. “Once you’ve learned Kubernetes, the incremental jump to running stateful services, if you have the proper tooling, is very minimal,” Ferranti says. “It’s maybe five to 20 percent. But the first 80 percent, getting comfortable with Kubernetes, is going to take teams 12 to 18 months.”
State isn’t the only challenge in Kubernetes. It’s often accompanied by an organizational shift, first of all. Secondly, you have to figure out how to do all sorts of things in a new way, often without a set of clear best practices. How do you set up an ingress controller? How to manage load balancing?
“Playing the guitar is really complicated,” Ferranti said. “Once you learn how to do it, you can make beautiful music, but getting there is more challenging than most people realize.”
Managing state on Kubernetes is the same, except it’s more like playing in an orchestra, because it requires everyone working on all the layers of the stack to get things right. To someone at the beginning of the Kubernetes journey, it might seem impossible, just like orchestras seem to work by magic to someone picking up an instrument for the first time.
CloudBees, MongoDB, Portworx, Red Hat and VMware are sponsors of InApps Technology.
