Henrik Johansson, senior developer at Eniro, gives a glowing review of the ScyllaDB database system for its part of a microservice-based pipeline used at the Swedish search and directory assistance company where he works.
Eniro initially used Redis for what seemed like a simple task: relate a large set of transient identifiers A to another large set of other identifiers B. Its previous experience with MongoDB and PostgreSQL had brought problems with scaling though, and though Redis was fast and stable for a time with this project, latency eventually became a problem.
A combination of the programming language Go, the web framework Echo, the open source messaging system NATS and ScyllaDB proved fast, with low latency and requiring no extra tuning, he explained in a blog post.
“This brings an almost perverse satisfaction … that makes me doubt a large chunk of my previous experience using other stacks,” he said.
ScyllaDB is an open source NoSQL database designed to be a drop-in replacement for Apache Cassandra, which is the 8th most widely used database system, according to DB-Engines’ database popularity rankings. Written in C++ rather than Java, ScyllaDB touts that it can handle a million operations per second with less than a minute latency, providing a 10x performance improvement overall.
ScyllaDB co-founder Dor Laor maintains that Cassandra is grossly inefficient, but has its good points.
“We could have written an entire database with different APIs, but decided to be compatible with Cassandra because it was widely used. There are goodies in Cassandra that made it what is today — high availability and fault tolerance. It can do 1,000-node clusters, multi-data center, out-of-the-box fantastic replication, ecosystem with Spark. We kept all of that with our design, but made it way better,” he said.
Cassandra uses multiple threads of execution that try to grab locks and try to grab resources. The problem is that resources get busy and access to them in bounded, so they have to wait for each other or sometimes interfere with each other he said, resulting in even 20 percent inefficiency even with x86.
Sharding at the Core
Scylla uses two levels of sharding, or partitioning the database into smaller faster parts. Sharding is usually done by server. Cassandra uses with cluster-wide sharding, but Scylla uses sharding at the core level, which is transparent to the user.
“At the server, we divide the data range into independent cores. Each core owns its own subset of the data, its own CPU, its own memory and many times it has direct access to the I/O. So it’s completely independent, doesn’t need to lock and many other goodies,” he explained.
“In the Cassandra design, you have these threads, but the kernel owns resources. It does the scheduling, the memory management, the I/O. But the kernel doesn’t have any idea which threads will handle which packet. There could be an incoming packet for core 10, but the application thread runs on core 20. So it needs to do locking, it needs to do context switching… it’s a huge overhead. If your memory is allocated on the remote sockets, the access to it is twice as expensive.”
Scylla runs multiple engines, but it’s one process with multiple threads. Each thread runs the subset.
“Because of that, memory is always NUMA (non-uniform memory access)-friendly because each engine has a subset of the RAM and it’s NUMA-local. We bind the memory to that. We also bind the threads, so it’s very efficient,” he explained.
From KVM Founders
Scylla founders Laor and Avi Kivity were part of the team that created the KVM hypervisor, which Red Hat acquired in its purchase of the virtualization startup Qumranet.
Laor and Kivity left Red Hat to form Cloudius Systems in 2012. It created the cloud operating system OSv and the server-side application framework Seastar. In 2015, they rebranded the Israeli-based company as ScyllaDB. The company just raised $16 Million in Series B funding, raising its total investment to $25 million.
Scylla claims 10x performance improvement, but that varies according to the hardware used. One of the early testers, advertising tech firm Outbrain, reported a 6x boost, with Scylla handling 20 times more traffic than Cassandra while providing 3X latency.
The Alibaba Group, in its Pedis NoSQL data store project, rewrote an application with the Redis protocol using Scylla’s Seastar framework as a library and reported 8X better results. Gabriel Mizrahi, Chief Technology Officer and Chief Architect at Investing.com, says Scylla enabled the company to shrink its cluster size by half.
Developers Marc Alonso and Thomas Mouron, on the Octo Talks blog, described Scylla’s advantages this way:
“The advantage of having an application written in C ++ is actually to reduce CPU usage by avoiding the program to be loaded into a JVM. Scylla also provides a custom network management that minimizes resource by bypassing the Linux kernel. No system call is required to complete a network request. Everything happens in the userspace, limiting the number of expensive switches with the kernel space.
Another advantage with C++ is the ability to have a finer but more complex memory management. Indeed, in Java, the garbage collector takes care of regularly browsing the allocated memory to release the unused space. This step is extremely costly in terms of processor cycles and it can stop the application up to several seconds on large memories, it is also known as ‘stop the world.’”
Sometimes latency, such as at ad tech companies, is a bigger issue with clients than throughput, Laor said.
Part of the improved latency is by eliminating Java’s garbage collection. Typically NoSQL data stores consist of a Java Virtual Machine (JVM) that runs on top of Linux using a page cache and complex memory allocation strategies to “trick” the JVM garbage collector to avoid pauses.
And configuring Java in Cassandra is complex and really hard, Laor says. Scylla’s ability to self-tune provides another benefit, Kivity noted on Quora.
“With Cassandra, one must tune almost everything: the number of reader and writer threads, compaction and streaming throughput, the Garbage Collector, sizes of various caches. This is time-consuming, error-prone, and difficult in a production environment that can change. ScyllaDB self-tunes itself and divides the processor, network, and disk bandwidth dynamically among user reads, user writes, and internal tasks such as compaction and streaming,” he wrote.
Still Immature Technology
In that post, Eric Lubow, co-author of “Practical Cassandra,” lists Scylla’s downsides as being a small company with few productions implementations and lacking the massive community surrounding Cassandra. It’s a far less mature technology than Cassandra, he noted and complained that it doesn’t support counters, a feature coming in version 1.7, along with the Seastar TCP stack and user space.
Scylla released version 1.6 in February and plans to launch its enterprise edition this month.
Lubow maintains that merely copying Cassandra’s features has limited Scylla’s ability to innovate and truly differentiate itself. Laor says Scylla will have implemented all of Cassandra’s features by the end of Q2, and it will be able to focus on a wealth of other things, including adding multi-tenancy, that can be done with Scylla.
IBM Compose offers Scylla as a hosted service integrated with Bluemix cloud service. Samsung SDS, one of its investors, also uses it for Internet of Things and mobile use. Other users include Arista Networks, ad tech company AppNexus, video social network musical.ly, India’s taxi-booking service Ola.
