There can be no denying that containers are quickly becoming a cornerstone of today’s application development process, though the adoption of the technologies best used to store containers still lag behind. Container-centric object storage didn’t catch the eye of hosted container provider Joyent until late 2011, when it began development on its distributed object storage and integrated compute service, Manta.
Bryan Cantrill, Joyent’s chief technology officer, noted that “The lightbulb went on for us in late 2011. The big light is that containers don’t pertain merely to compute, and that if you could have this authoritative object store upon which you could safely spin up one container, that container could see the underlying object without being able to see other objects, without being able to trash the system, in its own fully isolated container.”
As the years went on, Joyent continued to build software and services such as its container-as-a-service platform Triton. These projects caught the attention of Samsung, which recently announced it would be acquiring Joyent, which should give Joyent additional flexibility and scalability.
In this 101st episode of InApps Analysts embedded below, we dive into the pitfalls of working with the massive amounts of data present to enterprises today, why Joyent built its facility called Thoth to automate its crash dump management, and how object computing at the source is crucial for getting the most out of one’s data when working at scale. InApps founder Alex Williams spoke with Cantrill to hear his thoughts on these topics and more.
#102: How Joyent Manta Aims to Simplify Object Storage at Scale
Listen to all TNS podcasts on Simplecast.
The interview can also be heard on YouTube.
Many organizations have created massive towers of data which could not be worked with, Cantrill noted. “We’ve got, now, data of such size and mass that it becomes immovable, effectively, which is fine if you want to leave that data and do the Amazon Glacier approach, where long-term cold data is likely sitting on a spindle that is powered off and is only there for disaster recovery. Then maybe you can use these more conventional kind of solutions,” Cantrill explained.
Joyent harnessed the power of distributed systems in addition to open-sourcing Manta, building it out on top of its Triton container technology, which was also made open source alongside Manta. Manta utilized Oracle’s ZFS to store objects until they were called up for compute.
In addition to bringing in the power of distributed systems for object storage, Joyent later realized that revising Manta’s crash automation was crucial for their development team. “We wrote a facility called Thoth, a system that we built on Manta. Thoth allows us to do automated analysis of our core and crash systems. When a new core dump comes into Manta, we can automatically analyze it and see if it matches patterns of various known bugs, and if it does we can mark it,” Cantrill said.
The key to working with large amounts of data at scale is to get to the source, Cantrill later explained. “You need a lot of data to explore some of these hypotheses and the challenges, and you don’t want to have to move data around to accommodate them. You want to be able to compute on an object where it lives.”
Joyent is a sponsor of InApps.
Feature image via the New Old Stock site.
