GitLab sponsored this podcast.
Data engineering is 10 years behind the software engineering world and, in many ways, remains mired in “spreadsheet land,” Danielle Morrill, general manager for the GitLab DataOps-focused startup Meltano, said. As a way to make up for the lack of viable data engineering processes, Meltano can meet the needs of organizations without the in-house expertise of software developers who can invent a tailored solution from scratch. “A lot of people are great with spreadsheets and can’t write code or think they can’t write code, but conceptually, they already are,” Morrill said. “So, I think the roadmap [for Meltano] is very much about bringing them along.”
In this episode of InApps Technology Makers podcast recording during the GitLab Commit conference in Brooklyn, New York in September, Morrill described how Meltano casts as wide net as an open source alternative for data modeling, analysis and management.
GitLab’s Meltano, a Data Pipeline That Uses Git as the Source of Truth
Also available on Apple Podcasts, Google Podcasts, Overcast, PlayerFM, Pocket Casts, Spotify, Stitcher, TuneIn
Listen to more episodes and subscribe on Simplecast | Fireside.fm | Pocket Casts | Stitcher | Apple Podcasts | Overcast | Spotify | TuneIn
“Similar to GitLab being a pipeline tool for the developer experience, we’re a pipeline tool for the data analyst and engineer experience,” Morrill said. “We’re tying together best-in-class open source tools so that you can go from data all the way to a dashboard, without having to kind of pick and choose your tools and set them all up separately.”
While the “moment of truth” with GitOps is Git, the moment of truth with Meltano is the “dataset that you get after the transforms have been run,” Morrill said. “We definitely think people should be using version control and data engineering but that’s actually not what’s happening quite yet,” Morrill said.
However, Git should also be used for data management as well, Morrill said. “We want you to use Git,” Morrill said. “That is the direction we think things should go.”
Morrill said her experience as co-founder of Mattermark helped her to lay the groundwork for Meltano after learning first hand how difficult it is to build data management solutions in 2013. “We had to roll our own,” Morrill said. “When Macklemore got started in 2013, the tech stack for data engineering was much less developed. So, I witnessed a lot of the pain then and I think we have an opportunity to simplify that.”
Typically, an organization might require advanced data-management dashboard and engineering capabilities “but we can’t shell out $100,000 to get a license for either one expensive tool or a lot of times several expensive tools,” Morrill said. “So, we’re really doing a bottom-up approach of approaching SMEs and small tech companies, saying ‘look, you have all this data that’s yours and you shouldn’t need to spend a fortune to actually understand what you have,’” Morrill said. “And so that’s the scenario that we are seeing the most.”
Meltano 1.0 will launch “really soon” when “we can start telling people can build things on this and we’re going to support it, Morrill said. “We needed to make it so that if you’re not an engineer, this is still a friendly tool and will pull you into engineering through the tools,” Morrill said.
Feature image via Pixabay.
