Most data quality platforms focus on data quality issues once they are already manifesting in production, which can have knock-on effects on the business side. For example, deployment of bad data could cause people to look at inaccurate dashboards and make poor business decisions before any data quality issues were discovered. Monitoring is important, but detecting and fixing data quality issues prior to using data in production is a better solution. It is faster, easier, and most cost-efficient to fix an issue before the damage is done.
Datafold, founded in 2020, is focused on helping developers and data engineers overcome production data quality issues. The company builds tools to address data quality issues during the development phase, through automation of data testing. Automation is critical; it’s simply not possible to cover any meaningful portion of data with manually written tests. The company says it’s the only player focused on the automation of testing data within the developer workflow.
School of Hard Knocks
Gleb Mezhanskiy, the founder and CEO of Datafold, said the challenges he faced as a data engineer at Lyft led to the foundation of Datafold. He explained how, in one case, a deployed hotfix compromised an entire data pipeline, corrupting dashboards and machine learning feature sets, bringing the day’s productivity to a standstill.
Such experiences provided Mezhanskiy with the inspiration to build tools that prevent this type of production data incident. Mezhanskiy said, “At Lyft, I realized the tools for data engineers that can help automate a lot of manual tasks around data testing, data management, data observability, and lineage can be really helpful.”
Shifting Left
Datafold recently announced a partnership with dbt Labs, makers of dbt, a leading, open source data transformation tool. The partnership brings the fundamentals of reliability engineering from software engineering, such as version control, staging environments, CI/CD, and reliable change management into the data community. It also takes those practices to the next level by providing automation of testing and column-level lineage, the company said.
With the new integration, Datafold and dbt together ensure every code change gets full test coverage, which tests the most important assertions or assumptions about data. They deliver column-level lineage, Data Diff (regression testing for ETL) for dbt, and shareable impact reports. This increases the productivity of data teams by helping them save time on manual testing and, therefore, keeps them focused more on building better products.
How It Works
The new integration can be embedded within the workflow simply by going to dbt Cloud, getting the API key and plugging it into Datafold. Users can then install the Github application for Datafold to get automation for testing that works well with dbt Cloud and Github. Integration with Github helps users see the impact of their changes on data before deploying to production. Datafold, when combined with Github, runs Data Diff as soon as a pull request is opened, so users can see how the change in the source code is going to affect the data. The effect is analyzed in terms of number of rows, schema, primary key, and column differences.
Users can build a Datafold app to see a detailed explanation of the impact of their desired change. The impact of changes can also be visualized and explored at the distribution level so, for example, a skew or feature drift can be prevented before it hampers machine learning model performance. Additionally, Datafold’s column-level lineage capability provides information on how data flows between columns. This enables users to see how a change made to an upstream table can affect a downstream table.
Overall, Datafold enables users to test data for issues that are hard to account for manually. The main goal of all the new integration is to prevent issues before a change takes effect. That makes pull requests an important starting point since that is when developers collaborate to make changes.
Discovering the Future
Datafold has been automating the change management process with regression testing, making it possible to detect data quality issues using column-level lineage, which gives them an advantage in tracing and detecting anomalies while also showing where they are located.
Data lineage, and the access to so much valuable metadata, also puts Datafold in a unique position to improve data discovery, provide information on how data is related, and exchange metadata between tools to enable a cohesive experience. Datafold plans to continue building more automation on top of what it already offers. While the company is not intending its platform to become a full-fledged data discovery tool, Datafold does want it to be part of the data workflow.
Better Together
The data product ecosystem is still fragmented, and arguably a platform like Datafold’s adds some intricacy to the stack. However, companies are progressively joining forces, and such partnerships do simplify things, mitigating the complexity of adopting solutions from multiple vendors, and helping overcome integration issues between different products.
Datafold’s new integration has a laudable goal: to improve the data engineering workflow and help companies scale their analytics more reliably
