All Tools
D
DataFreeOpen Source
DEEQU
Unit tests for data quality at scale
Apache-2.0
ABOUT
Large-scale data pipelines accumulate quality issues — missing values, schema violations, duplicate records, and distribution drift — that silently degrade downstream ML models and analytics. Manually inspecting petabyte-scale datasets is impractical. Deequ provides declarative data quality constraints that run as Spark jobs, computing comprehensive metrics (completeness, uniqueness, consistency, distribution) and surfacing violations as actionable reports. Its constraint verification model enables automated data quality testing analogous to unit tests for software, catching issues before they propagate to production.
INTEGRATION GUIDE
1. Define constraint-based data quality tests that run on every ETL pipeline execution
2. Monitor data distribution drift over time with automated metric computation and alerting
3. Validate schema conformance and detect missing or malformed fields in large datasets
4. Compute data quality reports for regulatory compliance and data governance requirements
5. Integrate data quality checks into Spark-based ML pipelines to prevent garbage-in-garbage-out
TAGS
data-qualitysparktestingvalidationdata-engineeringbig-data