Data Quality Assessment

DQ score calculations are not as straightforward as one might think. Typically, there is a DQ rule calculation score, which is determined as the number of records that passed the rule divided by the total number of records. However, almost everyone wants to generate some kind of aggregated score for a dataset, coming up with a single number to measure data quality. This is where it gets interesting. Some DQ tools offer a DQ score calculated as an average of all DQ rule scores. This is just a number and often lacks meaningful interpretation. Other tools provide more sophisticated score calculations at the record level and subject level. These scores are more insightful: The record-level score shows the number of error-free records, while the subject-level score shows the number of error-free subjects. Subjects are high-level entities whose data is being assessed, such as customers or accounts (or loans, in the example here). Interestingly, different calculation methods can yield different results! Which method is the best? It’s the one that is understandable to the people involved in reviewing the DQ assessment results. Personally, I prefer calculating all kinds of scores and organizing them into a neat DQ scorecard. Examining scores from various perspectives gives me valuable information that I can use to draw actionable conclusions and perform data quality improvement exercises. What methods do you use?

If You Can't Trust Your Data, You Can't Trust Your Decisions. 𝗕𝗮𝗱 𝗱𝗮𝘁𝗮 𝗶𝘀 𝗲𝘃𝗲𝗿𝘆𝘄𝗵𝗲𝗿𝗲—𝗮𝗻𝗱 𝗶𝘁'𝘀 𝗰𝗼𝘀𝘁𝗹𝘆. Yet, many businesses don't realise the damage until too late. 🔴 𝗙𝗹𝗮𝘄𝗲𝗱 𝗳𝗶𝗻𝗮𝗻𝗰𝗶𝗮𝗹 𝗿𝗲𝗽𝗼𝗿𝘁𝘀? Expect dire forecasts and wasted budgets. 🔴 𝗗𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗲 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿 𝗿𝗲𝗰𝗼𝗿𝗱𝘀? Say goodbye to personalisation and marketing ROI. 🔴 𝗜𝗻𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲 𝘀𝘂𝗽𝗽𝗹𝘆 𝗰𝗵𝗮𝗶𝗻 𝗱𝗮𝘁𝗮? Prepare for delays, inefficiencies, and lost revenue. 𝘗𝘰𝘰𝘳 𝘥𝘢𝘵𝘢 𝘲𝘶𝘢𝘭𝘪𝘵𝘺 𝘪𝘴𝘯'𝘵 𝘫𝘶𝘴𝘵 𝘢𝘯 𝘐𝘛 𝘪𝘴𝘴𝘶𝘦—𝘪𝘵'𝘴 𝘢 𝘣𝘶𝘴𝘪𝘯𝘦𝘴𝘴 𝘱𝘳𝘰𝘣𝘭𝘦𝘮. ❯ 𝑻𝒉𝒆 𝑺𝒊𝒙 𝑫𝒊𝒎𝒆𝒏𝒔𝒊𝒐𝒏𝒔 𝒐𝒇 𝑫𝒂𝒕𝒂 𝑸𝒖𝒂𝒍𝒊𝒕𝒚 To drive real impact, businesses must ensure their data is: ✓ 𝗔𝗰𝗰𝘂𝗿𝗮𝘁𝗲 – Reflects reality to prevent bad decisions. ✓ 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗲 – No missing values that disrupt operations. ✓ 𝗖𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝘁 – Uniform across systems for reliable insights. ✓ 𝗧𝗶𝗺𝗲𝗹𝘆 – Up to date when you need it most. ✓ 𝗩𝗮𝗹𝗶𝗱 – Follows required formats, reducing compliance risks. ✓ 𝗨𝗻𝗶𝗾𝘂𝗲 – No duplicates or redundant records that waste resources. ❯ 𝑯𝒐𝒘 𝒕𝒐 𝑻𝒖𝒓𝒏 𝑫𝒂𝒕𝒂 𝑸𝒖𝒂𝒍𝒊𝒕𝒚 𝒊𝒏𝒕𝒐 𝒂 𝑪𝒐𝒎𝒑𝒆𝒕𝒊𝒕𝒊𝒗𝒆 𝑨𝒅𝒗𝒂𝒏𝒕𝒂𝒈𝒆 Rather than fixing insufficient data after the fact, organisations must 𝗽𝗿𝗲𝘃𝗲𝗻𝘁 it: ✓ 𝗠𝗮𝗸𝗲 𝗘𝘃𝗲𝗿𝘆 𝗧𝗲𝗮𝗺 𝗔𝗰𝗰𝗼𝘂𝗻𝘁𝗮𝗯𝗹𝗲 – Data quality isn't just IT's job. ✓ 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 – Proactive monitoring and correction reduce costly errors. ✓ 𝗣𝗿𝗶𝗼𝗿𝗶𝘁𝗶𝘀𝗲 𝗗𝗮𝘁𝗮 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 – Identify issues before they impact operations. ✓ 𝗧𝗶𝗲 𝗗𝗮𝘁𝗮 𝘁𝗼 𝗕𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗢𝘂𝘁𝗰𝗼𝗺𝗲𝘀 – Measure the impact on revenue, cost, and risk. ✓ 𝗘𝗺𝗯𝗲𝗱 𝗮 𝗖𝘂𝗹𝘁𝘂𝗿𝗲 𝗼𝗳 𝗗𝗮𝘁𝗮 𝗘𝘅𝗰𝗲𝗹𝗹𝗲𝗻𝗰𝗲 – Treat quality as a mindset, not a project. ❯ 𝑯𝒐𝒘 𝑫𝒐 𝒀𝒐𝒖 𝑴𝒆𝒂𝒔𝒖𝒓𝒆 𝑺𝒖𝒄𝒄𝒆𝒔𝒔? The true test of data quality lies in outcomes: ✓ 𝗙𝗲𝘄𝗲𝗿 𝗲𝗿𝗿𝗼𝗿𝘀 → Higher operational efficiency ✓ 𝗙𝗮𝘀𝘁𝗲𝗿 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗺𝗮𝗸𝗶𝗻𝗴 → Reduced delays and disruptions ✓ 𝗟𝗼𝘄𝗲𝗿 𝗰𝗼𝘀𝘁𝘀 → Savings from automated data quality checks ✓ 𝗛𝗮𝗽𝗽𝗶𝗲𝗿 𝗰𝘂𝘀𝘁𝗼𝗺𝗲𝗿𝘀 → Higher CSAT & NPS scores ✓ 𝗦𝘁𝗿𝗼𝗻𝗴𝗲𝗿 𝗰𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲 → Lower regulatory risks 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗱𝗮𝘁𝗮 𝗱𝗿𝗶𝘃𝗲𝘀 𝗯𝗲𝘁𝘁𝗲𝗿 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀. 𝗣𝗼𝗼𝗿 𝗱𝗮𝘁𝗮 𝗱𝗲𝘀𝘁𝗿𝗼𝘆𝘀 𝘁𝗵𝗲𝗺.

I’m obsessed with one truth: 𝗱𝗮𝘁𝗮 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 is AI’s make-or-break. And it's not that simple to get right ⬇️ ⬇️ ⬇️ Gartner estimates an average organization pays $12.9M in annual losses due to low data quality. AI and Data Engineers know the stakes. Bad data wastes time, breaks trust, and kills potential. Thinking through and implementing a Data Quality Framework helps turn chaos into precision. Here’s why it’s non-negotiable and how to design one. 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗗𝗿𝗶𝘃𝗲𝘀 𝗔𝗜 AI’s potential hinges on data integrity. Substandard data leads to flawed predictions, biased models, and eroded trust. ⚡️ Inaccurate data undermines AI, like a healthcare model misdiagnosing due to incomplete records.   ⚡️ Engineers lose their time with short-term fixes instead of driving innovation.   ⚡️ Missing or duplicated data fuels bias, damaging credibility and outcomes. 𝗧𝗵𝗲 𝗣𝗼𝘄𝗲𝗿 𝗼𝗳 𝗮 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 A data quality framework ensures your data is AI-ready by defining standards, enforcing rigor, and sustaining reliability. Without it, you’re risking your money and time. Core dimensions:   💡 𝗖𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆: Uniform data across systems, like standardized formats.   💡 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆: Data reflecting reality, like verified addresses.   💡 𝗩𝗮𝗹𝗶𝗱𝗶𝘁𝘆: Data adhering to rules, like positive quantities.   💡 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗻𝗲𝘀𝘀: No missing fields, like full transaction records.   💡 𝗧𝗶𝗺𝗲𝗹𝗶𝗻𝗲𝘀𝘀: Current data for real-time applications.   💡 𝗨𝗻𝗶𝗾𝘂𝗲𝗻𝗲𝘀𝘀: No duplicates to distort insights. It's not just a theoretical concept in a vacuum. It's a practical solution you can implement. For example, Databricks Data Quality Framework (link in the comments, kudos to the team Denny Lee Jules Damji Rahul Potharaju), for example, leverages these dimensions, using Delta Live Tables for automated checks (e.g., detecting null values) and Lakehouse Monitoring for real-time metrics. But any robust framework (custom or tool-based) must align with these principles to succeed. 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲, 𝗕𝘂𝘁 𝗛𝘂𝗺𝗮𝗻 𝗢𝘃𝗲𝗿𝘀𝗶𝗴𝗵𝘁 𝗜𝘀 𝗘𝘃𝗲𝗿𝘆𝘁𝗵𝗶𝗻𝗴 Automation accelerates, but human oversight ensures excellence. Tools can flag issues like missing fields or duplicates in real time, saving countless hours. Yet, automation alone isn’t enough—human input and oversight are critical. A framework without human accountability risks blind spots. 𝗛𝗼𝘄 𝘁𝗼 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗮 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 ✅ Set standards, identify key dimensions for your AI (e.g., completeness for analytics). Define rules, like “no null customer IDs.”   ✅ Automate enforcement, embed checks in pipelines using tools.   ✅ Monitor continuously, track metrics like error rates with dashboards. Databricks’ Lakehouse Monitoring is one option, adapt to your stack.   ✅ Lead with oversight, assign a team to review metrics, refine rules, and ensure human judgment. #DataQuality #AI #DataEngineering #AIEngineering

It took me 10 years to learn about the different types of data quality checks; I'll teach it to you in 5 minutes: 1. Check table constraints The goal is to ensure your table's structure is what you expect: * Uniqueness * Not null * Enum check * Referential integrity Ensuring the table's constraints is an excellent way to cover your data quality base. 2. Check business criteria Work with the subject matter expert to understand what data users check for: * Min/Max permitted value * Order of events check * Data format check, e.g., check for the presence of the '$' symbol Business criteria catch data quality issues specific to your data/business. 3. Table schema checks Schema checks are to ensure that no inadvertent schema changes happened * Using incorrect transformation function leading to different data type * Upstream schema changes 4. Anomaly detection Metrics change over time; ensure it's not due to a bug. * Check percentage change of metrics over time * Use simple percentage change across runs * Use standard deviation checks to ensure values are within the "normal" range Detecting value deviations over time is critical for business metrics (revenue, etc.) 5. Data distribution checks Ensure your data size remains similar over time. * Ensure the row counts remain similar across days * Ensure critical segments of data remain similar in size over time Distribution checks ensure you get all the correct dates due to faulty joins/filters. 6. Reconciliation checks Check that your output has the same number of entities as your input. * Check that your output didn't lose data due to buggy code 7. Audit logs Log the number of rows input and output for each "transformation step" in your pipeline. * Having a log of the number of rows going in & coming out is crucial for debugging * Audit logs can also help you answer business questions Debugging data questions? Look at the audit log to see where data duplication/dropping happens. DQ warning levels: Make sure that your data quality checks are tagged with appropriate warning levels (e.g., INFO, DEBUG, WARN, ERROR, etc.). Based on the criticality of the check, you can block the pipeline. Get started with the business and constraint checks, adding more only as needed. Before you know it, your data quality will skyrocket! Good Luck! - Like this thread? Read about they types of data quality checks in detail here 👇 https://lnkd.in/eBdmNbKE Please let me know what you think in the comments below. Also, follow me for more actionable data content. #data #dataengineering #dataquality

Many teams overlook critical data issues and, in turn, waste precious time tweaking hyper-parameters and adjusting model architectures that don't address the root cause. Hidden problems within datasets are often the silent saboteurs, undermining model performance. To counter these inefficiencies, a systematic data-centric approach is needed. By systematically identifying quality issues, you can shift from guessing what's wrong with your data to taking informed, strategic actions. Creating a continuous feedback loop between your dataset and your model performance allows you to spend more time analyzing your data. This proactive approach helps detect and correct problems before they escalate into significant model failures. Here's a comprehensive four-step data quality feedback loop that you can adopt: Step One: Understand Your Model's Struggles Start by identifying where your model encounters challenges. Focus on hard samples in your dataset that consistently lead to errors. Step Two: Interpret Evaluation Results Analyze your evaluation results to discover patterns in errors and weaknesses in model performance. This step is vital for understanding where model improvement is most needed. Step Three: Identify Data Quality Issues Examine your data closely for quality issues such as labeling errors, class imbalances, and other biases influencing model performance. Step Four: Enhance Your Dataset Based on the insights gained from your exploration, begin cleaning, correcting, and enhancing your dataset. This improvement process is crucial for refining your model's accuracy and reliability. Further Learning: Dive Deeper into Data-Centric AI For those eager to delve deeper into this systematic approach, my Coursera course offers an opportunity to get hands-on with data-centric visual AI. You can audit the course for free and learn my process for building and curating better datasets. There's a link in the comments below—check it out and start transforming your data evaluation and improvement processes today. By adopting these steps and focusing on data quality, you can unlock your models' full potential and ensure they perform at their best. Remember, your model's power rests not just in its architecture but also in the quality of the data it learns from. #data #deeplearning #computervision #artificialintelligence

Data-quality “dimensions” like completeness, accuracy, timeliness, consistency, etc. come from management theory. They’re useful for audits and KPIs, but they don’t help much when you sit down to implement tests in a pipeline. Why? Engineers and analysts usually write checks only after a concrete failure: • Broken joins → foreign-key mismatch, unexpected NULLs  • Wrong revenue → aggregation logic changed, currency drift  • Missing records → late-arriving files, partition gaps  • Silent drops → schema evolution not propagated Notice none of those map cleanly to a single “dimension.” Each failure touches several at once. Instead we can try classifying checks by the failure they prevent and the action they trigger: My friend and mentor Malcolm created this overview of high-level check types that can be used to build test cases, but more importantly, to classify data quality issues with. How are you using Data Quality Dimensions in practice?

Leadership want a silver bullet to improving data quality, but that doesn’t exist   Below, I list out how you can start to think about data quality with a holistic and logical approach   First, 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝 the lay of the land, including the technology landscape, pain points, and what data is there to fix. This includes: 💻 Data Technology & Tooling Audit/Strategy – Outlining what different tools do within the data journey and align that with data quality needs 🛠️ Root Cause Analysis – A systematic process helps teams understand why data issues occur and enable targeted interventions that address more than just the symptom 🏆 Critical & Master Data Assets – Help focus efforts and resources on the most impactful data   Next, 𝐒𝐭𝐚𝐧𝐝𝐚𝐫𝐝𝐢𝐬𝐞 what data quality means within the organization and have a strategy to tackle these fixes in a proactive (not reactive) way. This includes: 🎯 Data Governance Strategy – Understand how the organisation works with and governs data (including who owns it) 📝 Setting Data Quality Standards – Establish those clear and measurable criteria for data quality to serve as a benchmark for all people across the organisation 📑 Data Contracts – Set clear expectations and responsibilities between the downstream and upstream groups of data users   Finally, 𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭 tools, technologies and approaches to combat data quality issues (but don’t skip to this step without doing the others) ⚙️ Data Catalogue & Lineage Tooling – Allow users to search datasets, understand its content, provide access, define ownership, and construct the flow of data assets from source to consumption 🛑 Data Quality Gates – Define checkpoints at various stages of the data platform (usually contained within pipelines) that validate data against predefined criteria before it proceeds further 👀 Data Observability Tooling – Monitor data health metrics to detect, diagnose, and resolve data quality issues in real-time to reduce data downtime and improve visibility into issues   There are other things you can do as well, but the point is to think about these things holistically and in order of implementation.   Check out my article last week (link in comments) about defining data quality issues and stay tuned this week for a lot more on each of these approaches   #dataecosystem #dataquality #newsletter #datastrategy #dylandecodes

Not all data warehouses are designed and implemented equally. Here are 5 of my favorite techniques for ensuring data quality. 1. Validate sums and counts between loads and transformations to capture discrepancies. 2. Confirm that the files were delivered at the agreed-upon time. 3. Compare the latest file size to the previous files to check for anomalies. 4. Identify any changes in file formats to act quickly. 5. Check for metadata & dimension errors to troubleshoot variances. Yes, a lot of these techniques are focused on files. Files tend to cause more issues than sourcing data directly from APIs or database tables. These checks have caught 95% of potential errors in our clients’ data warehousing solutions. And have allowed the data teams to troubleshoot them before the business users ever knew there was an issue. It also significantly reduced their troubleshooting times because these data quality processes pointed them to where the issues were. The good news is that these techniques can be applied as enhancements to an existing implementation. It pays to spend the time and money on sound data quality checks to keep the business running smoothly. What are some of your go-to data quality techniques? #dataquality #data #analytics #snowflake

Data Quality is the foundation to build a robust data strategy over Data Quantity. Robust data quality practices helps the entire analytics house from collapsing when business stakeholders need trust worthy insights Data Quantity:- ➖Data Quantity is intrinsic and keeps growing as the business grows. ➖It's always quantifiable in Kbs, Mbs, Gbs, Tbs, Pbs and so on. ➖It's even scaling more with use every second, but can be expensive. Data Quality:- ➖Data can be small in size but should be quality data that can help us get some insights. ➖Data quality assesses certain standards of data and surpasses a lot of data measures. ➖It helps to gather quality input to various models and solutions. 🔍Exploring Data Quality Frameworks is always helpful but leveraging the data quality fundamentals is about smart, strategic checks that catch 90% of issues before they become headaches: 1. Table Constraints Are Your First Defense:- Think of these like security guards for your data. Enforce uniqueness, block nulls, prevent garbage in, garbage out. 2. Business Context Is King:- Data doesn't speak for itself. Talk to domain experts. Understanding business logic is more powerful than any algorithm. 3. Schema Integrity = Data Health:- Your schema is like the DNA of your data pipeline. One mutation can break everything. Monitor it religiously. 4. Anomaly Detection: Your Early Warning System:- Unexpected metric shifts? That's not a bug, that's a feature waiting to be investigated. Standard deviations are your best friend. 5. Distribution Matters:- Consistent row counts and segment sizes aren't boring—they're beautiful. Sudden changes scream "investigate me!" 6. Reconciliation: No Data Left Behind:- Every row counts. Ensure what goes in comes out transformed, not lost. 7. Audit Logs: Your Data's Biography:- Transparency isn't just a buzzword. Track every transformation, every step. 💡 Pro Tip: Start small. Master business checks first. Scale up strategically. If you want to scale the aspects of data quality, leverage these "Top 10 Data Quality Tools" crafted by Deepak Bhardwaj. Are you too obsessed with clean data? Drop a comment, and share your go-to quality check or tool you prefer using! #Data #Engineering #DataQuality #dataanalytics #bigdata