Data quality: how to measure it and who owns each field

Data quality is the problem nobody budgets for and everybody pays for. It shows up in small things — a duplicated customer, a delivery note with no date, a tax ID with stray spaces — until one day the sales report doesn't match accounting, a rep calls the same contact twice, and the forecasting model produces a number nobody believes. Then comes the usual line: "the data is bad." Which isn't a diagnosis. It's a complaint.

This guide is for whoever has to turn that complaint into something manageable: measuring data quality with actual numbers, setting up checks that warn you before your customer does and, above all, deciding who owns each field. You don't need an expensive platform or a new team. You need scope, measurement and ownership.

What is data quality and which dimensions measure it?

A piece of data has quality if it serves the decision you're about to make with it. That's an uncomfortable definition because it's relative: the same customer master can be good enough for invoicing and terrible for campaign segmentation. That's why data quality isn't measured "in general" — it's measured per dimension, over a specific dataset.

The six dimensions we use, all computable in SQL without buying anything:

| Dimension | Question it answers | Example metric | |---|---|---| | Completeness | Are the fields I need there? | % of active customers with industry filled in | | Uniqueness | Are there duplicates? | Number of customers sharing a tax ID or email | | Validity | Does the value match its format or list? | % of postcodes that are five real digits | | Consistency | Does it reconcile across systems? | Gap between ERP sales and CRM sales | | Accuracy | Does it match reality? | % of addresses that don't bounce on delivery | | Timeliness | Is it current? | Average age of the last recorded contact |

Two practical warnings. Accuracy is the only one you can't measure by looking at the database alone: it requires checking against the real world — a return, a phone call, a physical stock count — so it's sampled, not monitored continuously. And timeliness is the most forgotten dimension: a value that was correct in 2023 and never reviewed is a wrong value today wearing the costume of a good one.

If you can't write the query that computes a dimension, you're not measuring it. You're having an opinion about it.

Why does data degrade in a mid-sized company?

Almost never for one dramatic reason. It degrades through the accumulation of reasonable decisions taken at different times:

• Free-text fields where a list belongs. The classic hand-typed "industry": "Hospitality", "hospitality", "HORECA", "Bar" and "-". Five values, one concept.
• Half-finished migrations. The ERP changed three years ago, the essentials were moved over, and the older history kept a different encoding nobody documented.
• Integrations that never reconcile. Two systems write to the same entity without agreeing on which one wins. It's the most expensive failure and the slowest to become visible; we cover it in integrating ERP and CRM without breaking things.
• Forms asking for what the user won't give. If a rep needs 14 mandatory fields to save an opportunity, nine of them will get garbage.
• Nobody owns the field. When a field belongs to everyone, it belongs to no one. No tool fixes that.
• Processes that punish doing it well. If the team's target is records loaded, records will be loaded — good or bad.

Put another way: poor data quality is usually a faithful reflection of a badly designed process. Cleaning without touching the process is mopping with the tap running; six months later the mess is back in exactly the same spot.

Which automated checks are worth setting up?

The rule is simple: put the check as close to the source as possible, and make it fail loudly. Bad data caught in the dashboard has already contaminated three decisions on its way there.

| Control point | What it checks | What happens on failure | |---|---|---| | Entry form or screen | Format, closed lists, real mandatory fields | Won't save, with a clear message | | Load or integration | Schema, types, duplicates against existing records | Record rejected, owner notified | | Analytical warehouse | Expected volume, nulls, ranges, orphan keys | Dataset flagged as unreliable for that day | | Report or dashboard | Reconciliation against the accounting source | Warning shown on the panel itself | | Periodic review | Sampled accuracy, fuzzy duplicates | Task assigned with a deadline |

A minimum set of automated checks that covers 80 % of the nasty surprises, ordered by effort:

1. Freshness: did yesterday's data arrive today? The most frequent failure and the easiest to detect. 2. Volume: is the row count within the range of recent weeks? A day with half the usual orders is normally a load problem, not a business one. 3. Nulls in critical fields: threshold per field, not a global one. 4. Key uniqueness: one order, one row. Sounds obvious; fails more often than you'd think after an integration. 5. Referential integrity: orders whose customer doesn't exist in the master. 6. Ranges and domains: negative amounts, future dates, statuses outside the list. 7. Cross-system reconciliation: revenue totals in the warehouse against the ERP, with a tolerance that's been defined and accepted.

Two pieces of advice that prevent the most common failure — nobody reading the alerts: start with few rules and generous tolerances (tightening is easy, recovering credibility after a month of false positives is not), and route every alert to a named person, never a shared inbox.

Who owns each field?

This is where a data quality plan is won or lost. Ownership has three roles, and none of them is "IT":

• Data owner (business). Decides what the field means, which values are valid, and what quality level is liveable. The customer master usually belongs to sales; the item master to product or purchasing; the supplier master to purchasing.
• Operational steward. Whoever enters and corrects it day to day. They need correction to be easy, and their manager to count that work as work.
• Technical custodian. Whoever implements the checks, maintains the alerts and fixes the pipes when they break. That's a data engineering profile, not the person who decides business rules.

With those three roles, the minimum artefact is a one-page table per critical dataset: field name, source system, owner, one-sentence definition, validity rules and agreed threshold. Five or six datasets are enough to start — customers, items, orders, invoices, suppliers — and you expand once those are green. It's the operational slice of what we describe in data governance, cut down to the minimum that survives without a committee.

One detail that makes the difference: agree the threshold in writing and as a number. "Industry must be filled in for 95 % of active customers" is manageable. "We need good data" means nothing and guarantees the same argument every quarter.

How do you start a data quality plan in 90 days?

Without buying a tool and without stopping anything. The goal of the first quarter isn't perfect data: it's data that's measured, owned and trending in a known direction.

| Weeks | What happens | Deliverable | |---|---|---| | 1-2 | Pick 2 critical datasets by decision impact | Written, accepted scope | | 3-4 | Profiling: count nulls, duplicates, out-of-domain values | Baseline snapshot with numbers | | 5-6 | Define owners, definitions and thresholds | One-page table per dataset | | 7-8 | Deploy 5-7 automated checks with alerting | Alerts reaching actual people | | 9-10 | Clean up the backlog, prioritised by impact | Records corrected and traced | | 11-12 | Fix the process that produced the error | Closed list, mandatory field or source validation |

Weeks 11 and 12 are the ones almost everyone skips and the only ones that make the improvement last. If the duplicate was born because customers get created in two places, as long as that stays true you'll still be cleaning duplicates in 2028.

A warning about the initial profiling: be ready for the number to be worse than management expects. That's normal, and it's a good sign. The first data quality report isn't presented as an accusation — it's a baseline to compare against three months from now.

How does data quality affect AI projects?

More than the slide deck admits. A model learns from the history you give it: if 30 % of orders have no channel recorded, the model will learn that channel doesn't exist, and it will do so with total conviction. Three concrete effects:

• Silent bias. Badly filled records are almost never randomly distributed: they cluster in one branch, one shift or one customer type. The model inherits that hole and fails precisely where you least expect it.
• Models you can't evaluate. If the label — the outcome you want to predict — is recorded wrong, you can't tell whether the model is right. That's the problem that kills half of all pilots, as we analyse in why AI pilots fail.
• Confident, false answers. In assistants that query your own documentation, an outdated document doesn't produce an error: it produces an impeccable answer built on expired information. Document quality is data quality too.

That's why the order we defend never changes: first know what data you have and in what state, then decide the use case. We develop it in data before AI, and it's the difference between a pilot that stays and one that demos beautifully and never gets used.

Frequently asked questions

What is an acceptable level of data quality?

It depends on the use, which is why it's set per field rather than globally. For invoicing, tax ID and registered address need to sit close to 100 %; for segmenting a campaign, 90 % of industry filled in is usually plenty. The mistake is demanding perfection everywhere: it costs a fortune and improves no decision.

Do I need to buy a data quality tool?

Not at first. The first few dozen checks are written in SQL over the warehouse you already have and scheduled with the same orchestrator that moves the data. A dedicated tool makes sense when you have many datasets, several teams and a real need for lineage; before that it's usually a licence that hides the fact that nobody owns anything.

Who should lead data quality, business or technology?

Business defines, technology implements. If technology leads alone, rules get imposed that the business doesn't recognise and people find ways around them. If business leads without technical support, it stays a wish list with no automated checks holding it up.

Can AI clean up historical data?

Partly, and with judgement. Text normalisation, fuzzy duplicate detection and classification of free-text fields are handled well by models and save weeks of manual work. What you can't do is invent a value that was never recorded: if the creation date doesn't exist, no model recovers it — it can only estimate it, and that estimate should be flagged as such.

If you recognise your own company in three or four of the problems in this guide, what helps isn't a 40-page governance plan: it's looking at two or three real datasets, counting nulls, duplicates and reconciliation gaps, and putting numbers on what today is just a feeling. That's the audit, and it comes out as a prioritised list of what to fix first. If you'd rather tell us where it hurts before that, let's talk for half an hour and we'll say plainly whether your case is a cleaning problem, a process problem, or both.