Data catalog: what to document and how to stop it being abandoned

A data catalog is the inventory of what a company has stored: which datasets exist, where they live, what each field means, where they come from and who answers for them. Put that way it sounds like boring documentation, which is exactly why most data catalogs end up in a shared folder nobody opens. But the problem they solve is very concrete and very expensive: the analyst who spends three days finding the right table, the two sales reports that give different numbers with nobody sure which to believe, the new vendor who asks "and what is this?" and nobody can answer with any certainty.

This guide is for whoever has to build one in a mid-sized company, without a dedicated governance team and without buying a six-figure platform. It's about scoping, documenting the useful minimum and, above all, designing it so it's still alive a year from now. Because an out-of-date catalog is worse than no catalog: it gives answers that look like the truth.

What is a data catalog, and what isn't it?

A data catalog is a searchable index of the organisation's data assets, with enough metadata that someone who didn't build the table can decide whether it's useful to them. It has three layers:

• Technical metadata: table name, schema, field types, volume, load frequency, source system. This can be extracted automatically from the database or warehouse catalog.
• Business metadata: what this dataset means, what each field means in plain language, which filter you have to apply for it to match the official report. A person writes this. There's no shortcut.
• Governance metadata: who owns it, how sensitive it is, what legal basis covers it if it holds personal data, how long it's retained.

It's worth saying what it isn't, too. It isn't a data dictionary exported from the ERP: that's a list of columns, not a catalog. It isn't the BI tool: a dashboard shows indicators, it doesn't explain where they come from. It isn't an AI project: the catalog comes first, and its absence shows immediately, because any AI initiative starts by asking what data exists and what state it's in. And it is under no circumstances a 200-page Word document.

A catalog works if it answers "where is the data for X and can I trust it?" in under a minute. Any slower and the team goes back to asking on chat — at which point the catalog is already dead.

Why do most data catalogs get abandoned?

We've seen the same pattern in very different companies. The catalog launches with energy — three weeks of meetings, a beautiful template, 180 tables documented — and six months later it's out of date. The causes repeat:

Everything got documented. Someone decided to catalog all 400 ERP tables, including the 250 nobody ever queries. The effort dissolved into the irrelevant, and the 20 tables that actually get used ended up as badly explained as before.

There's no real owner, there's a nominal one. An owner was assigned to a cell because the template asked for it, but that person never found out, has no time allocated and is never consulted. An owner who gets no notification when their table changes is a name in a spreadsheet.

The catalog lives apart from the work. If documenting a change means opening another tool, with another password, and filling in a twelve-field form, it doesn't happen. Documentation has to sit where the work already happens: in the transformation repository, in the same place the table is defined.

Nobody measures usage. No one ever checked how many catalog lookups there are per month, or how many fields still have no description. What isn't measured gets abandoned quietly.

It was treated as a project, not a process. A catalog is never "finished". It's maintained. Without a routine — fifteen minutes in the weekly data meeting, a quarterly review of critical assets — entropy wins.

The practical conclusion: 25 well-documented, up-to-date datasets beat 400 photographed once. Full coverage is a trap.

What should you document for each dataset?

Our minimum record fits on one screen. If a field doesn't change a decision for whoever is reading it, drop it.

| Field | What it solves | Example | |---|---|---| | Business name | Being found by searching in normal language | "Sales by order line" | | Technical location | Where it actually lives | `dwh.sales.fact_order_line` | | Description (3 lines) | What it contains and what it does NOT | "One row per invoiced order line. Excludes quotes and credit notes" | | Source and frequency | Whether today's data is there | ERP, daily load at 05:00 | | Business owner | Who to ask | Head of Administration | | Technical contact | Who fixes the load when it breaks | Data team | | Sensitivity | Whether it holds personal data, and what kind | Contains tax ID and contact details: medium | | Definition of 5-10 key fields | The parts that genuinely confuse people | `net_amount` = taxable base, excluding VAT and shipping | | Quality metrics | Whether you can trust it today | Completeness of `sector`: 71% | | Sample query | So it gets used correctly from minute one | SQL with the official filter applied |

Two details make the difference. First: documenting what the dataset doesn't include prevents half of all misinterpretations; nearly every "this report doesn't match" argument starts from an implicit scope nobody wrote down. Second: linking quality metrics — completeness, uniqueness, freshness — turns the catalog into something consulted before using a dataset, not only when someone is lost. If you don't have those yet, data quality covers how to calculate them in SQL without buying anything.

How do you keep a data catalog alive?

Short answer: automate everything mechanical and ritualise the little that's left. The split that works:

Automated, no exceptions. The table inventory, schemas, types, volume, last load timestamp and basic lineage (which table feeds which) come from reading warehouse metadata and transformation code. If someone is copying column names into a spreadsheet by hand, the catalog has an expiry date.

Manual, but inside the workflow. Business descriptions are written by a person, yes — but in the same place the table is defined, not in a separate tool. In practice: a metadata file next to the transformation code, reviewed in the same pull request as the change. Change the logic without touching the description and the change doesn't merge.

Maintenance rules that enforce themselves. Three we recommend: a new asset doesn't reach production without a minimum record; a field with no description shows up flagged in the catalog view (public embarrassment outperforms reminders); a quarterly review of the 20-30 critical assets with their owners, with a one-page record of decisions.

Two indicators, no more. Percentage of critical assets with a complete record reviewed in the last six months, and number of catalog lookups per month. The first measures health, the second measures usefulness. If the first climbs and the second doesn't, your catalog is pretty and useless: ask three analysts why they don't open it.

What tooling do you actually need to start?

Less than you'll be sold. The criterion is the size of the problem, not the size of the ideal catalog.

| Archetype | When it makes sense | Cost and risk | |---|---|---| | Shared spreadsheet | Fewer than 30 assets, one team | Free. Goes stale fast; valid as a weeks-long starting point, not a years-long one | | Internal wiki or repository docs | Technical team in place, versioned transformations | Very cheap. Needs pull-request discipline, but it's what survives best | | Self-hosted open-source catalog | 100+ assets, several sources, automatic lineage | No licence cost, but you maintain the deployment: budget half a technical role | | Commercial governance platform | Regulatory obligations, several business units, hundreds of users | High licence plus an implementation project: only if there's already a process to support |

The expensive mistake is buying the platform so that it creates the process. It doesn't. The tool collects metadata automatically and renders lineage nicely, but business descriptions and accountability for each dataset stay human work. We've seen companies with an excellent catalog in a wiki. We've also seen expensive, empty platforms.

If you're further back on the path — still arguing about which system is the source of truth — the right order is in data governance and data strategy. The catalog makes that governance visible; it doesn't replace it.

Where do you start: a 60-day plan?

No new team, and without stopping what's already running.

| Weeks | What happens | What you have at the end | |---|---|---| | 1-2 | List the 10 most repeated business questions and trace which tables answer them | A short list of genuinely critical assets, not theoretical ones | | 3-4 | Extract technical metadata automatically and create the empty record per asset | An inventory that already updates itself | | 5-6 | 30-minute interviews with whoever uses each asset; write descriptions and scope | 20-30 records with business meaning documented | | 7-8 | Assign named owners, wire in quality metrics and publish | A searchable catalog and two indicators running |

Two traps in this plan. The 30-minute interview is worth more than the template: it almost always surfaces that two departments call different things an "active customer", and that conversation is worth more than a hundred records. And publication needs a push: announcing it on a channel isn't enough — you have to answer the next three data questions with a catalog link instead of with the answer. People learn it by using it.

Frequently asked questions

How much does building a data catalog cost?

Tooling can cost nothing if you start on the wiki or repository you already use. The real cost is time: roughly 60-80 hours split between the data team and business owners to properly document 25-30 critical assets, plus around two hours a month of maintenance.

Who should own a dataset?

Whoever makes decisions with it, not whoever stores it. The customer master is owned by Sales or Administration, not by the technical team. The technical contact is a separate, complementary role: they fix the load when it breaks, but they don't decide what "active customer" means.

Do you need a catalog before an AI project?

You don't need the whole catalog, but you do need the part the project touches. Any serious initiative starts by inventorying what data exists and what state it's in; if that isn't written down, it gets done during the project anyway — just more expensively, and without leaving anything behind for next time.

What if the data is spread across Excel files and several systems?

That's normal and it doesn't block cataloguing. You document the spreadsheet as another asset, with its location, its owner and its risk — a file on somebody's laptop is a risk, and writing that down is usually what unblocks moving it somewhere decent, whether that's a data warehouse or something simpler.

If you're trying to work out whether your problem is cataloguing, quality or architecture, the answer usually appears after a morning spent looking at the sources: the audit is exactly that inventory, and it tells you what deserves documenting now and what can wait. And if what you want is a second opinion on a plan you've already written, let's talk for half an hour — that's usually enough to see whether the catalog will survive its first quarter.