Data engineering: the invisible work that holds everything else up

Data engineering is the part of the job nobody shows in the results presentation. Nobody puts on screen the process that collects orders from the ERP every night, fixes the badly entered dates and has the table ready by six in the morning. They put up the dashboard. But when the dashboard breaks — and it does — the problem is almost never in the chart: it's in the pipe feeding it. This guide is about that pipe, when it's worth building properly and how to do it without opening a two-year project.

We're writing it because it's the conversation we repeat most often. A company calls us about a predictive model or a dashboard and, by the third question, the same bottleneck shows up: the data exists, but it's spread across six systems, nobody joins it in a repeatable way and every report is assembled by hand. That's not a missing data scientist. That's missing data engineering.

What is data engineering and how is it different from BI?

Data engineering is the set of processes that move a piece of data from where it's created to where it's used — automatically, repeatably and verifiably. Pull it out of the source system, clean it, give it a stable structure, load it somewhere queryable and raise a flag when something breaks.

The difference from BI is one of layer, not importance:

• Data engineering answers "does the table exist, is it complete and is it up to date?".
• BI and analytics answer "what is this table telling us and what do we decide with it?".

If BI is the tap, data engineering is the plumbing of the whole building. When it works, it's invisible. When it doesn't, everyone blames the tap. That's why so many companies accumulate visualisation licences and zero investment in what sits underneath: what's underneath isn't visible, doesn't demo well and doesn't excite a steering committee.

Nor should it be confused with data science. The data scientist builds models; the data engineer builds the supply those models consume every single day. A model that works on a laptop with a manually exported spreadsheet isn't in production: it's a mock-up. We cover that in machine learning in production.

80% of what people call an "AI project" is really a data engineering project with a small model at the end.

What signs tell you your company needs data engineering?

You don't need an expensive diagnosis. These signs show up in a single meeting:

• Someone spends days a month assembling the same report. Download, paste into a spreadsheet, cross-reference with a lookup, eyeball it. If that happens every month, you're already paying for data engineering — just by hand and without version control.
• Two departments present different figures for the same thing. The classic symptom: there's no official table, there are six extracts built on different criteria. The root cause is usually definitional, not technical, and you tackle it with data governance.
• The dashboard has been stale for days and nobody noticed. Without monitoring, a process that fails silently is worse than no process: people decide on old data believing it's yesterday's.
• One ERP change breaks three reports. If nobody knows what depends on what, every source-system update is a game of roulette.
• Historical data can't be reconstructed. The operational system overwrites the current state; if you haven't stored a daily snapshot, the history any forecast needs simply doesn't exist.
• The AI vendor asks for "a clean CSV" and nobody knows who produces it. Decisive sign.

With two of those six, your next euro returns more in the pipe than in the visualisation tool.

What does a data engineer actually do?

Less glamour and more craft than people imagine. The day-to-day splits into five blocks:

Ingestion. Connecting to source systems: ERP, CRM, e-commerce platform, files a supplier sends over, third-party APIs. The difficulty here is rarely technical; it's getting credentials, permissions and a time window where the ERP isn't closing the day.

Modelling. Turning operational tables — designed to make the application fast — into analytical tables, designed so a business question is answered in one query. This is where you decide what a "customer" is, what counts as a "valid order" and how returns are handled. These are business decisions dressed up as technical ones.

Quality and testing. Automatic rules that run on every load: the primary key isn't duplicated, the amount isn't negative, today's row count hasn't dropped 90% versus yesterday. Without this, errors get discovered in a board meeting.

Orchestration. Deciding which process runs, in what order, with which dependencies, what happens if one fails and who gets notified. A process that fails without alerting is a process that lies.

Documentation and lineage. Knowing where each field comes from and which reports depend on it. It's the first thing abandoned and the first thing missed when the engineer who built it leaves.

What does a sensible data architecture look like for a mid-sized company?

You don't need the architecture of a thousand-person tech firm. You need the simplest one that holds up for three years. This is the reference we propose almost every time, by layer:

| Layer | What it does | Sign you need it now | |---|---|---| | Ingestion | Periodic copy of source systems, untransformed | Someone exports by hand every week | | Warehouse / raw | Stores data exactly as it arrived, with load date | You can't reconstruct what the system said a month ago | | Transformation | Cleaning and analytical model versioned in code | Business rules live inside spreadsheet formulas | | Serving | Final tables consumed by BI, reports and models | Every report recalculates its own version of truth | | Observability | Alerts for failure, freshness and volume | You learn about failures from an angry salesperson |

Two important caveats. First: the raw layer isn't optional. Keeping data as it arrived is what lets you recalculate the past when you discover a criteria error, and it takes up far less space than people fear. Second: transformation must live in versioned code, not in a tool's interface. If the logic is trapped in clicks inside a product, switching products costs double and reviewing what someone changed on Tuesday is impossible.

On build order: first the source of truth for ONE domain — usually sales — then extend. Doing everything at once is the recipe for the project that dies in the analysis phase. We develop that in data strategy.

What does it cost, and should it sit inside or outside?

The real cost splits into three lines and almost nobody budgets the third:

• Build. Setting up ingestion and the model for one domain: weeks, not months, if the scope is contained.
• Infrastructure. Storage and compute. For mid-sized volumes it's usually the smallest line, and the one that scares people most out of unfamiliarity.
• Maintenance. The source system changes, the business adds a product line, a subsidiary appears. Budget maintenance from day one or you'll own a pipe that degrades on its own.

On in-house versus outsourced, our honest criteria:

| Situation | Recommendation | |---|---| | First project, no data profile on staff | External to build, with a handover and documentation commitment | | Someone technical already maintains it part-time | External for design, internal for daily operation | | Data is the product or the core of the business | In-house, no debate | | One-off need for a single report | Neither: don't build a platform for one report |

The question to ask any vendor — us included — is: "when you're done, who can touch this without you?". If the answer implies permanent dependency, the real price is higher than the quote. That criterion and nine others are in how to choose an AI consultancy.

How do you start without opening a two-year project?

With a ridiculously small scope and a short deadline. The sequence that works:

1. Pick a decision, not a system. "Knowing every Monday what margins the week left by product line" is a scope. "Integrate the ERP" isn't. 2. Map backwards from that decision. Which fields you need, which system they live in, how often they change and who owns their definition. 3. Build the pipe for those fields only. Ingestion, raw, transformation, final table. No ambition to cover the rest yet. 4. Add three quality tests and one alert. Uniqueness, range and freshness. With an alert going somewhere a human will actually see it. 5. Plug the dashboard in at the end. Not at the start. The dashboard is the consumer, not the project. 6. Switch the manual report off. If the old process is still alive, you haven't finished: you've added work.

In six to eight weeks that's in production and the difference shows. And the most valuable part isn't the output, it's what you learn along the way about the real state of your systems, which is usually different from what the official inventory claims.

Frequently asked questions

Do I need a data warehouse if I only have an ERP?

If all the relevant data lives in a single system and its reports are enough for you, no. A warehouse earns its place when you need to join sources, keep history the operational system overwrites, or take heavy query load off the system your operations team uses every day.

Can I do data engineering with the tools I already have?

Often yes, at least to start. Many companies already have an underused database and an available scheduler. What almost never works is holding the logic in spreadsheet macros: they aren't versioned, aren't tested and depend on one particular laptop being switched on.

How long until the return shows?

The immediate return is recovered time: the hours someone spent assembling the report by hand. That's measurable in the first or second week after the switchover. The return from better decisions takes longer and is harder to attribute, but the first one alone usually pays for the build.

Do I need data engineering before doing AI?

For an isolated test, no. To put AI into production, yes: a model needs fresh data with the same shape every day, and that's exactly what a well-built pipe produces. The full argument is in data first, AI second.

If you recognised three or four of those signs in your own company, the next step isn't picking technology: it's narrowing down one concrete decision and seeing what's missing to feed it. That's what we do in the audit — two weeks, real systems, a map of what's there and what isn't — and if you'd rather talk it through first with no strings attached, let's talk.