Fraud detection with your own data: where rules stop working

Almost every company that asks us about fraud detection already has a system running: a list of rules someone wrote years ago, with an exception bolted on every time something bad happened. "If the amount is over X and the customer is new, block it." "If the IP is outside the country, flag it." It works — until whoever is trying to get something past you learns where the lines are and stays just under them.

This guide is about what comes next: when rules are still the right answer, what a model genuinely adds, why your number one enemy isn't fraud but the false positive, and how to tell whether the whole setup is paying for itself. No promises about "detecting 99 % of fraud": that figure, quoted without telling you how many legitimate transactions it kills along the way, means nothing.

What is fraud detection and what kinds exist in a company?

Fraud detection means deciding, on a specific transaction and usually within seconds, whether it looks legitimate. What counts as a transaction varies enormously by business, and that changes everything:

• Payment fraud. Stolen cards, chargebacks, orders that never get paid. The textbook case: millions of transactions, decisions in milliseconds.
• Internal fraud. Duplicate invoices, ghost suppliers, inflated expenses, off-policy discounts. Far lower volume, far more damage per case.
• Claims fraud. Exaggerated or outright invented claims. The classic territory in insurance.
• Promo and account abuse. Multiple accounts to reuse a welcome discount, systematic returns, resale.
• Identity fraud at onboarding. Manipulated documents, recycled identities, automated mass sign-ups.

Each one needs different data and tolerates different errors. Wrongly blocking a €40 payment is annoying; wrongly blocking the onboarding of a corporate client worth €200,000 a year is a board-level problem. Before any technology conversation, write down in one line which transaction is being scored, with what data is actually available at that instant, and what happens when the system is wrong in each direction.

Fraud isn't a detection problem. It's a decision problem under uncertainty with two asymmetric costs: the fraud that gets through, and the legitimate customer you stopped.

Rules or models? What each one actually solves

The question is badly framed, because in production they almost always coexist. But they're good at different things.

| | Rules | Model | |---|---|---| | How it decides | Conditions written by a person | Pattern learned from history | | Explainability | Total: "rule 14 fired" | Takes extra work to justify each decision | | Cases covered | The ones someone already saw | Combinations nobody ever formulated | | Reaction to change | Have to be rewritten | Adapts by retraining, if new labels exist | | Who maintains it | Business and risk | Data team + business | | Start-up cost | Low | Medium-high: data, labels, deployment | | Where it fails | Fraudsters learn the threshold | Inherits bias from history and ages |

What a model adds and a rule can't: combining many weak signals. None of these is suspicious on its own — slightly high amount, new shipping address, purchase at 3 a.m., three card attempts, account created 11 minutes ago — but together they describe a profile that a rule with five chained conditions will never capture properly.

What a rule adds and a model can't: obligation. If your policy says a transfer to a particular country doesn't leave without the CFO's signature, that's a rule, full stop. It isn't voted on and it isn't scored.

The architecture we see working in mid-sized companies: hard rules on top (compliance, limits, lists), a model in the middle scoring everything else, and business thresholds underneath that turn the score into an action. Starting the other way round — model first, rules later — usually ends with a model nobody dares to plug in.

What data do you need for fraud detection?

This is where half of these projects die. You need three things, and the third is the one that's almost never there.

1. Transaction history. With the variables exactly as they existed at the moment of the transaction, not as they look today. If your customer table holds the current status ("blocked"), and you train on that, the model will learn that blocked customers commit fraud. Congratulations: you've built a useless time machine. It's called data leakage, and it's the most common and most expensive mistake here.

2. Labels. Which transactions actually turned out to be fraud. Confirmed chargebacks, closed internal cases, claims rejected by the assessor. Without these there is no supervised learning.

3. A record of what you never saw. If you've spent years blocking everything your rules flagged, you'll never know whether those transactions were fraudulent. Your history only contains the fraud you let through and later discovered. That's why it pays to deliberately let a small, controlled sample of low-value suspicious transactions go ahead: it's the only way to learn what's on the other side of the threshold.

Two more things that matter just as much: the real discovery date (a chargeback arrives 60 days later; ignore that and you're measuring on cases that haven't matured yet) and velocity: nearly all the good signals are rate-based — attempts per minute, devices per account, accounts per device, deviation from that customer's own behaviour. If your data quality doesn't let you reconstruct those sequences, that's where the project starts, not at the model.

On volume: a reasonable supervised model wants on the order of several hundred confirmed fraud cases, not total transactions. With 20 cases you don't train anything; you write rules and set up anomaly alerts.

Why is the false positive the expensive problem?

Fraud is typically 0.1 %–1 % of transactions. At that rate, a system that said "all legitimate" would be right 99 % of the time. That's why overall accuracy is a useless metric here, and why anyone presenting it as an argument hasn't worked in this field.

Four numbers matter instead:

• Recall: of the real fraud, what share you catch.
• Precision: of what you flag, what share was genuinely fraud.
• Review rate: what share of the total lands in a human queue. That's your payroll bill.
• Friction on legitimate customers: how many good customers you block, and what they're worth.

The first two trade off against each other by moving a single number: the threshold. Lower it and you catch more fraud and flood the review queue; raise it and the queue clears while cases slip through. There's no "good" setting — there's a setting that corresponds to a business decision.

The honest way to set it is in euros. Put three figures on the table: average loss per fraud case that gets through, cost of reviewing one alert (minutes × hourly cost), and value lost when you stop a legitimate customer — which isn't the order value, it's the order plus the chance that customer never comes back. With those three, the threshold falls out on its own and the reasoning is documented. It's the same ROI exercise we ask for in any project: if you don't know what being wrong costs, you can't calibrate anything.

One nuance that gets forgotten: it isn't only block or allow. In between sit cheap actions — step-up verification, holding the shipment for 24 hours, requiring prepayment, a phone call — that cut fraud without the bluntness of a block. Designing that middle tier usually delivers more than squeezing two more points out of the model.

Where does human review fit?

The model doesn't decide: it prioritises. On the cases that matter, a person still decides. For that to work you need three pieces that almost never get budgeted:

• A work queue ordered by risk and by amount, not by arrival. Reviewers have finite time; spend it where the money is.
• Per-case explanation. "Risk 0.87" is useless. "High risk: third different card in 10 minutes, new shipping address, account created today" is not. This can be generated with standard attribution techniques and is fair to demand from any vendor.
• Feedback. Whatever the reviewer decides has to come back into the system as a label. If resolutions live in a separate spreadsheet, the model never learns and you'll repeat the same false positives every week.

Add the legal side: GDPR gives people the right not to be subject to solely automated decisions with significant effect, and blocking a customer qualifies. In practice: log who or what decided, keep the reason, and offer a human review path. None of this is optional, and it's far easier designed in at the start than patched on afterwards.

How do you build the first case in 90 days?

A sequence that works, with the deliverable for each stretch:

| Weeks | What happens | What you end up with | |---|---|---| | 1–2 | Pick one fraud type and quantify current losses | A baseline in euros | | 3–5 | Rebuild history without leakage and review labels | An audited training set | | 6–8 | Baseline model + comparison against current rules | Does it add anything? Yes or no, with numbers | | 9–10 | Threshold set in euros, plus a tier of intermediate actions | A written decision policy | | 11–12 | Shadow pilot: score everything, block nothing | Real comparison against live operations | | 13 | Gradual rollout + monitoring dashboard | A production system with an owner |

The shadow phase is non-negotiable. For two or three weeks the system scores every transaction and nobody acts on its decision; at the end you compare what it would have blocked against what actually happened. It's the only way to learn the impact without risking a single customer, and it has prevented more than one rollout that would have taken down half a day of legitimate orders.

Then comes maintenance. A fraud model ages faster than any other, because there's someone on the other side actively trying to break it. Watch for drift, retrain on a schedule rather than in a panic, and keep versions: the same MLOps practices you'd apply to any production model, with more reason here.

Mistakes we see repeated

• Training on data from the future. The classic. If a variable improves the model suspiciously well, it probably contains information from after the transaction.
• Measuring with overall accuracy. At 0.5 % fraud, 99.5 % is the floor, not an achievement.
• Not counting review cost. A system that catches 30 % more fraud but triples the queue can be a worse deal than the old one.
• Leaving the old rules untouched. Plug a model on top of 300 accumulated rules and nobody will know what blocked what. Clean up first; half of them nobody understands any more.
• Punishing your best customer. The system has to tell the difference between an unusual transaction and a transaction that's unusual for that customer. An €8,000 order is normal if they've been placing them for six years.

Frequently asked questions

How much data do I need to start?

The number of confirmed fraud cases matters more than total volume. Below a few hundred labelled cases a supervised model doesn't hold up, and well-calibrated rules plus anomaly detection are the better bet. In parallel, start recording clean labels today: in a year they'll be your asset.

Can I detect fraud without labels, using anomalies alone?

Yes, and it's the usual route when there's no history. Anomaly detection flags whatever departs from normal behaviour, but it doesn't know whether that's fraud or just an unusual customer. It generates more noise and demands more human review; it works as a starting point and for surfacing new fraud types, not as the final system.

Do I have to tell customers an automated system is reviewing them?

You have to disclose the processing in your privacy policy and, if the decision is automated and significantly affects them, offer human intervention. What you shouldn't do is publish the specific signals you use: that's the same as publishing the manual for dodging them.

Is it better to buy a fraud solution or build one?

If your fraud is standard — card payments in ecommerce, say — off-the-shelf products come with models trained on millions of transactions you don't have, and they're hard to beat. If your fraud is specific to your business (claims, suppliers, your own promotions), nothing off the shelf fits and you build on your own data.

---

If you're at the "we have rules and we suspect something is getting past us, but we don't know how much" stage, the first step isn't a model: it's putting a number on current losses and checking whether your history lets you learn from it. That's exactly what we look at in the audit, and if you'd rather talk it through before moving any data, let's talk.