Sentiment analysis: what it really measures and where it gets it wrong

Sentiment analysis is one of those things that sounds solved and isn't. You run ten thousand reviews and twenty thousand support tickets through a model, it gives you back 68 % positive, you paint it on a dashboard with a green gauge and… nobody does anything with it. The number goes up, the number goes down, and Monday's meeting is exactly the same as last week's. That's how most sentiment analysis projects we come across end up: technically working, operationally useless.

This guide is about the opposite. About what a sentiment label really measures, about the specific places it gets things wrong (which are fairly predictable), about how to tell whether your classifier is good enough, and above all about how to get from "68 % positive" to "the problem is in the returns flow and it affects 400 customers a month".

What is sentiment analysis and what does it really measure?

Sentiment analysis means classifying a piece of text by the emotional charge it expresses: positive, negative, neutral. In its more useful form it adds intensity and, crucially, what that charge is directed at.

What it actually measures — worth being clear about before promising anything:

• It measures expression, not experience. A customer can be furious and still write an impeccably polite ticket. The model will see politeness.
• It measures the text you have, not the text you don't. The customer who leaves without saying a word leaves no sentiment to analyse. That's what churn prediction is for — it works on behaviour, not words.
• It measures at document level unless you ask for something else. "The product is excellent but delivery was a disaster" is a mixed review. Labelling it "negative" throws away half the information.

That last point is the line between a project that pays off and one that doesn't. The overall sentiment of a review is almost always redundant with the star rating the customer already gave you. What you don't have is the breakdown by aspect: delivery, price, support, product, billing. That's where the new information lives.

If your sentiment analysis tells you the same thing as the average star rating, you haven't built anything: you've built an expensive way to compute an average.

Which text sources are worth analysing first?

Not every source carries the same value or the same preparation cost. Roughly in order of return for a mid-sized company:

• Support tickets. The best source, and almost always the forgotten one. They're internal, they come with metadata (customer, product, date, agent, resolution time) and they describe concrete problems. If you can only tackle one source, tackle this one.
• Public reviews. Marketplaces, business listings, industry portals. Upside: they include your competitors, so you can compare. Downside: extreme-response bias (the delighted and the furious write, nobody else does) and fake reviews.
• Surveys with free-text fields. The "anything else you'd like to tell us?" at the end of an NPS. Low volume, very high density. Usually the best place to start validating a model because someone in-house has already read them.
• Chat and messaging conversations. Useful if you already run a chatbot or conversational support channel with history. Careful: lots of noise ("hi", "thanks", "ok").
• Social media and mentions. The flashiest in a demo and the least actionable. High volume, weak link to actual customers, irony everywhere.

One warning before you analyse anything, because it saves months: these texts contain personal data. Names, order numbers, sometimes ID numbers typed straight into a ticket. If processing leaves your environment, that has to be solved first, not later — we cover it in data anonymisation.

Lexicon, trained classifier or language model?

Three families, three cost and control profiles. There's no outright winner.

| | Lexicon / dictionary | Trained classifier | Language model | |---|---|---|---| | How it works | Word list with polarity plus rules | Learns from your labelled examples | Natural-language instruction | | What you need | Nothing, works on day one | 500–2,000 hand-labelled texts | A good prompt and examples | | Accuracy in your domain | Low to medium | High if the data is good | Medium to high untrained | | Cost per 100,000 texts | Near zero | Near zero once trained | Material — measure it | | Explains why | Yes, traceable | Partly | Yes, but it can invent the reason | | Aspect extraction | Very limited | Must be trained separately | Comes almost for free |

In practice the setup that works best is hybrid: a language model for the ramp-up phase and for extracting aspects (no prior labelling required), and then, once volume grows, your own classifier trained on the labels that model produced and a human reviewed. Cheap to run, and it keeps your domain's judgement.

What we almost never recommend is starting by training from scratch. Hand-labelling 2,000 tickets is weeks of someone who knows the business, and the usual outcome is discovering halfway through that the category taxonomy was wrong in the first place.

Where does sentiment analysis get it wrong?

This is the part vendors tend to gloss over. The failures aren't random — they're systematic and they repeat:

• Negation and scope. "It's not that it's bad" isn't negative. "Never again" is. Dictionaries break here constantly.
• Irony and sarcasm. "Great, another missed delivery." The most quoted failure and, honestly, the least important in support tickets: people are blunt there. On social media it does hurt.
• Domain vocabulary. In insurance, "claim" is a neutral operational word. In a generic lexicon it leans negative. Same problem with "critical", "urgent", "incident", "cancellation". A generic model will paint your entire technical support red.
• The neutral class that doesn't exist. Many systems have no decent neutral class and split charge-free texts between positive and negative. Result: noise that drifts month to month without anything real happening.
• Mixed languages. English, Spanish and whatever your international customers write, all in the same inbox. Measure accuracy per language, never in aggregate.
• Text that isn't opinion. Email signatures, auto-replies, confirmations, one-word answers. If you don't filter them out, they dilute any signal.
• Channel bias. The average sentiment of a complaints form is negative by construction. Comparing channels without adjusting for that says nothing.

The practical rule: never compare sentiment across different sources, compare each source against itself over time. Trends within a channel are reliable; absolute levels between channels are not.

How do you turn sentiment into a decision?

This is the section that justifies the whole project. A polarity label on its own moves nothing. What moves things is the combination of three columns:

1. Aspect — what it's about (delivery, price, support, the app, billing, product X). 2. Volume — how many distinct customers say it, not how many messages there are. A customer who writes fifteen times is one case, not fifteen. 3. Impact — what revenue, what segment or what support cost sits behind those customers.

With those three columns you get a prioritised list a product committee can read in five minutes: "complaints about the returns process, 412 unique customers this quarter, up 31 % on the previous one, concentrated in the segment with the highest repeat rate". That is actionable. "68 % positive" is not.

And one piece that's almost always missing: the owner. Every aspect in the taxonomy needs someone who receives its list and answers for it, exactly as with any other indicator — the same principle we apply when choosing business KPIs. A sentiment report with no specific recipient becomes a monthly PDF nobody opens after month three.

Crossing it with your customer segmentation multiplies the value: the same problem can be anecdotal in your low-value segment and critical in the high-value one.

How do you evaluate whether the model is right?

Don't trust the out-of-the-box accuracy figure: it was measured on English film reviews, not on your tickets. The minimum serious protocol:

| Step | What to do | Effort | |---|---|---| | 1. Sample | 300–500 random texts, stratified by channel and language | 1 day | | 2. Double labelling | Two people from the business label separately, blind | 2–3 days | | 3. Human agreement | Measure how often they agree with each other | 1 hour | | 4. Comparison | Model against the human consensus | 1 hour | | 5. Error by class | Confusion matrix: what gets mistaken for what | half a day | | 6. Repeat | Re-sample every quarter | ongoing |

Step 3 is the one everyone skips and the most revealing. If your two human reviewers only agree on 75 % of cases, 75 % is your ceiling: asking the model for 90 % is asking it to be right about something people themselves can't agree on. That figure also tells you whether your taxonomy is badly defined — high disagreement usually means overlapping categories, not careless reviewers.

And measure per class, never in aggregate. A model with 85 % overall accuracy that gets 95 % on positive and 40 % on negative is useless, because negative is exactly what you were looking for. It's the same reasoning we apply to measuring AI ROI: the aggregate number hides precisely what matters.

How do you build the first use case without it dragging on?

A realistic ramp-up in four or five weeks, with a single source:

• Week 1. Pick the source (tickets, nearly always) and extract 12 months of history. Strip signatures, auto-replies and empty messages. Anonymise what needs anonymising.
• Week 2. Define the aspect taxonomy with the people who deal with customers, not with the technical team. Between 8 and 15 categories. Past 20, nobody will use it.
• Week 3. Process the full history with a language model and a prompt that asks for polarity per aspect, not overall. The history alone gives you trends on day one.
• Week 4. Evaluate using the protocol above. Fix the taxonomy with what you learn. There are usually one or two categories that need splitting in two.
• Week 5. Build the report prioritised by aspect, volume and impact, and assign an owner to each category. First review meeting.

Processing history before real time is deliberate. It's cheaper, it touches no production system and it answers the question that actually matters — "does this tell us anything we didn't already know?" — before you've invested in integrations. If the answer is no, you've spent a month instead of a quarter. And none of these phases works if the source text is dirty: the same rules apply here as in data quality.

Frequently asked questions

What accuracy is reasonable to expect from sentiment analysis?

On your own domain texts with a well-defined taxonomy, 75 % to 85 % agreement with the human consensus is a solid result. Anyone promising more than 95 % without having seen your data is quoting an academic English-language benchmark, not your tickets.

Do you need hand-labelled data to get started?

To get started, no: a language model with good instructions gives usable results from day one. Manual labelling is still essential for evaluation — you need those 300–500 human-reviewed texts — and it pays off for training your own classifier once volume makes the per-text cost matter.

Is sentiment analysis worth it with a low volume of reviews?

Under a few hundred texts a month, no: reading them is faster and more reliable than building a system. Automated analysis starts to pay off when nobody can read everything, or when you need to compare trends across periods, products or locations consistently.

Can I analyse my competitors' reviews?

Public reviews can be analysed, and comparing aspects against competitors is often the deliverable management cares about most. Check each platform's terms of use before downloading anything in bulk, and assume those sources carry more noise and more fake reviews than your own data.

---

If you have a support inbox with years of history nobody has read end to end, there's a prioritised list of product problems sitting inside it waiting for someone to pull it out. Our audit reviews which text sources you have, whether they're clean enough to analyse and what concrete decision could rest on them. If you already know that and would rather get to the point, let's talk.