Cognitive Biases That Derail Investigations

BY DR. LELAND W. SMITH

Most failed investigations do not fail for lack of data. They fail because the analyst settled on an answer too early, read every new fact as support for it, and never seriously tested the alternatives. The tools were fine. The thinking was not.

Cognitive biases are systematic patterns in how people judge and decide. They are not character flaws and they do not go away with experience or intelligence. Experienced analysts are often more confident, which can make some biases worse. The practical response is not to try harder to be objective. It is to build checks into your process so the biases have fewer places to hide.

This guide covers six biases that cause the most trouble in open-source investigations, the structured techniques that counter them, and how to fold those techniques into an ordinary workflow without slowing it to a crawl.

Why investigators are vulnerable

OSINT work has features that invite bias. Clients often arrive with a theory and a deadline. Search tools return results ranked by relevance to your query, so a biased query returns biased results. Online information is abundant, so you can almost always find something that supports a hypothesis. And much of the work is done alone, with nobody to ask the awkward question.

Richards Heuer's Psychology of Intelligence Analysis, written for the CIA and now freely available, remains the best introduction to this problem. Its central point is still the right starting place: knowing about biases does not by itself protect you from them. You need methods.

Six biases that derail investigations

Confirmation bias

Confirmation bias is the tendency to seek, notice and give weight to information that supports what you already believe, and to discount what contradicts it. In practice it shows up in the queries you write. If you suspect that a company is a shell, you search for "shell company" indicators and find some. You are less likely to search for evidence of real operations.

It also shows up in how you treat ambiguous items. A post that could mean two things gets read the way that fits your theory, and then counted as evidence.

Anchoring

Anchoring is the pull of the first number, name or explanation you encounter. Later estimates drift towards it, even when it was arbitrary. A client who says "we think he moved about 200 thousand through those accounts" has given you an anchor. So has the first search result that names a suspect.

Anchors are especially strong when they come with authority, such as a client's briefing or a respected colleague's first impression.

Availability

The availability heuristic leads people to judge how likely something is by how easily examples come to mind. If you recently worked a case involving romance fraud, the next suspicious money transfer looks like romance fraud. Recent, vivid or well-publicized patterns crowd out duller but more common explanations.

Mirror imaging

Mirror imaging is assuming that other people think, value and act as you would in their position. An analyst might conclude that nobody would register a company under their real home address if they were committing fraud, because the analyst would not. But the subject may be careless, may not care about exposure, may operate in a culture where that is normal, or may be a victim whose identity was used.

Satisficing

Satisficing, a term coined by Herbert Simon, means accepting the first answer that seems good enough rather than comparing the options. Under time pressure it is often rational. In an investigation it means you stop collecting as soon as one hypothesis fits the facts, without asking whether another fits them equally well or better.

Groupthink

Groupthink, described by Irving Janis, is the tendency of cohesive groups to converge on a view and suppress dissent to preserve harmony. In investigative teams it appears as nobody wanting to challenge the lead analyst, or as a client meeting where everyone nods along to a narrative that only one person has really examined. Solo practitioners are not immune. A one-person shop that only ever discusses cases with the client can drift into a group of two.

Structured countermeasures

The intelligence community developed a family of structured analytic techniques to counter these problems. Several are simple enough to use on small commercial cases.

Key assumptions check

List every assumption your current judgement depends on. Then, for each one, ask: how confident am I in this, what is it based on, and what would happen to my conclusion if it were wrong?

The value is in writing them down. Assumptions that seem obvious in your head often look thin on paper. Typical examples in OSINT work include "this account is controlled by the person named in its profile", "the photo was taken where the caption says" and "the company's registered address is where it operates".

Analysis of competing hypotheses

Analysis of competing hypotheses, developed by Heuer, asks you to list several plausible explanations and then assess each piece of evidence against all of them, usually in a simple matrix. The key move is to look for evidence that is inconsistent with a hypothesis, not merely consistent with it. Evidence that fits every hypothesis does not help you choose. The hypothesis with the least evidence against it is usually the strongest candidate.

You do not need software. A spreadsheet with hypotheses across the top and evidence down the side, marked consistent, inconsistent or neutral, is enough.

Devil's advocacy

Devil's advocacy assigns someone to argue against the prevailing view as strongly as possible. The role works best when it is formal and rotated, so arguing the other side is a job rather than a personal challenge. If you work alone, write the opposing case yourself in a separate note, and try to make it convincing.

Premortem

The premortem, popularized by the psychologist Gary Klein, asks the team to imagine that it is some months later and the conclusion has turned out to be badly wrong. Each person then writes down why it failed. Because the failure is treated as given, people feel freer to raise doubts they would otherwise keep quiet. The reasons they list become checks to run before the report goes out.

Other useful habits

• Search for disconfirmation. For every query designed to find support, write one designed to find contradiction.
• Delay the client's theory. Where you can, do an initial pass of collection before reading the client's preferred explanation in detail.
• Use base rates. Before deciding that something unusual is going on, ask how common the ordinary explanation is.
• Separate collection from judgement. Log raw findings neutrally first and interpret them in a later step.

Building the checks into a workflow

Structured techniques fail when they feel like extra paperwork. The way to make them stick is to attach each one to a stage you already have.

1. At intake: record the client's theory as one hypothesis among several. Write at least two alternatives before collection begins, even if they feel unlikely.
2. At planning: for each hypothesis, note what evidence would support it and what would undermine it. Plan queries for both.
3. During collection: log findings in neutral language with source, date and reliability. Avoid writing interpretations in the evidence log.
4. At the midpoint: run a short key assumptions check. Ask whether any new information changes which hypotheses are live.
5. Before drafting: fill in a competing hypotheses matrix, even a small one. Note which evidence is diagnostic.
6. Before delivery: run a premortem, alone or with a colleague, and act on the most plausible failure reasons.
7. In the report: state the main alternative explanations you considered and why you rejected them, with confidence levels.

On a small case these steps might add an hour. On a case where a wrong answer could damage a person's reputation or a client's decision, that hour is cheap.

A worked example

Suppose a client tells you that a former employee is running a competing business using stolen customer lists. They give you the former employee's name and a website they believe is his.

Without checks: you search the name with "new business" and find a company registered last year with a director who shares his name. The website lists services similar to your client's. You find a social media post congratulating him on a new venture. You report that he is running a competing business. That last search took twenty minutes and everything fitted.

With checks: at intake you write three hypotheses. He is running the competitor. Someone else with the same name is running it. He is involved in a new venture that is not a competitor. Your key assumptions check flags two weak points: that the director is the same person, and that "similar services" means the same market.

You then look for disconfirming evidence. The company filing gives a director's month and year of birth that does not match what the client knows. The congratulations post links to a business in a different sector. Your competing hypotheses matrix shows that most of your evidence fits all three hypotheses, and the only diagnostic item, the birth date, points away from the client's theory.

Your report says that the evidence does not support the claim that the former employee runs this company, explains why, and suggests what further lawful checks could resolve the question. The client is less pleased in the short term and much better protected.

Limits and common mistakes

• Treating techniques as rituals. A competing hypotheses matrix filled in after the conclusion is written only decorates the bias. Use the tools before you commit.
• Straw-man alternatives. Listing hypotheses you know are silly gives a false sense of rigour. Alternatives should be ones a reasonable colleague might hold.
• Bias hunting in others only. It is easy to spot anchoring in a client's briefing and miss it in your own first impression.
• Overcorrection. Endless doubt is its own failure. The aim is calibrated confidence, stated clearly, not refusal to conclude.
• Ignoring incentives. If you are paid more when the answer is dramatic, or the client clearly wants a particular result, say so to yourself and build an extra check.

An ethical note

Biased conclusions about people can do real harm: lost jobs, damaged reputations, misdirected legal action, or harassment by others who read your findings. That makes debiasing an ethical duty as well as a quality issue. When your work names a person, hold the strongest version of the alternative explanations, state your confidence honestly, and resist pressure to overstate. If a client asks you to find evidence for a fixed conclusion rather than to test a question, reframe the engagement or decline it.

Checklist

1. Record the client's theory as one hypothesis and add at least two plausible alternatives.
2. List what would support and what would undermine each hypothesis before collecting.
3. Write disconfirming queries alongside confirming ones.
4. Log evidence neutrally, separate from interpretation.
5. Run a key assumptions check at the midpoint.
6. Build a simple competing hypotheses matrix before drafting.
7. Assign devil's advocacy, or write the opposing case yourself.
8. Hold a premortem before delivery and act on its findings.
9. State alternatives considered and confidence levels in the report.
10. Decline or reframe work that asks you to prove a fixed conclusion.