Saying How Sure You Are: Estimative Language and Confidence Levels

BY DR. LELAND W. SMITH

Every finished piece of analysis ends with a judgment, and every judgment carries a hidden question: how sure are you? Junior analysts often answer that question badly, not because their research is weak, but because the words they choose mean different things to different readers. "Probably" can mean 55 percent to one reader and 85 percent to another. "Possible" can mean almost anything. This guide explains how to express uncertainty so that your reader takes away the same level of belief you intended.

The approach here draws on a long tradition in intelligence analysis, starting with Sherman Kent's work in the 1960s and continuing through the analytic standards that government agencies publish today. You do not need to work for a government to use it. The same discipline helps in due diligence, fraud investigation, journalism, and any OSINT report where someone will make a decision based on what you wrote.

Why words of probability cause trouble

Uncertainty is normal in open-source work. You rarely have a signed confession or a complete document trail. You have fragments: a business registration, a social media post, a photo with a partly visible street sign, a domain registered on a certain date. From those fragments you infer something, and the inference is never certain.

The problem is not uncertainty itself. The problem is that ordinary English handles uncertainty loosely. Words like "may", "could", "likely", "possibly", and "there is a chance" have no fixed meaning. Readers fill in the gap with their own assumptions, and those assumptions vary widely. When a client reads "the account is likely operated by the same person", one reader hears "near certain" and acts accordingly. Another hears "somewhat more than a coin flip" and asks for more work. Neither reader is wrong. The writer failed to pin the meaning down.

Sherman Kent and the words of estimative probability

Sherman Kent was a Yale historian who became one of the founders of modern US intelligence analysis and led the CIA's Office of National Estimates. In 1964 he published an essay titled "Words of Estimative Probability" in the CIA's internal journal, Studies in Intelligence. The essay has since been declassified and is widely available.

Kent described a case that bothered him. A 1951 National Intelligence Estimate on Yugoslavia said that an attack in that year should be considered a "serious possibility". When Kent later asked colleagues what odds they had in mind when they agreed to that phrase, the answers ranged widely, from roughly 20 percent to 80 percent. The people who had drafted and approved the same sentence did not share a meaning for it. If they could not agree, the policymakers reading it had no chance.

Kent's answer was to tie a small set of words to numerical ranges. His proposed scale, as published in the essay, looked like this:

• Certain: 100 percent
• Almost certain: about 93 percent, give or take about 6 percent
• Probable: about 75 percent, give or take about 12 percent
• Chances about even: about 50 percent, give or take about 10 percent
• Probably not: about 30 percent, give or take about 10 percent
• Almost certainly not: about 7 percent, give or take about 5 percent
• Impossible: 0 percent

Kent also described a cultural split among analysts. He called one group the "mathematicians", who wanted numbers, and the other the "poets", who felt that numbers implied false precision and preferred prose. His scale was a compromise: words, but words with agreed meanings. That compromise remains the basis of most estimative language standards.

Modern scales you can borrow

You do not have to invent your own scale. Two published standards are widely used and easy to adopt.

US Intelligence Community Directive 203

The Office of the Director of National Intelligence publishes Intelligence Community Directive (ICD) 203, "Analytic Standards". It includes a table that maps probability terms to ranges. The terms come in pairs, and an analyst is expected to pick one column and use it consistently:

• Almost no chance / remote: 1 to 5 percent
• Very unlikely / highly improbable: 5 to 20 percent
• Unlikely / improbable: 20 to 45 percent
• Roughly even chance / roughly even odds: 45 to 55 percent
• Likely / probable: 55 to 80 percent
• Very likely / highly probable: 80 to 95 percent
• Almost certain / nearly certain: 95 to 99 percent

The UK Probability Yardstick

The UK's Professional Head of Intelligence Assessment publishes a similar scale, often called the Probability Yardstick. It runs from "remote" through "highly unlikely", "unlikely", "realistic possibility", "likely or probably", and "highly likely" to "almost certain", and it leaves small deliberate gaps between some bands to remind readers that the boundaries are approximate.

Choosing one for your practice

Pick one scale, write it down, and put it in every report, either in a short methodology note or an appendix. Consistency matters more than the specific choice, and mixing terms from different scales in one report defeats the purpose. For most private-sector OSINT work, the ICD 203 table is a good default because it covers the whole range without gaps and is easy to explain.

Likelihood and confidence are different things

This is the distinction that most new analysts miss, and it is the one that does the most to improve a report.

Likelihood is your estimate of how probable something is. "It is likely that the two accounts are operated by the same person."

Analytic confidence is your assessment of how solid the basis for that estimate is. It reflects the quality and quantity of your sources, how well they agree, and how much you had to assume to reach the judgment. ICD 203 and many other standards use three levels: high, moderate, and low.

The two vary independently. You can say something is likely with low confidence, which tells the reader that your best guess points one way but the evidence is thin and could easily shift. You can also say something is unlikely with high confidence, which tells the reader you have strong, consistent evidence that it is not the case.

Keep them in separate sentences or clearly separate clauses. Do not write "we are highly confident it is likely", because that blends the two and invites confusion. A cleaner form is: "We assess the two accounts are likely operated by the same person. We have moderate confidence in this judgment, because the overlap rests on writing style and posting times rather than on a shared identifier."

What drives confidence up or down

Confidence should not be a mood. It should follow from things you can name. When you set a confidence level, check these factors and say which ones applied.

Factors that raise confidence

• Independent corroboration. Two or more sources that did not draw on each other point the same way. A company registry entry and an unrelated court filing that both list the same director are stronger than three news articles that all repeat one press release.
• Source reliability with a track record. Official registries, court records, and primary documents generally carry more weight than anonymous posts, though each still needs checking.
• Direct observation. Evidence that shows the thing itself, such as a filed document, rather than someone's description of it.
• Few assumptions. The judgment follows from the evidence without long chains of inference.
• Alternatives tested and rejected. You considered other explanations and found evidence against them.

Factors that lower confidence

• Single-source reporting. Everything rests on one document or one account.
• Circular reporting. Several sources that look independent turn out to copy one origin.
• Stale information. The evidence is old, and the situation may have changed.
• Key assumptions. The judgment depends on something you believe but have not verified, such as "the user name is unique to this person".
• Possible deception. The subject has a motive and the means to plant false signals.
• Gaps you know about. You could not access a relevant source, or a search returned nothing where you would expect a result.

A worked example

Suppose a client asks whether a small online retailer is connected to a person who ran a similar business that closed after customer complaints. You find the following:

• The new retailer's domain was registered five months after the old business closed.
• Both websites use the same uncommon e-commerce theme with the same customised footer text, including the same spelling error.
• The returns address on the new site is a commercial mail-receiving agency in the same city as the old business.
• The state business registry lists a different name as the new company's organiser, and you find no public link between that name and the earlier owner.
• Domain registration details are privacy-protected for both sites.

A badly worded judgment would read: "The new store is probably run by the same people and may be a continuation of the earlier operation." This tells the reader very little. "Probably" and "may" are doing different jobs, nothing explains the basis, and the reader cannot tell how much weight to put on it.

A better version:

We assess it is likely (55 to 80 percent) that the new retailer is connected to the operator of the earlier business. The strongest indicator is the identical customised footer text, including a shared spelling error, which is unlikely to arise independently. We have low confidence in this judgment. The connection rests on website similarities that a third party could copy, the registered organiser has no visible link to the earlier owner, and registration privacy prevents us from comparing domain records. A records search for the organiser's other business filings would be the most useful next step.

The reader now knows your best estimate, how much to trust it, why, and what would change it. That last part matters. A judgment that names its own weak points is more useful, and more honest, than one that sounds certain.

Examples of good and bad wording

Here are some common patterns, with fixes.

• Weak: "The image could have been taken in Lisbon." Better: "We assess the image was likely taken in Lisbon, based on the tram livery and the paving pattern. Moderate confidence: we did not find the exact street."
• Weak: "It is possible the email address belongs to the subject." Better: "We assess it is unlikely (20 to 45 percent) that the address belongs to the subject. The only link is a shared first name and city. Low confidence."
• Weak: "There is no evidence the director has other companies." Better: "We found no other directorships for this person in the UK Companies House register. We did not search other jurisdictions." The fix here is not a probability word at all. It is a clear statement of what you searched, which prevents the reader from treating absence of results as proof.

Common mistakes and limits

Hedging every sentence

Some writers add "may", "possibly", or "appears to" to every line in an attempt to be careful. The result is a report the reader cannot act on. Separate facts from judgments. State facts plainly: "The company was registered on 3 March 2024." Apply estimative language only to the judgments you draw from those facts.

Using probability words for facts you have not checked

"The subject likely lives in Denver" is sometimes written when the analyst simply has not looked at available records. That is not an estimate, it is incomplete research. Check what can be checked before you estimate.

Treating confidence as a measure of effort

Spending many hours on a question does not raise confidence by itself. Confidence comes from the evidence, not the time spent finding it.

The limits of the method

Estimative language reduces miscommunication. It does not make your judgment correct. A well-worded estimate based on a poor analysis is still a poor estimate. The scale also cannot tell you how to weigh evidence, only how to report the result. Pair it with structured methods such as analysis of competing hypotheses when the stakes justify the effort.

An ethics note

How you express certainty has real consequences for the people you research. An overstated judgment about a person's identity or conduct can lead a client to fire someone, refuse a deal, or make an accusation. Where your findings concern an identifiable individual, lean toward stating confidence explicitly and naming the gaps. If your confidence is low, say so prominently, not in a footnote. The goal is to give the reader an accurate picture of what you know, which includes an accurate picture of what you do not.

Checklist

1. Choose one published probability scale and use it for the whole report.
2. Include a short key explaining your terms and confidence levels.
3. State verified facts plainly, without probability words.
4. Express each judgment with a likelihood term from your scale, using the range rather than a single number.
5. State analytic confidence (high, moderate, or low) separately from likelihood.
6. Name the factors behind your confidence level: corroboration, source quality, assumptions, gaps.
7. Say what you searched when reporting a negative result.
8. Note the single most useful next step that would change your judgment.
9. Reread every judgment about an identifiable person and check that the wording does not overstate the evidence.