- The Intuition, What Does Bayes Actually Do?
- Derivation from Conditional Probability
- The Law of Total Probability
- Bayesian Vocabulary, Prior, Likelihood, Posterior
- The Extended Form of Bayes' Theorem
- Tree Diagrams, Organising Bayesian Problems
- Fully Worked Examples
- Real-World Applications
- Common Mistakes
- 10-Question Quiz
§ 01The Intuition — What Does Bayes Actually Do?
Bayes' Theorem answers a single question: given that I have observed event B, how should I update my probability that event A is true?
Here is the classic scenario. A rare disease affects 1 in 1000 people. A diagnostic test for it is 99% accurate, it correctly identifies 99% of sick people and correctly clears 99% of healthy people. You test positive. Should you panic?
Most people's intuition says yes, 99% accurate sounds extremely reliable. But Bayes' Theorem reveals something surprising. Out of every 1000 people tested, roughly 1 is actually sick and tests positive, while roughly 10 healthy people also test positive (the 1% false-positive rate applied to the 999 healthy people). So of the ~11 positive results, only 1 is a true positive. Your probability of actually being sick, given a positive test, is only about 9%: not 99%.
This is not a flaw in the test. It is a mathematical consequence of the disease being rare. Bayes' Theorem is the tool that makes this calculation precise and general. It is used in medical diagnosis, spam filtering, courtroom evidence evaluation, machine learning, search engines, and scientific hypothesis testing.
§ 02Derivation from Conditional Probability
The derivation of Bayes' Theorem requires nothing beyond the definition of conditional probability applied twice. Recall that for any two events A and B with P(B) > 0:
Both expressions involve the same joint probability P(A ∩ B) in the numerator. Setting them equal gives Bayes' Theorem immediately.
From the second equation: P(A ∩ B) = P(B | A) · P(A). Substituting into the first:
Provided P(B) > 0. This is the complete theorem. Everything else (the law of total probability, extended form, Bayesian inference) is elaboration on how to compute P(B) in the denominator.
The theorem tells you how to "invert" a conditional probability: if you know P(B | A) (how likely is B given A?), you can find P(A | B) (how likely is A given B?): provided you also know P(A) and P(B). In most applications, P(B) is not given directly and must be computed using the law of total probability.
§ 03The Law of Total Probability
Bayes' Theorem requires P(B) in the denominator. When P(B) is not given directly, it can almost always be computed by conditioning on a partition of the sample space.
A set of events {A₁, A₂, …, Aₙ} is a partition of the sample space if the events are mutually exclusive (no two can happen simultaneously) and exhaustive (one of them must happen). For example, {Disease, No Disease} is a partition of the health-status space.
For the two-event case (A and its complement Aᶜ), this simplifies to:
P(B) = P(B | A)·P(A) + P(B | Aᶜ)·P(Aᶜ)This is the most-used form. The full Bayes' formula with total probability expanded in the denominator is:
P(A | B) = P(B|A)·P(A) / [P(B|A)·P(A) + P(B|Aᶜ)·P(Aᶜ)]This combined formula is what you actually use in practice. The numerator is the probability of the path "A happened, then B happened." The denominator is the total probability of B happening across all possible paths (whether A happened or not). Dividing gives the fraction of all B-outcomes that came from path A.
§ 04Bayesian Vocabulary — Prior, Likelihood, Posterior
Bayesian statistics introduces specific terminology for each component of the formula. Understanding these terms makes it much easier to set up problems correctly.
The symbol ∝ means "proportional to", dividing by P(B) just normalises to make the posterior a valid probability. The essential message is: your new belief is your old belief multiplied by how well A explains the evidence, then rescaled.
§ 05The Extended Form of Bayes' Theorem
When the hypothesis space has more than two possibilities, for example, a disease could be one of three types, or a manufacturing defect could come from one of four machines, the extended form of Bayes' Theorem handles them all simultaneously.
where {A₁, A₂, …, Aₙ} is a partition of the sample space, and k is any one of the hypotheses whose posterior probability you want. The denominator is the same for every k, it is just the law of total probability applied to the full partition.
The practical procedure is always the same: for each hypothesis Aᵢ, compute the product P(B | Aᵢ) × P(Aᵢ). Then divide each product by their sum. This gives the posterior probability of each hypothesis given the evidence B.
A factory has three machines: M₁ produces 50% of output with a 2% defect rate, M₂ produces 30% with a 3% defect rate, M₃ produces 20% with a 5% defect rate. A randomly selected item is defective. What is the probability it came from each machine?
Despite M₃ producing the least output, its high defect rate makes it equally likely as M₁ to be the source of a defective item.
§ 06Tree Diagrams — Organising Bayesian Problems
A probability tree is the most reliable visual tool for Bayesian problems. Each path through the tree represents one combination of events; the probability of the path is the product of the probabilities along its branches. Bayes' Theorem then becomes a matter of identifying the right paths and dividing.
The Three-Step Tree Method
For any Bayesian problem, follow this procedure systematically:
§ 07Fully Worked Examples
A disease affects 0.5% of the population. A test has 95% sensitivity (true positive rate) and 90% specificity (true negative rate, so 10% false-positive rate). A patient tests positive. What is the probability they have the disease?
D = has disease, T⁺ = tests positive. P(D) = 0.005, P(Dᶜ) = 0.995.
P(T⁺|D) = 0.95 (sensitivity) P(T⁺|Dᶜ) = 0.10 (false-positive rate)Even with a 95%-sensitive test, the low base rate means only about 1 in 22 people who test positive actually have the disease. The high false-positive rate (10%) swamps the true positives from the rare disease.
A bag contains two coins: Coin A is fair (P(H) = 0.5) and Coin B is biased (P(H) = 0.8). You pick one coin at random and flip it. It lands heads. What is the probability you picked Coin B?
The heads outcome has updated our belief toward the biased coin, but only moderately, one flip is not conclusive.
Using the same bag of coins as Example 03, you flip the chosen coin a second time and it lands heads again. Update the probability that you have Coin B.
Use the posteriors from Example 03 as the new priors:
P(B) = 8/13 ≈ 0.615, P(A) = 5/13 ≈ 0.385Each new observation of heads strengthens our belief that the biased coin was chosen. This sequential updating is the essence of Bayesian learning, the posterior from one update becomes the prior for the next.
40% of emails are spam. The word "offer" appears in 60% of spam emails and 5% of legitimate emails. An email contains the word "offer." What is the probability it is spam?
P(Spam) = 0.4, P(Legit) = 0.6, P("offer"|Spam) = 0.6, P("offer"|Legit) = 0.05
A single suspicious keyword raises the spam probability from the prior 40% to 89%. Real spam filters do this for hundreds of words simultaneously, taking the product of likelihoods (Naïve Bayes classifier).
Urn I contains 3 red and 7 blue balls. Urn II contains 6 red and 4 blue balls. An urn is selected at random and a ball is drawn. It is red. What is the probability it came from Urn II?
At an airport, 0.01% of passengers carry a prohibited item. The scanner detects prohibited items 98% of the time and has a 2% false-alarm rate (flags a safe passenger). A passenger is flagged. What is the probability they are carrying a prohibited item?
P(Prohibited) = 0.0001, P(Flag|Prohibited) = 0.98, P(Flag|Safe) = 0.02
Less than 1 in 200 flagged passengers actually carries a prohibited item. This drives real-world secondary screening procedures, the first alarm is rarely conclusive when the base rate is very low.
§ 08Real-World Applications
Bayes' Theorem is not a classroom curiosity: it underlies some of the most important reasoning tools in science, medicine, law, and technology.
DNA at a crime scene matches a suspect. The probability of a random person's DNA matching by chance is 1 in 1,000,000. The prosecutor argues: "There is only a 1-in-a-million chance the suspect is innocent." Explain the flaw using Bayes' Theorem.
The prosecutor is confusing P(Match | Innocent) = 0.000001 with P(Innocent | Match). These are not the same quantity.
Suppose the city has 1,000,000 people, and initially P(Guilty) = 1/1,000,000 (one guilty person among a million).
P(Match|Guilty) ≈ 1 P(Match|Innocent) = 0.000001 P(Match) ≈ 1/1,000,000 + 0.000001×999,999/1,000,000 ≈ 0.000001 + 0.000001 = 0.000002The correct posterior is 50%, not 99.9999%. Additional evidence (motive, alibi, eyewitness testimony) must be used to update this further. A DNA match alone is not nearly as conclusive as the prosecutor claims.
Three doors: a car behind one, goats behind the others. You pick Door 1. The host (who knows where the car is) opens Door 3, revealing a goat. Should you switch to Door 2?
P(Car at Door 1) = P(Car at Door 2) = P(Car at Door 3) = 1/3
P(Host opens Door 3 | Car at Door 1) = 1/2 (host chooses randomly between Doors 2 and 3)
P(Host opens Door 3 | Car at Door 2) = 1 (host must open Door 3, since Door 2 has the car)
P(Host opens Door 3 | Car at Door 3) = 0 (host cannot reveal the car)
You should always switch. The host's action (which is not random) transfers probability from Door 3 to Door 2. Bayes' Theorem makes this rigorous.
Using the disease scenario from Example 02 (prevalence 0.5%, sensitivity 95%, false-positive rate 10%): a patient tests positive on the first test, then takes a second independent test and is positive again. What is the final probability of disease?
From Example 02: P(D | T₁⁺) ≈ 0.0456. Use this as the new prior.
Two positive results push the probability from 4.6% to 31%, substantial, but still less than even odds. The base rate of 0.5% is so low that even two positive tests leave considerable uncertainty. A third positive test would raise it to approximately 77%.
§ 09Common Mistakes
§ 10Test Your Understanding
Ten questions covering all aspects of Bayes' Theorem: from vocabulary and formula recall to numerical application and conceptual traps. Work each one before revealing the explanation.