Probability Basics

Introduction

Probability quantifies uncertainty with a number between 0 and 1. It is the bridge between descriptive statistics (what happened) and inferential statistics (what to expect). Understanding the basic rules here makes hypothesis testing later feel intuitive instead of magical.

Core Definitions

• Experiment: a process with uncertain outcomes (roll a die).
• Outcome: a single possible result (roll a 4).
• Event: a set of outcomes (roll an even number).
• Probability: P(event) = number of favorable outcomes / total outcomes (for equally likely cases).

import itertools
Probability of rolling an even number with a fair die
outcomes = list(range(1, 7))
even = [o for o in outcomes if o % 2 == 0]
print(f"P(even) = {len(even)}/{len(outcomes)} = {len(even)/len(outcomes)}")

Output:

P(even) = 3/6 = 0.5

The Complement Rule

The probability an event does NOT happen is 1 minus the probability it happens:

P(not A) = 1 - P(A)

p_rain = 0.3
print(f"P(no rain) = {1 - p_rain}")

Output:

P(no rain) = 0.7

The Addition Rule

For two events that cannot happen together (mutually exclusive), the probability of A or B is the sum:

P(A or B) = P(A) + P(B)          (if mutually exclusive)
P(A or B) = P(A) + P(B) - P(A and B)   (general case)

p_mon = 1/7; p_tue = 1/7
print(f"P(Mon or Tue) = {p_mon + p_tue:.3f}")  # mutually exclusive
General case: P(red card or king) in a deck
p_red = 26/52; p_king = 4/52; p_red_king = 2/52
print(f"P(red or king) = {p_red + p_king - p_red_king:.3f}")

Output:

P(Mon or Tue) = 0.286
P(red or king) = 0.538

The Multiplication Rule and Conditional Probability

For two independent events:

P(A and B) = P(A) * P(B)          (if independent)

The general version uses conditional probability:

P(A and B) = P(A) * P(B | A)
P(B | A) = P(A and B) / P(A)

Example: Drawing Without Replacement

A bag has 4 red and 6 blue marbles. Draw two without replacement:

# P(first red) = 0.4
P(second red | first red) = 3/9
p_rr = (4/10) * (3/9)
p_rb = (4/10) * (6/9)
p_br = (6/10) * (4/9)
p_bb = (6/10) * (5/9)
print(f"P(RR) = {p_rr:.2f}, P(RB) = {p_rb:.2f}, P(BR) = {p_br:.2f}, P(BB) = {p_bb:.2f}")
print(f"sum: {p_rr + p_rb + p_br + p_bb:.2f}")

Output:

P(RR) = 0.13, P(RB) = 0.27, P(BR) = 0.27, P(BB) = 0.33
sum: 1.00

The probabilities of all outcomes always sum to 1 - a quick sanity check.

Independence

Two events are independent if knowing one happened does not change the probability of the other:

P(B | A) = P(B)   (independence)

• Independent: coin flips, dice rolls, two separate website visitors.
• Dependent: drawing cards without replacement, disease prevalence given a test result.

Bayes' Rule (Preview)

Sometimes you know P(B | A) but need P(A | B). Bayes' rule flips the condition:

P(A | B) = P(B | A) * P(A) / P(B)

Example: Spam Filter

# Known: 2% of emails are spam; 95% of spam contains "free";
10% of all emails contain "free". What is P(spam | contains "free")?
p_spam = 0.02
p_free_given_spam = 0.95
p_free = 0.10
p_spam_given_free = (p_free_given_spam * p_spam) / p_free
print(f"P(spam | 'free') = {p_spam_given_free:.2f}")

Output:

P(spam | 'free') = 0.19

Even though "free" is common in spam, most emails containing "free" are not spam - because the base rate of spam is low. This is the famous base rate fallacy, and it trips up beginners constantly.

Common Pitfalls

• Adding probabilities of non-exclusive events without subtracting the overlap.
• Treating dependent events as independent (drawing without replacement!).
• Ignoring base rates when interpreting conditional probabilities.

Summary

• Probability runs from 0 to 1; complements sum to 1.
• Addition rule: P(A or B) = P(A) + P(B) - P(A and B).
• Multiplication rule: P(A and B) = P(A) * P(B | A).
• Conditional probability updates beliefs with new information.
• Base rates matter: P(A | B) is not the same as P(B | A).

Next Lesson

The most important distribution in statistics is next: the normal distribution, z-scores, and the 68-95-99.7 rule.