Axioms of Probability — Probability Fundamentals | OneNoughtOne

0/106

Machine LearningProbability Fundamentals

Probability Fundamentals

LevelBeginner

Duration90 mins

TopicProbability Fundamentals

2 / 5

Axioms of Probability

Why Probability Needs Axioms

Having established sample spaces and events as the vocabulary of probability, we now face a fundamental question: How do we assign numbers to events in a consistent, meaningful way? Intuitively, we might say 'probability is the frequency with which an event occurs if we repeat the experiment many times.' This frequentist interpretation is useful but immediately runs into problems: What about events that can't be repeated (e.g., 'Will this startup succeed?')? How do we know the limiting frequency actually converges? Different sequences of experiments might converge to different limits. Alternatively, we might interpret probability as 'degree of belief'—a Bayesian view. But then: How do we ensure different people's beliefs are consistent? What constrains valid beliefs? The solution, developed by Andrey Kolmogorov in 1933, was to axiomatize probability —to specify minimal rules that any valid probability assignment must satisfy. These axioms don't tell us what probability means; they tell us how probability must behave.

What You Will Learn

By the end of this page, you will understand Kolmogorov's axioms of probability, derive fundamental probability rules from these axioms, and recognize how these abstract principles underpin concrete ML computations like loss functions, model calibration, and probabilistic predictions.

Kolmogorov's Axioms: The Foundation

Let (Ω, ℱ) be a measurable space where Ω is the sample space and ℱ is a σ-algebra of events. A probability measure P is a function P: ℱ \to [0, 1] that assigns a number to each event, satisfying three axioms: Axiom 1: Non-Negativity For any event A \in ℱ: P(A) \geq 0 Probabilities cannot be negative. This seems obvious but is essential for interpretation—a negative probability has no meaning. Axiom 2: Normalization P(Ω) = 1 The probability that something happens is 1. The sample space, containing all possible outcomes, has probability 1 because exactly one outcome must occur. Axiom 3: Countable Additivity (σ-additivity) For any countable sequence of mutually exclusive events A₁, A₂, A₃, ... \in ℱ (i.e., Aᵢ \cap Aⱼ = \emptyset for i \neq j): P(\cupᵢ₌₁^\infty Aᵢ) = Σᵢ₌₁^\infty P(Aᵢ) If events cannot overlap, the probability of 'at least one occurring' equals the sum of their individual probabilities. This extends to infinitely many events, which is crucial for continuous probability. The triple (Ω, ℱ, P) is called a probability space .

The Beauty of Minimalism

These three axioms appear deceptively simple, yet they are sufficient to derive ALL of probability theory. Every theorem about expectations, variances, conditional probabilities, limit theorems, and more follows from these three statements. This is the power of axiomatic mathematics: from minimal assumptions, vast structures emerge.

Kolmogorov's Axioms Summary
Axiom	Statement	Intuition	Why Essential
Non-Negativity	P(A) ≥ 0 for all events A	Probability measures 'how likely'—can't be negative	Enables interpretation as proportion/frequency
Normalization	P(Ω) = 1	Something must happen	Provides reference scale for all probabilities
Countable Additivity	P(∪ Aᵢ) = Σ P(Aᵢ) for disjoint Aᵢ	Non-overlapping events add up	Allows infinite sums; essential for continuous distributions

Immediate Consequences of the Axioms

From Kolmogorov's three axioms, we can derive many fundamental properties of probability. These derivations demonstrate the power of axiomatic reasoning—each property is a logical consequence, not an additional assumption. Property 1: Probability of the Empty Set Theorem: P(\emptyset) = 0 Proof: Consider the countable sequence: A₁ = Ω, A₂ = \emptyset, A₃ = \emptyset, A₄ = \emptyset, ... These are disjoint (\emptyset doesn't overlap with anything) Ω \cup \emptyset \cup \emptyset \cup ... = Ω By Axiom 3: P(Ω) = P(Ω) + P(\emptyset) + P(\emptyset) + ... By Axiom 2: 1 = 1 + P(\emptyset) + P(\emptyset) + ... The only way this holds (with P(\emptyset) \geq 0 by Axiom 1) is if P(\emptyset) = 0 ∎ Interpretation: An impossible event has zero probability—nothing can occur outside all possibilities.

Property 2: Complement Rule Theorem: P(A^c) = 1 - P(A) Proof: A and A^c are disjoint (A \cap A^c = \emptyset) A \cup A^c = Ω (together they cover everything) By Axiom 3: P(A \cup A^c) = P(A) + P(A^c) By Axiom 2: P(Ω) = 1 Therefore: 1 = P(A) + P(A^c) Rearranging: P(A^c) = 1 - P(A) ∎ Interpretation: The probability of 'not A' equals one minus the probability of A. If there's a 70% chance of rain, there's a 30% chance of no rain. ML Application: If a classifier predicts class 1 with probability 0.8, it implicitly predicts class 0 (in binary classification) with probability 0.2. This is the foundation of probabilistic predictions.

Property 3: Probability Bounds Theorem: For any event A, 0 \leq P(A) \leq 1 Proof: P(A) \geq 0 by Axiom 1 (lower bound) P(A^c) \geq 0 by Axiom 1 P(A) = 1 - P(A^c) \leq 1 (upper bound) ∎ Interpretation: Probabilities are always between 0 and 1. This seems obvious but is a derived property, not an axiom. ML Application: This is why neural network outputs for classification use softmax or sigmoid—to constrain outputs to [0, 1] and be interpretable as probabilities.

Property 4: Monotonicity Theorem: If A ⊆ B, then P(A) ≤ P(B) Proof: If A ⊆ B, then B = A ∪ (B \ A) A and (B \ A) are disjoint By Axiom 3: P(B) = P(A) + P(B \ A) P(B \ A) ≥ 0 by Axiom 1 Therefore: P(B) ≥ P(A) ∎ Interpretation: If event A is contained in event B (A implies B), then A is at most as likely as B. 'Drawing an ace' is less likely than 'drawing a face card or ace.' ML Application: If your model's correct predictions are a subset of another model's correct predictions, the second model has higher accuracy.

The Art of Proof

Notice how each proof uses only the three axioms and previously proven properties. This chain of logical deduction is the essence of mathematics. Every probability theorem you'll ever use—from Bayes' theorem to the central limit theorem—ultimately traces back to these three axioms.

The Addition Rule: Unions of Events

Axiom 3 tells us how to compute P(A \cup B) when A and B are disjoint. But what if A and B overlap? We need the General Addition Rule (also called the Inclusion-Exclusion Principle for two sets). General Addition Rule for Two Events Theorem: For any events A and B: P(A \cup B) = P(A) + P(B) - P(A \cap B) Proof: We decompose A \cup B into three disjoint pieces: A \cup B = (A \cap B^c) \cup (A \cap B) \cup (B \cap A^c) But let's use a simpler approach: B = (A \cap B) \cup (B \cap A^c), and these are disjoint So P(B) = P(A \cap B) + P(B \cap A^c) Thus P(B \cap A^c) = P(B) - P(A \cap B) Also: A \cup B = A \cup (B \cap A^c), and these are disjoint So P(A \cup B) = P(A) + P(B \cap A^c) = P(A) + P(B) - P(A \cap B) ∎

Why Subtract the Intersection?

When we add P(A) + P(B), outcomes in the intersection A \cap B get counted twice—once when counting A, once when counting B. Subtracting P(A \cap B) corrects this double-counting. This is the intuition behind inclusion-exclusion.

Addition Rule: Email Classification

A spam filter flags emails based on multiple criteria. Let's compute overlap probabilities.

Scenario: 25% of emails contain suspicious links (event A) 15% of emails contain suspicious words (event B) 10% of emails contain both suspicious links AND words (A \cap B) Question: What percentage of emails trigger at least one spam criterion? Solution using Addition Rule: P(A \cup B) = P(A) + P(B) - P(A \cap B) = 0.25 + 0.15 - 0.10 = 0.30 Answer: 30% of emails trigger at least one criterion. Without subtracting the intersection: P(A) + P(B) = 0.40 — this would overcount by 10%!

Inclusion-Exclusion for Three Events P(A \cup B \cup C) = P(A) + P(B) + P(C) - P(A \cap B) - P(A \cap C) - P(B \cap C) + P(A \cap B \cap C) The pattern: add singles, subtract pairs, add triples. This extends to n events: P(\cupᵢ₌₁ⁿ Aᵢ) = Σᵢ P(Aᵢ) - Σᵢ<ⱼ P(Aᵢ \cap Aⱼ) + Σᵢ<ⱼ<ₖ P(Aᵢ \cap Aⱼ \cap Aₖ) - ... + (-1)ⁿ⁺¹ P(\capᵢ₌₁ⁿ Aᵢ)

inclusion_exclusion.py
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
from typing import List, Callable, Set, Any
from itertools import combinations
 
def probability_of_union(
    events: List[Set[Any]], 
    sample_space: Set[Any]
) -> float:
    """
    Compute P(A₁ ∪ A₂ ∪ ... ∪ Aₙ) using inclusion-exclusion.
    
    For finite sample spaces, this computes exact probabilities
    by working with the sets directly.
    
    Parameters:
        events: List of event sets
        sample_space: The complete sample space Ω
    
    Returns:
        Probability of at least one event occurring
    """
    n = len(events)
    if n == 0:
        return 0.0
    
    omega_size = len(sample_space)
    total_prob = 0.0
    
    for k in range(1, n + 1):
        # Sum over all k-subsets of events
        sign = (-1) ** (k + 1)  # Alternating signs
        
        for event_subset in combinations(events, k):
            # Compute intersection of all events in subset
            intersection = event_subset[0]
            for event in event_subset[1:]:
                intersection = intersection & event
            
            # Add (or subtract) probability of this intersection
            total_prob += sign * len(intersection) / omega_size
    
    return total_prob
 
 
def probability_of_at_least_one_simple(
    events: List[Set[Any]], 
    sample_space: Set[Any]
) -> float:
    """
    Alternative: compute P(at least one) by finding the actual union.
    
    Simpler but less illustrative of inclusion-exclusion.
    """
    union = set()
    for event in events:
        union = union | event
    return len(union) / len(sample_space)
 
 
# Example: Two dice, multiple events
if __name__ == "__main__":
    # Sample space for two dice
    Omega = {(i, j) for i in range(1, 7) for j in range(1, 7)}
    
    # Define events
    A = {(i, j) for (i, j) in Omega if i + j == 7}  # Sum is 7
    B = {(i, j) for (i, j) in Omega if i == j}      # Doubles
    C = {(i, j) for (i, j) in Omega if i >= 5}      # First die ≥ 5
    
    print("Event A (sum = 7):", len(A), "outcomes")
    print("Event B (doubles):", len(B), "outcomes")
    print("Event C (first ≥ 5):", len(C), "outcomes")
    
    # Two-event inclusion-exclusion
    p_a = len(A) / len(Omega)
    p_b = len(B) / len(Omega)
    p_a_and_b = len(A & B) / len(Omega)
    
    print(f"
P(A) = {p_a:.4f}")
    print(f"P(B) = {p_b:.4f}")
    print(f"P(A ∩ B) = {p_a_and_b:.4f}")
    print(f"P(A ∪ B) via inclusion-exclusion = {p_a + p_b - p_a_and_b:.4f}")
    print(f"P(A ∪ B) via direct computation = {len(A | B) / len(Omega):.4f}")
    
    # Three-event inclusion-exclusion
    print(f"
P(A ∪ B ∪ C) via function = {probability_of_union([A, B, C], Omega):.4f}")
    print(f"P(A ∪ B ∪ C) via direct = {probability_of_at_least_one_simple([A, B, C], Omega):.4f}")

Continuity of Probability

A powerful consequence of countable additivity (Axiom 3) is that probability behaves 'continuously' with respect to limits of events. This is crucial for working with infinite sequences and continuous distributions. Monotone Convergence for Events Theorem (Continuity from Below): If A₁ ⊆ A₂ ⊆ A₃ ⊆ ... is an increasing sequence of events, and A = ∪ᵢ₌₁^∞ Aᵢ, then: P(A) = lim_{n→∞} P(Aₙ) Theorem (Continuity from Above): If A₁ ⊇ A₂ ⊇ A₃ ⊇ ... is a decreasing sequence of events, and A = ∩ᵢ₌₁^∞ Aᵢ, then: P(A) = lim_{n→∞} P(Aₙ)

Intuition for Continuity

If events get progressively larger (A₁ \subseteq A₂ \subseteq ...) and converge to some limit event A, the probabilities also converge: P(Aₙ) \to P(A). There are no 'jumps' at infinity. This mirrors how continuous functions behave at limits and is essential for defining probability on continuous spaces.

Continuity Example: At Least One Success

Consider repeatedly flipping a fair coin. What's the probability of getting at least one head?

Setup: Flip a fair coin infinitely many times. Define: Aₙ = 'At least one head in the first n flips' This is an increasing sequence: A₁ ⊆ A₂ ⊆ A₃ ⊆ ... The limit A = ∪ₙ Aₙ = 'At least one head ever' Computing P(Aₙ): P(Aₙ) = 1 - P(all tails in first n flips) = 1 - (1/2)ⁿ Applying Continuity: P(A) = lim_{n→∞} P(Aₙ) = lim_{n→∞} [1 - (1/2)ⁿ] = 1 Conclusion: With probability 1, you will eventually get a head. The probability of getting tails forever is 0. Note: P(tails forever) = 0 doesn't mean it's impossible—the outcome TTTTTT... exists in Ω. But it has probability zero, which is fine for events in a continuous or countably infinite space.

Why Countable Additivity Matters The distinction between finite additivity (works for any finite collection of disjoint events) and countable additivity (works for countable collections) is subtle but crucial. Finite additivity alone would not guarantee: Continuity theorems for limits of events Well-defined probabilities for continuous distributions Convergence guarantees needed for statistical inference Kolmogorov's choice of countable additivity as the third axiom was deliberate—it provides just enough structure to build all of probability theory without being so restrictive as to exclude useful probability spaces.

Constructing Probability Measures

The axioms tell us what properties a probability measure must have. But how do we actually construct one? Different constructions lead to different probability spaces. Method 1: Classical (Counting) Probability For finite sample spaces with equally likely outcomes: P(A) = |A| / |Ω| This satisfies all three axioms: P(A) = |A|/|Ω| \geq 0 ✓ P(Ω) = |Ω|/|Ω| = 1 ✓ For disjoint A, B: P(A \cup B) = |A \cup B|/|Ω| = (|A| + |B|)/|Ω| = P(A) + P(B) ✓

Classical Probability Examples
Experiment	Sample Space	Example Event	Probability
Fair coin flip	Ω = {H, T}	A = {H}	P(A) = 1/2
Fair die roll	Ω = {1,2,3,4,5,6}	A = {even} = {2,4,6}	P(A) = 3/6 = 1/2
Drawing a card	Ω = 52 cards	A = {aces} = 4 cards	P(A) = 4/52 = 1/13
Two fair dice	Ω = 36 pairs	A = {sum = 7} = 6 pairs	P(A) = 6/36 = 1/6

Method 2: Assigning Probabilities to Outcomes (Discrete) For discrete sample spaces, we can assign probabilities directly to each outcome: p(ω) = probability of outcome ω Constraints (from axioms): p(ω) ≥ 0 for all ω Σ_{ω ∈ Ω} p(ω) = 1 Then for any event A: P(A) = Σ_{ω ∈ A} p(ω)

Weighted Die (Non-uniform Probability)

A loaded die has different probabilities for each face.

Outcome probabilities:

ω	1	2	3	4	5	6
p(ω)	0.1	0.1	0.1	0.1	0.2	0.4

Verify axioms: All p(ω) \geq 0 ✓ Σ p(ω) = 0.1 + 0.1 + 0.1 + 0.1 + 0.2 + 0.4 = 1.0 ✓ Computing event probabilities: P(even) = P({2, 4, 6}) = p(2) + p(4) + p(6) = 0.1 + 0.1 + 0.4 = 0.6 P(greater than 4) = P({5, 6}) = 0.2 + 0.4 = 0.6 This weighted die shows 6 with probability 0.4—far from the fair 1/6 \approx 0.167.

Method 3: Probability Density Functions (Continuous) For continuous sample spaces like Ω = ℝ, we cannot assign positive probability to individual points (otherwise we'd exceed 1 by summing uncountably many). Instead, we use a probability density function (PDF) f(x): P(a ≤ X ≤ b) = ∫ₐᵇ f(x) dx Constraints: f(x) ≥ 0 for all x ∫_{-∞}^{∞} f(x) dx = 1 The PDF gives probability per unit length, not probability of points. Example: Uniform distribution on [0, 1] f(x) = 1 for x ∈ [0, 1], and f(x) = 0 otherwise P(0.2 ≤ X ≤ 0.5) = ∫_{0.2}^{0.5} 1 dx = 0.3

Common Confusion: PDF vs. Probability

The value f(x) is NOT a probability—it's a density . For continuous random variables: • P(X = 0.5) = 0 (probability of any single point is zero) • f(0.5) can be any non-negative number • f(x) can exceed 1 (e.g., uniform on [0, 0.1] has f(x) = 10) Only integrals of f(x) yield probabilities, and those must be \leq 1.