Entropy, Information and Uncertainty

Why uncertainty can be measured#

Information theory begins with a restrained but powerful question: how much uncertainty is resolved when an outcome becomes known? A result that was nearly certain carries little surprise. A result drawn from many equally plausible alternatives carries more.

For a discrete random variable $X$ with possible outcomes $x_1, x_2, \ldots, x_n$, Shannon entropy is defined as

The logarithm uses base two when information is measured in bits. The negative sign ensures that the sum is non-negative because $0 < p(x_i) \leq 1$ implies $\log_2 p(x_i) \leq 0$.

The fair coin#

For a fair coin,

Therefore,

One binary decision resolves the uncertainty completely. A biased coin has lower entropy because one outcome is easier to anticipate.

export function entropy(probabilities: readonly number[]): number {
  const total = probabilities.reduce((sum, probability) => sum + probability, 0)
 
  if (Math.abs(total - 1) > 1e-9) {
    throw new Error("Probabilities must sum to one")
  }
 
  return -probabilities.reduce((information, probability) => {
    if (probability === 0) return information
    return information + probability * Math.log2(probability)
  }, 0)
}

Maximum entropy#

Among all distributions over $n$ outcomes, entropy is greatest when every outcome has equal probability:

This result explains why uniform distributions represent maximum uncertainty when no additional constraints distinguish one outcome from another.

From equation to practice#

Entropy appears in compression, decision trees, statistical learning, cryptography, and experimental design. Its usefulness comes from turning uncertainty into a quantity that can be compared, optimized, and decomposed.

Closing observation#

The elegance of entropy lies in its economy. A single expectation combines probability, surprise, and the practical cost of representation. It is both a theorem-bearing mathematical object and a disciplined language for talking about what is not yet known.