Shannon's Entropy
H(X) = -sum(p log p). Shannon showed that information has a precise mathematical quantity, bridging physics and communication.
Shannon entropy H(X) = -sum(p(x) log p(x)) measures the average surprise or uncertainty of a random variable. A fair coin has H = 1 bit; a loaded coin has less. This elegant formula, independently discovered in the same mathematical form as Boltzmann's thermodynamic entropy, quantifies the fundamental limits of data compression (source coding theorem) and transmission (noisy channel coding theorem). Shannon entropy is the foundation of modern compression algorithms, error-correcting codes, and the theoretical underpinning of machine learning's cross-entropy loss. The deep connection between physical entropy and informational entropy suggests that information is not just abstract — it is physical.