Information Theory

Notes for a hypothetical graduate course in information theory in the Department of Statistics at the University of Auckland.

(c) 2026 Brendon J. Brewer

LICENCE: CC-BY-SA 4.0 International. See LICENCE file for details.

Course Outline

Week 1

• Review of probability theory and probability distributions.
• Historical background.
• Definition and basic properties of Shannon entropy (e.g., non-negativity).

Week 2

• Interpretation of Shannon entropy in discrete and continuous cases.
• Joint entropy, conditional entropy, and mutual information.
• Relative entropy, cross entropy, and Kullback-Leibler divergence.

Week 3

• Derivation of entropies of some common distributions.
• Simple applications --- quantifying uncertainty and relevance.

Week 4

• Infinitesimal KL divergence and the Fisher metric.
• Jeffreys priors.

Week 5

• Markov chains and entropy rates.
• Asymptotic equipartition property.

Week 6

• Communication channels.
• Noisy channel theorem.
• Binary symmetric channel and AWGN channel examples.

Week 7

• Linear block codes and generator matrices.
• Hamming codes and syndrome decoding.
• Repetition codes and simple decoding rules.
• Relationship between codes and channel capacity.

Week 8

• The principle of maximum entropy.
• Updating probabilities with maximum entropy.
• Canonical distributions.
• Bayesian updating, Jeffrey conditionalisation, entropic priors.

Week 9

• Statistical mechanics.
• Liouville's theorem.
• The second law of thermodynamics.

Week 10

• Lossy compression and distortion measures.
• Rate-distortion function and its interpretation.
• Trade‑offs between fidelity and compression.

Week 11

• Applications in modern statistics and machine learning.
• Variational inference and ELBO.
• Bayesian experimental design.
• Logarithmic scoring rules.

Week 12

• Foundations of probability: ordering of statements.
• Foundations of entropy: ordering of questions.
• Summary and unifying themes.