Edit a discrete source distribution, see the entropy H(X) update. Build a Huffman or Shannon-Fano code; check the Kraft inequality, the average code length, and the coding efficiency. Compare BSC, BEC, and Z-channel capacities as you vary the error/erasure probability. Plot Shannon-Hartley capacity C = B log₂(1 + S/N) across SNR for different bandwidths. Built to address the common confusions: H(X) vs log₂(N), efficiency vs redundancy, Kraft inequality vs completeness, capacity vs spectral efficiency.
⚠ Common confusion: H(X) ≠ log₂(N). H(X) = −Σ p log₂ p, and equals log₂(N) only when the source is uniform. For a skewed source, H(X) is much less than log₂(N) — that's why Huffman works.
① Source distribution & entropy H(X)
H(X) = −Σ pi log₂ pi bits/symbol. Bars show pi; the H(X) marker on the y-axis shows how much of the "log₂(N) ceiling" the source uses. For uniform: H(X) = log₂(N) (max). For deterministic (one symbol with p=1): H(X) = 0. Most natural sources (text, speech, images) are between these extremes.
pi (probability) H(X) (in bits)
H(p) for a binary source (one symbol with probability p, the other with 1−p). H peaks at p = 0.5.
② Source code (Huffman / Shannon-Fano)
Edit the source distribution in the left panel. Pick Huffman (optimal prefix code) or Shannon-Fano (sub-optimal, but historical). The code tree on the right shows the merge sequence; the code table below shows the codeword, length, and contribution to L̄. The KV panel shows L̄ (average length), η = H(X) / L̄ (efficiency), and 1 − η (redundancy). By Shannon's source-coding theorem, L̄ ≥ H(X) with equality only for the ideal Shannon code; Huffman gives L̄ < H(X) + 1.
The channel capacity C is the maximum mutual information I(X; Y) over all input distributions p(x). For BSC, optimal p(x) is uniform. For Z-channel, the optimum is asymmetric. For generic 2×2 channels we use the Blahut-Arimoto algorithm. The plot shows C as a function of the error/erasure probability for the selected channel type.
BSC: C = 1 − H(p) BEC: C = 1 − ε Z-channel selected (current slider)
Generic 2×2 channel matrix visualization (current selection). Rows = input, columns = output. Hover the cell to see p(y | x).
④ Shannon-Hartley theorem (AWGN channel capacity)
C = B · log₂(1 + S/N) bits/second. This is the absolute maximum error-free bit rate over a bandlimited AWGN channel with bandwidth B and signal-to-noise ratio S/N. It is achievable with capacity-approaching codes (LDPC, Turbo, Polar) at infinite block length. The spectral efficiency C/B = log₂(1 + S/N) is the bits/s/Hz your modulation+code can approach. Practical systems (cellular, WiFi, satellite) sit 5-10 dB from this limit.
📖 Deep-dive blog (planned)
Shannon's source-coding theorem — the noiseless-coding theorem (1948): L̄ ≥ H(X), and Huffman achieves L̄ < H(X) + 1. Why this is almost optimal and when it's exact.
Kraft inequality — why a prefix code's lengths must satisfy Σ 2−ℓᵢ ≤ 1, and why every valid length sequence can be realized as a prefix code.
Huffman optimality proof — the swap argument: any optimal tree can be transformed so the two least-probable symbols are siblings, and the bottom-up merge is greedy-optimal.
BSC vs BEC vs Z-channel — three canonical DMCs, their capacities, and why BEC is the "cleanest" (linear capacity, no input optimization needed).
Blahut-Arimoto algorithm — iterative computation of DMC capacity for arbitrary K×K channels. The general case beyond 2×2.
Shannon-Hartley derivation — from sphere-packing in N dimensions to log₂(1 + S/N). The "5 dB gap" between theory and practice.
Channel coding theorem (1948) — the noisy coding theorem: any rate R < C is achievable, any rate R > C has BER → ½. Capacity-approaching codes (LDPC, Turbo, Polar).
Status: scaffold shipped (engine + page). Blog content to follow.
Source (live)
Distribution
N symbols:—
Σ pi:—
max pi:—
Entropy
H(X) (measured):—
Hmax = log₂ N:—
H(X) / Hmax:—
Redundancy 1 − H/Hmax:—
Source coding (live)
Algorithm
Code:—
Codebook
Avg length L̄:—
H(X):—
η = H / L̄:—
1 − η (redundancy):—
L̄ − H (gap):—
Kraft Σ 2−ℓᵢ:—
Prefix-free?—
Channel capacity (live)
Selected channel
Type:—
Parameter:—
Capacity C:—
Shannon-Hartley (live)
AWGN
Bandwidth B:—
SNR:—
S/N (linear):—
C = B log₂(1 + S/N):—
Spectral eff. C/B:—
🚫 Pitfall: Don't confuse channel capacity with data rate. Capacity is the maximum error-free bit rate — achievable only with capacity-approaching codes at infinite block length. Real systems (5G, WiFi 6) sit a few dB from capacity.
⚠ Kraft vs completeness: Kraft inequality Σ 2−ℓᵢ ≤ 1 is necessary for a prefix-free code but not sufficient for a code to exist — every length sequence satisfying Kraft can be realized as a prefix code (the converse is the sufficiency). Equality holds for a complete (instantly decodable, no unused code space) prefix code.
⚠ Source rate vs channel rate: A Huffman code compresses the source to an average of L̄ bits/symbol; the channel must carry this at a rate R < C. Together: compression (L̄) + channel coding (rate k/n) give end-to-end rate L̄ · n / k bits/channel-use.