> For the complete documentation index, see [llms.txt](https://isubasinghe.gitbook.io/isithas-wiki/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://isubasinghe.gitbook.io/isithas-wiki/computer_science/compression/entropy.md).

# Entropy

## Basic idea

Shannon entropy measures the average information content (in bits) of a random variable. It is the lower bound on the average code length for lossless compression.

### Shannon entropy

![H(X)=-\sum\_{i=1}^{n} P(x\_i) \log{P(x\_i)}](https://i.upmath.me/svg/H%28X%29%3D-%5Csum_%7Bi%3D1%7D%5E%7Bn%7D%20P%28x_i%29%20%5Clog%7BP%28x_i%29%7D)

### Joint entropy

![H(X,Y)=-\sum\_{x \in X}\sum\_{y \in Y} P(x,y) \log{P(x,y)}](https://i.upmath.me/svg/H%28X%2CY%29%3D-%5Csum_%7Bx%20%5Cin%20X%7D%5Csum_%7By%20%5Cin%20Y%7D%20P%28x%2Cy%29%20%5Clog%7BP%28x%2Cy%29%7D)

### Conditional entropy

![H(Y|X)=-\sum\_{x \in X}\sum\_{y \in Y} P(x,y) \log{P(y|x)}](https://i.upmath.me/svg/H%28Y%7CX%29%3D-%5Csum_%7Bx%20%5Cin%20X%7D%5Csum_%7By%20%5Cin%20Y%7D%20P%28x%2Cy%29%20%5Clog%7BP%28y%7Cx%29%7D)

### Mutual information

![I(X;Y)=H(X)+H(Y)-H(X,Y)](https://i.upmath.me/svg/I%28X%3BY%29%3DH%28X%29%2BH%28Y%29-H%28X%2CY%29)

### Kullback–Leibler divergence

![D\_{KL}(P|Q)=\sum\_{x \in X} P(x) \log{\frac{P(x)}{Q(x)}}](https://i.upmath.me/svg/D_%7BKL%7D%28P%5C%7CQ%29%3D%5Csum_%7Bx%20%5Cin%20X%7D%20P%28x%29%20%5Clog%7B%5Cfrac%7BP%28x%29%7D%7BQ%28x%29%7D%7D)

### Cross-entropy

![H(P,Q)=-\sum\_{x \in X} P(x) \log{Q(x)}](https://i.upmath.me/svg/H%28P%2CQ%29%3D-%5Csum_%7Bx%20%5Cin%20X%7D%20P%28x%29%20%5Clog%7BQ%28x%29%7D)