Skip to main content

Cross-Entropy Loss for Classification

Understand cross-entropy loss for classification: interactive demos of binary and multi-class CE, the -log(p) curve, softmax gradients, and focal loss.

Cross-Entropy Loss: The Language of Classification

Cross-entropy loss is the default objective for training classification models in deep learning. It measures the gap between a model's predicted probability distribution and the true labels, rooted in a simple idea from information theory: how surprised are you when the true answer is revealed? A confident and correct model is barely surprised. A confident and wrong model is maximally surprised. Cross-entropy quantifies this surprise and turns it into a smooth, differentiable signal that drives learning.

What makes cross-entropy special among loss functions is the elegance of its gradients. When combined with softmax, the gradient simplifies to p_i - y_i — just the difference between prediction and truth. This is the derivative with respect to each logit. It avoids an additional saturating output factor for confidently wrong predictions; gradients with respect to parameters still involve the network Jacobian.

The Surprise Analogy

Think of classification as a guessing game. Before seeing the answer, a model assigns probabilities to each possible class. When the true class is revealed, the model experiences "surprise" proportional to how unlikely it considered that outcome. If the model assigned 95% probability to the correct class, the surprise is tiny. If it assigned 5%, the surprise is enormous. Cross-entropy loss is exactly this surprise, measured in nats (using the natural logarithm) or bits (using log base 2). Training a classifier means teaching it to be less surprised by the training data.

Compare confident, uncertain and wrong models, then press Play to reveal the surprise for each animal.

The surprise guessing game
Model
Step 1 / 6

The model assigns high probability to the correct class — low surprise, low loss

True Answer: Cat

🐱 Cat1.0
🐶 Dog0.0
🐦 Bird0.0
🐟 Fish0.0

Model's Predicted Probabilities

🐱 Cat85%
🐶 Dog10%
🐦 Bird3%
🐟 Fish2%
True distributionModel prediction

Mathematical Definition

Binary Cross-Entropy

For a single sample with true label y \in \{0, 1\} and predicted probability \hat{y}:

\mathcal{L} = -y \log(\hat{y}) - (1-y) \log(1-\hat{y})

When y = 1, only the first term is active: -\log(\hat{y}). When y = 0, only the second term matters: -\log(1 - \hat{y}). In both cases, the loss decreases as the predicted probability moves closer to the true label.

Categorical Cross-Entropy

For multi-class classification with one-hot encoded labels over C classes:

\mathcal{L} = -\sum_{i=1}^{C} y_i \log(\hat{y}_i)

Since y is one-hot, only one term survives — the negative log-probability of the true class. This is equivalent to the negative log-likelihood under a categorical distribution, which connects cross-entropy directly to maximum likelihood estimation.

Connection to Information Theory

Cross-entropy between distributions P (truth) and Q (model) measures the average number of bits needed to encode events from P using Q's coding scheme:

H(P, Q) = -\sum_{x} P(x) \log Q(x)

This always satisfies H(P, Q) \geq H(P), with equality only when P = Q. The excess H(P, Q) - H(P) is the KL divergence — the wasted bits from using the wrong code. Since H(P) is constant during training, minimizing cross-entropy is equivalent to minimizing KL divergence between the model's predictions and the true distribution.

Adjust the predicted probabilities and watch how cross-entropy loss responds on the -log(p) curve. The rose dot shows the current operating point.

Cross-entropy explorer
True Class
0.70
0.20
0.10
Cat (true)70.0%
Dog 20.0%
Bird 10.0%
Negative log probability curve and selected operating point
-log(p)Current operating point

Moderate confidence. The loss gradient is gentle here, so learning will be steady but not urgent.

Binary Cross-Entropy

Binary cross-entropy (also called log loss) handles two-class problems. The model outputs a single probability p representing the likelihood of the positive class. The loss has a characteristic asymmetric shape: it rises gently when the model is slightly wrong but explodes toward infinity when the model is confidently wrong.

For y=1, the derivative with respect to probability is \partial L/\partial p=-1/p. Through a sigmoid logit z, it simplifies to \partial L/\partial z=p-y. These are different derivative variables: the first diverges as p approaches zero, while the second stays bounded and tends to zero for a confident correct prediction.

In practice, binary cross-entropy is always computed from raw logits (pre-sigmoid values) rather than probabilities. Combining the sigmoid and log operations into a single numerically stable function avoids the catastrophic cancellation that occurs when computing \log(\sigma(z)) for large negative z.

Toggle the true label and drag the predicted probability to inspect both loss curves. The amber arrow shows gradient direction and magnitude.

Binary cross-entropy
True Label (y)
0.70
Both binary cross-entropy curves, active point and signed gradient arrow
y=1: -log(p)y=0: -log(1-p)Gradient direction and magnitude

0.01 · Decision boundary (0.5) · 0.99

Correct side of the boundary but not fully confident. The gradient is moderate, pushing the model toward higher confidence.

Multi-Class Cross-Entropy

For problems with more than two classes, the model produces a vector of logits that passes through softmax to create a probability distribution. The softmax function converts raw scores into probabilities that sum to one:

\text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{C} e^{z_j}}

The combined softmax + cross-entropy loss yields the most elegant gradient in deep learning:

\frac{\partial \mathcal{L}}{\partial z_i} = p_i - y_i

For the true class, the gradient is p_{\text{true}} - 1, always negative, pushing the logit up. For every other class, the gradient equals the softmax probability itself, always positive, pushing those logits down. The magnitude is proportional to how wrong the prediction is — larger errors produce stronger corrections.

For an unweighted individual example, each logit derivative lies in [-1,1]. It approaches zero when the prediction matches the target and remains substantial for a confidently wrong true-class prediction. Parameter gradients multiply these values by derivatives of the logits, so hidden or weight gradients can still vanish or explode.

Adjust raw logits for each class and watch the softmax probabilities, per-class loss contributions and gradients update.

Multi-class cross-entropy
True Class
Preset
3.5
0.8
0.2
-0.5
🐱 Cat (true)Logit: 3.5 · Softmax: 89.1%
Loss: 0.115Grad: -0.109
🐶 Dog Logit: 0.8 · Softmax: 6.0%
Loss: 0.000Grad: +0.060
🐦 Bird Logit: 0.2 · Softmax: 3.3%
Loss: 0.000Grad: +0.033
🐟 Fish Logit: -0.5 · Softmax: 1.6%
Loss: 0.000Grad: +0.016
CatDogBirdFish

The gradient for softmax + cross-entropy has an elegant form: gradient = softmax(z) - y. For the true class (y=1), the gradient is -0.109 (negative, pushing the logit up). For all other classes (y=0), the gradient equals the softmax probability itself — always positive, pushing those logits down.

Correct prediction but not fully confident. Cross-entropy will keep pushing the true class logit higher relative to the others.

Comparing Classification Losses

Cross-entropy is the default, but it is not the only option. Different classification losses make different tradeoffs between gradient quality, class imbalance handling, and calibration. Understanding these tradeoffs helps you choose the right tool for your specific problem.

Compare the losses across imbalance, calibration, numerical stability and gradient quality.

Classification losses compared

Cross-Entropy

-y log(p)

Gradient: p - y

Imbalance
moderate
Weight classes manually
Calibration
excellent
Penalizes underconfidence for true class
Stability
excellent
Stable with log-sum-exp trick
Gradients
excellent
Gradient never vanishes

Best for: General classification — the default choice for most tasks

MSE (L2)

(y - p)^2

Gradient: 2(p - y)

Imbalance
poor
No built-in mechanism
Calibration
poor
Slow gradient when p is far from y
Stability
excellent
No log operations
Gradients
poor
Vanishes near 0 and 1 with sigmoid

Best for: Regression tasks — not recommended for classification

Focal Loss

-(1-p)^gamma log(p)

Gradient: Complex (modulated)

Imbalance
excellent
Down-weights easy examples
Calibration
excellent
Focuses on hard examples
Stability
moderate
Extra terms add complexity
Gradients
excellent
Strong signal for hard cases

Best for: Object detection, severe class imbalance (RetinaNet)

Hinge Loss

max(0, 1 - y*f(x))

Gradient: -y or 0

Imbalance
moderate
Margin-based, not probability
Calibration
moderate
No gradient once margin met
Stability
excellent
Simple max operation
Gradients
moderate
Zero gradient for correct predictions

Best for: SVMs, max-margin classifiers

Label Smoothing CE

-Σᵢ [(1-ε)yᵢ + ε/C] log(pᵢ)

Gradient: dL/dzᵢ = pᵢ - [(1-ε)yᵢ + ε/C]

Imbalance
moderate
Same as CE
Calibration
excellent
Prevents overconfidence
Stability
excellent
Same as CE
Gradients
excellent
Never fully satisfied

Best for: Large-scale classification, knowledge distillation, calibration

Use cross-entropy when...

  • Training any standard classification model
  • You want well-calibrated probability outputs
  • Class imbalance is mild to moderate
  • You need reliable gradients throughout training

Consider alternatives when...

  • Severe class imbalance exists (use focal loss)
  • You need max-margin separation (use hinge loss)
  • Overconfidence is a problem (use label smoothing)
  • Doing regression, not classification (use MSE/MAE)

Connection to KL Divergence

Minimizing cross-entropy loss is mathematically equivalent to minimizing the KL divergence between the true distribution and the model's predicted distribution. Since the entropy of the true labels H(P) is fixed (zero for hard labels, positive for soft labels), the only way to reduce H(P, Q) is to reduce the KL divergence term D_{KL}(P \| Q) = H(P, Q) - H(P).

This equivalence has a profound implication: cross-entropy training is maximum likelihood estimation in disguise. Maximizing the likelihood of the data under the model is the same as minimizing the cross-entropy between the empirical data distribution and the model's predictions. This is why cross-entropy is not just a heuristic — it is the principled, information-theoretically optimal loss for classification.

When using label smoothing, the true distribution is no longer one-hot but a mixture: y'_i = (1-\epsilon) \cdot y_i + \epsilon / C. This makes H(P) positive, and the model can never drive the loss to zero. The smoothed targets act as a regularizer, preventing the model from becoming overconfident and improving generalization — especially in large-scale classification with many classes.

Common Pitfalls

Numerical Instability

Computing \log(p) directly from softmax probabilities is dangerous. When a logit is very large and negative, the softmax output underflows to zero, and \log(0) produces negative infinity. The solution is the log-sum-exp trick: compute cross-entropy directly from logits as -z_{\text{true}} + \log \sum_j e^{z_j}, subtracting the maximum logit for numerical stability. Every major deep learning framework implements this combined operation internally.

Wrong Loss for the Problem

Using mean squared error for classification seems reasonable but fails in practice. The gradient of MSE through sigmoid saturates when the model is confidently wrong — exactly when you need the strongest learning signal. The logit derivative of sigmoid plus cross-entropy avoids that extra sigmoid factor for confidently wrong predictions. This does not guarantee faster convergence for every model.

Ignoring Class Imbalance

Standard cross-entropy treats all classes equally. When one class dominates the dataset, the model learns to predict that class for everything and achieves low loss. Solutions include class-weighted cross-entropy (scaling each class's loss by its inverse frequency), focal loss (down-weighting easy examples), or oversampling the minority class.

Probability Clipping

When implementing cross-entropy manually, always clip predicted probabilities away from zero and one. A prediction of exactly zero for the true class produces infinite loss and NaN gradients. A small epsilon (typically 1e-7) prevents this while having negligible effect on the loss value.

Confusing Loss Variants

Binary cross-entropy expects a single sigmoid output per sample. Categorical cross-entropy expects a softmax distribution. Using the wrong variant — for example, applying binary cross-entropy to softmax outputs — produces incorrect gradients and poor training. Multi-label classification (where samples can belong to multiple classes) requires binary cross-entropy applied independently to each class, not categorical cross-entropy.

If you found this explanation helpful, consider sharing it with others.

Mastodon