Cross-Entropy Loss: The Language of Classification
Cross-entropy loss is the default objective for training classification models in deep learning. It measures the gap between a model's predicted probability distribution and the true labels, rooted in a simple idea from information theory: how surprised are you when the true answer is revealed? A confident and correct model is barely surprised. A confident and wrong model is maximally surprised. Cross-entropy quantifies this surprise and turns it into a smooth, differentiable signal that drives learning.
What makes cross-entropy special among loss functions is the elegance of its gradients. When combined with softmax, the gradient simplifies to p_i - y_i — just the difference between prediction and truth. This is the derivative with respect to each logit. It avoids an additional saturating output factor for confidently wrong predictions; gradients with respect to parameters still involve the network Jacobian.
The Surprise Analogy
Think of classification as a guessing game. Before seeing the answer, a model assigns probabilities to each possible class. When the true class is revealed, the model experiences "surprise" proportional to how unlikely it considered that outcome. If the model assigned 95% probability to the correct class, the surprise is tiny. If it assigned 5%, the surprise is enormous. Cross-entropy loss is exactly this surprise, measured in nats (using the natural logarithm) or bits (using log base 2). Training a classifier means teaching it to be less surprised by the training data.
Compare confident, uncertain and wrong models, then press Play to reveal the surprise for each animal.
The model assigns high probability to the correct class — low surprise, low loss
True Answer: Cat
Model's Predicted Probabilities
Mathematical Definition
Binary Cross-Entropy
For a single sample with true label y \in \{0, 1\} and predicted probability \hat{y}:
When y = 1, only the first term is active: -\log(\hat{y}). When y = 0, only the second term matters: -\log(1 - \hat{y}). In both cases, the loss decreases as the predicted probability moves closer to the true label.
Categorical Cross-Entropy
For multi-class classification with one-hot encoded labels over C classes:
Since y is one-hot, only one term survives — the negative log-probability of the true class. This is equivalent to the negative log-likelihood under a categorical distribution, which connects cross-entropy directly to maximum likelihood estimation.
Connection to Information Theory
Cross-entropy between distributions P (truth) and Q (model) measures the average number of bits needed to encode events from P using Q's coding scheme:
This always satisfies H(P, Q) \geq H(P), with equality only when P = Q. The excess H(P, Q) - H(P) is the KL divergence — the wasted bits from using the wrong code. Since H(P) is constant during training, minimizing cross-entropy is equivalent to minimizing KL divergence between the model's predictions and the true distribution.
Adjust the predicted probabilities and watch how cross-entropy loss responds on the -log(p) curve. The rose dot shows the current operating point.
Moderate confidence. The loss gradient is gentle here, so learning will be steady but not urgent.
Binary Cross-Entropy
Binary cross-entropy (also called log loss) handles two-class problems. The model outputs a single probability p representing the likelihood of the positive class. The loss has a characteristic asymmetric shape: it rises gently when the model is slightly wrong but explodes toward infinity when the model is confidently wrong.
For y=1, the derivative with respect to probability is \partial L/\partial p=-1/p. Through a sigmoid logit z, it simplifies to \partial L/\partial z=p-y. These are different derivative variables: the first diverges as p approaches zero, while the second stays bounded and tends to zero for a confident correct prediction.
In practice, binary cross-entropy is always computed from raw logits (pre-sigmoid values) rather than probabilities. Combining the sigmoid and log operations into a single numerically stable function avoids the catastrophic cancellation that occurs when computing \log(\sigma(z)) for large negative z.
Toggle the true label and drag the predicted probability to inspect both loss curves. The amber arrow shows gradient direction and magnitude.
0.01 · Decision boundary (0.5) · 0.99
Correct side of the boundary but not fully confident. The gradient is moderate, pushing the model toward higher confidence.
Multi-Class Cross-Entropy
For problems with more than two classes, the model produces a vector of logits that passes through softmax to create a probability distribution. The softmax function converts raw scores into probabilities that sum to one:
The combined softmax + cross-entropy loss yields the most elegant gradient in deep learning:
For the true class, the gradient is p_{\text{true}} - 1, always negative, pushing the logit up. For every other class, the gradient equals the softmax probability itself, always positive, pushing those logits down. The magnitude is proportional to how wrong the prediction is — larger errors produce stronger corrections.
For an unweighted individual example, each logit derivative lies in [-1,1]. It approaches zero when the prediction matches the target and remains substantial for a confidently wrong true-class prediction. Parameter gradients multiply these values by derivatives of the logits, so hidden or weight gradients can still vanish or explode.
Adjust raw logits for each class and watch the softmax probabilities, per-class loss contributions and gradients update.
The gradient for softmax + cross-entropy has an elegant form: gradient = softmax(z) - y. For the true class (y=1), the gradient is -0.109 (negative, pushing the logit up). For all other classes (y=0), the gradient equals the softmax probability itself — always positive, pushing those logits down.
Correct prediction but not fully confident. Cross-entropy will keep pushing the true class logit higher relative to the others.
Comparing Classification Losses
Cross-entropy is the default, but it is not the only option. Different classification losses make different tradeoffs between gradient quality, class imbalance handling, and calibration. Understanding these tradeoffs helps you choose the right tool for your specific problem.
Compare the losses across imbalance, calibration, numerical stability and gradient quality.
Cross-Entropy
-y log(p)
Gradient: p - y
- Imbalance
- moderateWeight classes manually
- Calibration
- excellentPenalizes underconfidence for true class
- Stability
- excellentStable with log-sum-exp trick
- Gradients
- excellentGradient never vanishes
Best for: General classification — the default choice for most tasks
MSE (L2)
(y - p)^2
Gradient: 2(p - y)
- Imbalance
- poorNo built-in mechanism
- Calibration
- poorSlow gradient when p is far from y
- Stability
- excellentNo log operations
- Gradients
- poorVanishes near 0 and 1 with sigmoid
Best for: Regression tasks — not recommended for classification
Focal Loss
-(1-p)^gamma log(p)
Gradient: Complex (modulated)
- Imbalance
- excellentDown-weights easy examples
- Calibration
- excellentFocuses on hard examples
- Stability
- moderateExtra terms add complexity
- Gradients
- excellentStrong signal for hard cases
Best for: Object detection, severe class imbalance (RetinaNet)
Hinge Loss
max(0, 1 - y*f(x))
Gradient: -y or 0
- Imbalance
- moderateMargin-based, not probability
- Calibration
- moderateNo gradient once margin met
- Stability
- excellentSimple max operation
- Gradients
- moderateZero gradient for correct predictions
Best for: SVMs, max-margin classifiers
Label Smoothing CE
-Σᵢ [(1-ε)yᵢ + ε/C] log(pᵢ)
Gradient: dL/dzᵢ = pᵢ - [(1-ε)yᵢ + ε/C]
- Imbalance
- moderateSame as CE
- Calibration
- excellentPrevents overconfidence
- Stability
- excellentSame as CE
- Gradients
- excellentNever fully satisfied
Best for: Large-scale classification, knowledge distillation, calibration
Use cross-entropy when...
- Training any standard classification model
- You want well-calibrated probability outputs
- Class imbalance is mild to moderate
- You need reliable gradients throughout training
Consider alternatives when...
- Severe class imbalance exists (use focal loss)
- You need max-margin separation (use hinge loss)
- Overconfidence is a problem (use label smoothing)
- Doing regression, not classification (use MSE/MAE)
Connection to KL Divergence
Minimizing cross-entropy loss is mathematically equivalent to minimizing the KL divergence between the true distribution and the model's predicted distribution. Since the entropy of the true labels H(P) is fixed (zero for hard labels, positive for soft labels), the only way to reduce H(P, Q) is to reduce the KL divergence term D_{KL}(P \| Q) = H(P, Q) - H(P).
This equivalence has a profound implication: cross-entropy training is maximum likelihood estimation in disguise. Maximizing the likelihood of the data under the model is the same as minimizing the cross-entropy between the empirical data distribution and the model's predictions. This is why cross-entropy is not just a heuristic — it is the principled, information-theoretically optimal loss for classification.
When using label smoothing, the true distribution is no longer one-hot but a mixture: y'_i = (1-\epsilon) \cdot y_i + \epsilon / C. This makes H(P) positive, and the model can never drive the loss to zero. The smoothed targets act as a regularizer, preventing the model from becoming overconfident and improving generalization — especially in large-scale classification with many classes.
Common Pitfalls
Numerical Instability
Computing \log(p) directly from softmax probabilities is dangerous. When a logit is very large and negative, the softmax output underflows to zero, and \log(0) produces negative infinity. The solution is the log-sum-exp trick: compute cross-entropy directly from logits as -z_{\text{true}} + \log \sum_j e^{z_j}, subtracting the maximum logit for numerical stability. Every major deep learning framework implements this combined operation internally.
Wrong Loss for the Problem
Using mean squared error for classification seems reasonable but fails in practice. The gradient of MSE through sigmoid saturates when the model is confidently wrong — exactly when you need the strongest learning signal. The logit derivative of sigmoid plus cross-entropy avoids that extra sigmoid factor for confidently wrong predictions. This does not guarantee faster convergence for every model.
Ignoring Class Imbalance
Standard cross-entropy treats all classes equally. When one class dominates the dataset, the model learns to predict that class for everything and achieves low loss. Solutions include class-weighted cross-entropy (scaling each class's loss by its inverse frequency), focal loss (down-weighting easy examples), or oversampling the minority class.
Probability Clipping
When implementing cross-entropy manually, always clip predicted probabilities away from zero and one. A prediction of exactly zero for the true class produces infinite loss and NaN gradients. A small epsilon (typically 1e-7) prevents this while having negligible effect on the loss value.
Confusing Loss Variants
Binary cross-entropy expects a single sigmoid output per sample. Categorical cross-entropy expects a softmax distribution. Using the wrong variant — for example, applying binary cross-entropy to softmax outputs — produces incorrect gradients and poor training. Multi-label classification (where samples can belong to multiple classes) requires binary cross-entropy applied independently to each class, not categorical cross-entropy.
Related concepts
Learn focal loss for deep learning: down-weight easy examples, focus on hard ones. Interactive demos of gamma, alpha balancing, and RetinaNet.
Learn KL divergence for machine learning: measure distribution differences in VAEs, knowledge distillation, and variational inference.
Interactive guide to MSE vs MAE for regression: explore outlier sensitivity, gradient behavior, and Huber loss with visualizations.
Explore an illustrative region-tiling policy: resize tiles, calculate visual tokens, and distinguish analytical savings from measured model quality.
Understand contrastive loss for representation learning: interactive demos of InfoNCE, triplet loss, and embedding space clustering with temperature tuning.
Understand dilated (atrous) convolutions: how dilation rates expand receptive fields without extra parameters and how to avoid gridding artifacts.
