TL;DR
- LeNet-5 builds in locality and shared filters so a detector can respond at many image positions, while subsampling reduces sensitivity to small shifts.
- The original network uses learned subsampling, scaled tanh, partial C3 connectivity and fixed RBF prototypes; these details distinguish it from common modern teaching versions.
- The manuscript reports about 0.95% MNIST test error without distortion training and 0.8% with it, under the paper’s training conditions.
- The explorers and runnable reference isolate mechanisms using synthetic inputs and calculated outputs. The broader paper also trains document-recognition systems across segmentation and recognition.
Why handwriting needs spatial structure
A handwritten digit shifts a few pixels, changes slant, or acquires a slightly different stroke. A classifier should recognize the same digit without relearning every displaced version as an unrelated pattern. LeNet’s design makes that requirement part of the network: local receptive fields detect small patterns, shared weights reuse detectors, and subsampling reduces sensitivity to exact position.
The 1998 paper by Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner is much broader than a digit-classification diagram. It develops gradient-based document recognition, compares learning methods, and connects trainable modules into systems that handle segmentation and recognition together. LeNet-5 is its concrete convolutional example, rather than the entire contribution. This review follows the author-hosted 1998 manuscript; the publication record identifies the final article in Proceedings of the IEEE, 86(11), 2278–2324. The date above belongs to this write-up.
Every interactive figure below uses authored synthetic inputs and analytically calculated outputs. The Python reference isolates mechanisms; it does not train a network or reproduce MNIST accuracy. Paper results are identified separately with their training conditions.
Locality changes what must be learned
A fully connected first layer treats every input position as a different source. For an image, this discards useful structure: nearby pixels are related, and a small stroke can matter in many locations. LeNet restricts each unit to a small neighborhood and ties the weights of units within one feature map. One detector can therefore respond at many positions. Original paper, §II-A
For C1 in LeNet-5, the input is a 32×32 grayscale image, the kernels are 5×5, and six output feature maps have size 28×28. Each map has 25 shared kernel weights plus one bias, giving 156 parameters. The 4,704 output activations do not require 4,704 independent filters. Weight sharing turns a spatial assumption into a much smaller parameterization.
One kernel, sixteen locations
Move a shared 3×3 edge filter over an authored 6×6 stripe. LeNet-5 uses 5×5 filters; this smaller stencil makes weight sharing inspectable.
Input and selected receptive field
Shared kernel
Output map
Output (0,0) = 3. The same nine kernel coefficients serve every position; moving the window creates no new parameters.
The explorer uses a smaller 3×3 stencil so each multiplication is visible. Its positive and negative outputs are a fixed edge response, not learned digit features. Sliding the same stencil demonstrates the relevant constraint: updating one coefficient changes its behavior at every location. During backpropagation, contributions from all those uses accumulate into the same coefficient’s gradient.
This does not make recognition exactly invariant to every translation. A convolution shifts a feature response with the input; finite image boundaries and later operations matter. The paper combines this equivariant reuse with spatial aggregation and training examples to reduce sensitivity to small distortions. General robustness comes from the complete design, not from weight sharing alone.
Original subsampling is not max-pooling
S2 follows C1 with six 14×14 maps. Each output combines a nonoverlapping 2×2 neighborhood within the corresponding map. The paper describes adding the four values, multiplying by a trainable coefficient, adding a trainable bias, and applying a squashing function. A map therefore adds two trainable parameters, for twelve across S2. S4 repeats the pattern over sixteen maps and adds thirty-two parameters. §II-B
Subsampling is a learned operation
Average four inputs, apply a shared coefficient and bias, then the original scaled tanh. Change the coefficient to see that this is not max-pooling.
mean 0.5 → weighted input 0.50 → output 0.5517
Our reference averages the four inputs before applying the coefficient. This is an equivalent parameterization: multiplying the average by four times the paper’s coefficient produces the same preactivation as its sum. The distinction is explicit because it affects how one interprets a particular supplied coefficient, even though it does not change the family of learnable functions.
The hidden-unit squashing function is a scaled hyperbolic tangent, with amplitude 1.7159 and the commonly used slope parameter 2/3. It is neither ReLU nor a modern sigmoid probability. As the learned coefficient changes, a subsampling unit can operate in a nearly linear region or push into saturation. The original discussion relates these regimes to averaging and more selective combinations of nearby evidence.
A popular teaching network with ReLU, parameter-free max-pooling and a learned softmax classifier can be a useful contemporary MNIST model. Its performance is evidence about that model, however. Those substitutions should not be presented as a literal implementation of the 1998 architecture.
C3 deliberately breaks full connectivity
The next convolution does not connect every output map to all six S2 maps. C3 has sixteen 10×10 maps, but its input subsets follow the hand-designed pattern in original Table I. The first six outputs each combine three input maps; the next nine each combine four; the final output combines all six. That is 60 input/output-map links, rather than 96. Table I and §II-B
C3 connects selected feature maps
Pick an output map to reveal which of the six S2 maps feed its 5×5 filters. A full 6-to-16 convolution would have a different parameter count.
C3 0 reads 3 input maps: 0, 1, 2. Its kernel bank has 76 parameters. All sixteen maps together use 1,516, versus 2,416 for full connectivity.
Each link owns a 5×5 filter, and each output map owns a bias. The count is consequently 60×25 + 16 = 1,516 trainable parameters. A fully connected six-input, sixteen-output convolution would have 96×25 + 16 = 2,416. Counting only the number of output maps would miss this architectural difference.
The paper gives two motivations: controlling the connection count and breaking symmetry so different maps can learn complementary combinations. This is structured, prescribed connectivity. It does not say that training discovers a sparse mask, and it is not the same as a depthwise convolution in which each output initially sees just one input channel.
After S4, C5 uses 120 5×5 filters connected to all sixteen 5×5 input maps. On a 32×32 input, its outputs are 120 scalars. It is still described as a convolutional layer because its connectivity would become spatial if the preceding maps were larger. F6 then maps those 120 values to 84 hidden units. Spatial shape explains why C5 can look fully connected in this particular diagram while retaining a convolutional interpretation.
The last stage measures distance
LeNet-5’s output consists of ten Euclidean radial-basis-function units, each reading the 84-dimensional F6 vector. Each output is a squared distance to a prototype, so smaller means closer. The described prototypes are fixed ±1 patterns derived from 7×12 character bitmaps. The paper discusses their role as distributed output codes and as penalties for mismatch to a class representation. §II-B, Equation 7
The output compares feature vectors to prototypes
Move a two-dimensional feature vector and inspect squared Euclidean distances. The original output used ten 84-dimensional RBF units.
Prototype A (1, -1): distance² = 0.500 · closest
Prototype B (-1, 1): distance² = 4.500
Prototype C (1, 1): distance² = 2.500
Closest prototype: A. Smaller distance is better; these outputs are not a softmax distribution.
The two-coordinate explorer reduces this geometry to something inspectable. Its three prototypes are authored vectors, not the original digit codes. It shows why raw RBF outputs need not sum to one and why choosing the largest output would reverse the intended decision. Training still adjusts the upstream features so the correct prototype becomes compatible with the input.
The six trainable stages contain 60,000 parameters: C1 156, S2 12, C3 1,516, S4 32, C5 48,120 and F6 10,164. The fixed RBF centers are excluded. Adding a learned 84-to-10 affine classifier would add 850 parameters and alter the model. Distinguishing weights from connections also matters: sharing allows many spatial connections to reuse the same learned value.
Original evidence: MNIST and distortion training
The paper constructs MNIST from NIST’s handwritten-digit collections, with 60,000 training and 10,000 test examples. Its preprocessing preserves aspect ratio while fitting the digit into a 20×20 region, then centers it in a 28×28 field using its center of mass. LeNet’s 32×32 input leaves additional room around that normalized digit. The regular and deskewed databases discussed in the paper are different input conditions. §III-A
For the regular MNIST experiment, the author manuscript reports a 0.95% test error after the test curve levels off, while the training error reaches 0.35% after nineteen passes. These values belong to the stated training procedure, including its learning-rate schedule and curvature-based scaling, rather than a generic guarantee for any LeNet-shaped network. §III-B and Figure 5
The authors then generate 540,000 distorted examples in addition to the original 60,000, using random combinations of affine transformations such as translation, scaling, squeezing and shear. With those augmented inputs, the reported test error becomes 0.8%, compared with roughly 0.95% without distortion. The training schedule changes with the larger dataset; the paper notes the comparable total number of presented patterns. This is a result for distortion-trained LeNet-5, not a fresh measurement made by the explorer and not an ensemble result. §III-B, Figures 6–8
That comparison is valuable precisely because it separates architectural bias from coverage of plausible handwriting variations. Locality and sharing help, but additional realistic transformations still improve generalization. Conversely, MNIST’s normalized, centered, grayscale digits are much simpler than arbitrary natural images or full documents. A low error there does not establish robustness to unconstrained backgrounds, new alphabets or a different acquisition process.
Paper, equations and a runnable reference
The rows below align original PDF excerpts with the relevant arithmetic and canonical Python regions. They intentionally implement suboperations, not a trainable reproduction. The partial C3 map and parameter count are literal; the convolution input and reduced RBF prototypes are teaching examples. No checkpoint, optimizer or dataset download is hidden behind these calculations.
1. Apply the same local filter across the image
Original paper
Original C1: 32×32×1 → 28×28×6 with 5×5 filters.
Math and tensor shapes
Original C1: 32×32×1 → 28×28×6 with 5×5 filters.
The executable toy uses a 3×3 stencil on a 6×6 image, before squashing. A filter coefficient is shared across positions; C1 therefore has 6(25+1)=156 trainable parameters rather than a separate kernel at each location.
Python implementation
def convolve(image, kernel):
"""Valid cross-correlation, the convention commonly called convolution."""
k = len(kernel)
return [[sum(kernel[dy][dx] * image[y+dy][x+dx]
for dy in range(k) for dx in range(k))
for x in range(len(image[0])-k+1)]
for y in range(len(image)-k+1)]2. Learn the scale of local averaging
Original paper
C1 28×28×6 → S2 14×14×6; two trainable scalars per map.
Math and tensor shapes
C1 28×28×6 → S2 14×14×6; two trainable scalars per map.
The original sums four inputs before its learned coefficient. Our equivalent average formulation rescales that coefficient by four. This is learned subsampling with a squashing nonlinearity, not modern parameter-free max-pooling.
Python implementation
def squash(x):
return 1.7159 * math.tanh(2*x/3)
def subsample(values, weight=1.0, bias=0.0):
"""Average, learned coefficient and bias, then scaled tanh. Not max-pool."""
return squash(weight * sum(values)/len(values) + bias)3. Count the original partial C3 connections
Original paper
S2 14×14×6 → C3 10×10×16; 60 map-to-map links.
Math and tensor shapes
S2 14×14×6 → C3 10×10×16; 60 map-to-map links.
Each connected input/output-map pair receives its own 5×5 kernel. The table is a structured connectivity design, not a claim that trained filters are sparse. Fully connecting all six inputs to all sixteen outputs would instead require 2,416 parameters.
Python implementation
C3 = [[0,1,2], [1,2,3], [2,3,4], [3,4,5], [0,4,5], [0,1,5],
[0,1,2,3], [1,2,3,4], [2,3,4,5], [0,3,4,5], [0,1,4,5],
[0,1,2,5], [0,1,3,4], [1,2,4,5], [0,2,3,5], [0,1,2,3,4,5]]
def connection_count():
"""One 5x5 kernel for each connected input map, plus one bias per output."""
return sum(25*len(inputs)+1 for inputs in C3)4. Read squared distance to class prototypes
Original paper
F6: 84 features → 10 Euclidean RBF distances.
Math and tensor shapes
F6: 84 features → 10 Euclidean RBF distances.
The original prototypes are manually chosen ±1 codes derived from 7×12 character bitmaps. Their centers are fixed in the described network. Lower distance indicates a better match. A learned 84→10 affine layer and softmax changes the historical architecture.
Python implementation
def rbf(features, prototypes):
"""Squared distances, with smaller output indicating a closer prototype."""
return [sum((x-p)**2 for x,p in zip(features, prototype))
for prototype in prototypes]Original source excerpts: Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner. 1998 author manuscript submitted to Proceedings of the IEEE. Cropped and rasterized from the hash-pinned author PDF; colors unchanged. Source page, rectangle and image hashes are recorded in the crop manifest. No open license is asserted where none was verified.
Run python examples/lenet.py for self-checks or add --json for deterministic outputs. Download the reference, requirements and provenance notes. The standard-library script checks the 1,516-parameter C3 count, learned subsampling arithmetic and distance geometry. The browser uses the same operations, with parity checked against Python.
Beyond isolated digit recognition
A document system also has to decide which pixels form a character. Segment first and recognize second is brittle when the correct boundary depends on recognizing the content. The later sections of the paper develop graph transformer networks, in which alternatives for segmentation and recognition can be represented and scored together, with gradients reaching trainable components. That systems perspective is easy to lose when the paper is reduced to its convolutional architecture figure. Original paper, §§IV–VI
The enduring lesson is to make useful structure learnable across the right boundaries. LeNet shares detectors across space, lets several stages be optimized together, and evaluates the resulting design inside an actual recognition problem. Its historical details also show that foundational ideas can survive changes in their surrounding implementation. Modern CNNs retain local shared filters while often replacing the original activation, partial connectivity, subsampling and RBF head. Understanding which parts changed makes that lineage more informative than treating every small CNN as the same network.
