Skip to main content

Long Short-Term Memory: The Original 1997 LSTM

Explore the original no-forget-gate LSTM: gated writes and reads, the constant error carousel, path derivatives and the delayed-sequence experiments behind the 1997 paper.

TL;DR

  • The original 1997 LSTM uses an additive state with input and output gates, without the forget gate found in later variants.
  • The constant error carousel preserves a direct state derivative of one; readout saturation, gate dependencies and other paths still affect learning.
  • Writing, retaining and exposing information serve separate roles. The explorers use supplied gates to make those roles visible, rather than demonstrating learned behavior.
  • The paper tests delayed credit assignment on artificial sequence problems. Its results depend on each task’s topology, training procedure and success criterion.
  • The Python reference checks state and derivative arithmetic; reproducing the experiments also requires the original learning algorithm and network setup.

Why delayed events need a learning signal

An early event can determine the right answer many steps later. A recurrent network can represent that dependency in principle, but learning it requires the later error to influence the parameters responsible for the earlier event. Repeated multiplication by recurrent derivatives can make that signal vanish or explode before it becomes useful.

Sepp Hochreiter and Jürgen Schmidhuber’s 1997 Long Short-Term Memory paper changes the recurrent unit so it contains a protected additive state path. Learned gates decide when to write new information and when to expose stored information. Its central mechanism is the constant error carousel, or CEC: a linear unit with a fixed self-connection of weight one. Original paper, §§3–4

This review uses the author-hosted manuscript of the Neural Computation 9(8), 1735–1780 article. It is the 1997 paper, rather than the distinct 1996 technical report. The write-up date above is separate. A crucial historical boundary runs through the whole explanation: this original cell has input and output gates, but no forget gate. The familiar modern recurrence that multiplies the old state by a learned forget gate is a later development, discussed in Gers, Schmidhuber and Cummins, “Learning to Forget” (2000).

Why ordinary recurrence can lose a delayed signal

Consider one path through a recurrent computation. Its backward sensitivity contains a product of the weights and activation derivatives encountered along that path. Even a moderate factor of 0.8 becomes about 0.0115 after twenty steps. Repeating a factor above one creates the opposite problem. The original analysis accounts for many paths and their possible interactions; a scalar product is an illustration of the mechanism, not a substitute for the full network Jacobian. §3.1, Equation 2

Keeping every unit in a nearly linear regime does not by itself solve the practical problem. A unit that stores something must also receive and emit information. The same connections may be asked both to preserve an old value and to respond to a new one. The paper describes these as input and output weight conflicts: useful behavior at one time can interfere with useful behavior at another.

A bare unit-weight linear self-loop preserves its state, but irrelevant inputs can still disturb that state. Its output can also interfere with other computations while the value is waiting to be used. LSTM adds two distinct gates to handle those conflicts while leaving the direct internal path intact.

Store by addition, protect by gating

For one original cell, let s be its internal state, i its input gate, a its candidate preactivation and o its output gate. The central update is:

s_t=s_{t-1}+i_tg(a_t),\qquad y_t=o_th(s_t)

The input gate scales the new write, not the old state. When it closes, the previous state passes through unchanged. When it opens, a positive or negative candidate can be added. Negative writes can cancel stored values, but there is no separate learned multiplicative erase operation in this architecture. §4, original Figure 1

Write once, protect memory, read later

A positive input arrives first; an irrelevant negative input arrives at t=3. Close the input gate to preserve the first write, then open the output gate at t=6.

Input: 1
Input gate: 1

Cell state
0.9242

Output gate: 0
Visible output: 0.0000

At t=1, the input gate permits a write. The output is hidden while the state remains available.

Original 1997 no-forget cell with externally supplied gates. g(x)=2 tanh(x/2), h(s)=tanh(s/2). This is arithmetic, not a trained sequence classifier. All steps are manual.

The authored sequence writes a positive value at its first step and presents a distracting negative value later. With the input gate closed after the first write, the distraction is blocked. With every write allowed, the negative candidate cancels the earlier positive contribution. That difference demonstrates gating arithmetic; it does not demonstrate a network learning to choose the correct gates.

The reference supplies gate values directly so the causal roles remain inspectable. In the original network, sigmoid gate units compute their activations from weighted network inputs. Their weights are learned. A memory-cell block may contain multiple cells sharing input and output gates, so a single gate decision can control a group of stored values. This historical block structure should not be confused with the tensor batching used by a contemporary recurrent library.

For the default illustrative functions, the paper’s appendix uses a candidate squashing range of [-2,2] and a readout range of [-1,1]. Our code expresses those choices as g(a)=2 tanh(a/2) and h(s)=tanh(s/2), equivalent to scaled logistic functions. Some original experiments select different functions and topologies; the paper is not one immutable modern library cell repeated across every task.

Reading does not erase the state

The output gate decides whether stored information should affect the rest of the network now. Setting that gate to zero makes the visible output zero while leaving the state available for later use. This addresses a different issue from preventing an irrelevant input from modifying the state.

Writing and reading solve different conflicts

Start with stored state s=1. The input gate controls the new write; the output gate controls exposure of the updated state. Neither gate scales the old state on its self-loop.

-1
0
1
Previous state
1.0000
+ gated write → next state
1.0000
h(state) × output gate
0.4621

Readout derivative with respect to the next state: 0.3932. Setting the output gate to zero hides the state without erasing it.

The 1997 cell has no forget gate. A later forget-gated LSTM would add a learned multiplier on the old state. This explorer uses the original g/h scaling.

Try opening the write gate while closing the read gate. The internal state changes, but the downstream output remains hidden. Then close writing and open reading: the already stored value becomes visible without requiring a new candidate. This separates storage, modification and exposure into different operations.

The state can accumulate beyond the bounded range of g, because the update repeatedly adds bounded writes. The readout h can therefore saturate even while the internal path remains additive. Saturation makes the derivative from the visible output into the state small. A closed output gate can make it zero. Preserving memory internally is only one condition for learning a useful delayed computation; the network must also learn when to write and when to make the state observable.

A modern forget-gated cell makes another operation explicit: scaling the previous state before adding the new write. In schematic form it uses s_t=f_t s_(t−1)+i_t g(a_t). Setting f to one recovers the original direct retention path, but a learned f changes both forward retention and the product of derivatives along that path. Introducing that gate into code without labeling the change would explain a different historical architecture.

What the constant derivative does and does not prove

If gates and candidate writes are treated as fixed, the derivative of the next state with respect to the previous state is exactly one. Chaining that direct self-loop path across many steps still gives one. Unlike the repeated 0.8 example, the path does not progressively shrink just because the interval is long.

A protected path avoids repeated shrinkage

Compare a product of identical recurrent derivatives with the unit derivative along the cell's direct self-loop. This isolates one path while holding gate values and input writes fixed.

20
0.8

Path-derivative magnitude · log₁₀ scale

CEC direct self-loopOrdinary recurrent path

At t=20, the ordinary path is 0.01153; the direct CEC path is 1. The vertical domain expands to include both.

Direct CEC derivative: 1. Ordinary path: 1.153e-2.

A value above one can explode; a value below one decays. A preserved internal path does not guarantee that the output gate opens or that the readout avoids saturation.

Analytical path derivatives. Total network gradients also include gates, readouts and other recurrent paths. The original training algorithm truncates selected gradient paths; it is not unrestricted modern autodiff BPTT.

This is a statement about an available path, not a theorem that every parameter in an LSTM receives a constant gradient. Gates can depend on recurrent activations. Other network paths contribute additional terms. The readout has its own derivative, and task losses can introduce further scaling. The explorer deliberately holds those dependencies fixed when comparing the CEC with an ordinary repeated product.

The original learning procedure also matters. It maintains local sensitivities for cell-related weights and truncates selected gradient paths where the authors argue that doing so avoids harmful effects while retaining useful delayed credit assignment. Its stated locality and constant per-weight, per-step complexity refer to that algorithm. It should not be identified wholesale with unrestricted backpropagation through every edge of a modern autodiff unroll. §4 and Appendix A

These distinctions explain why a tiny cell simulation is valuable but insufficient as a reproduction. Correct state arithmetic can be established independently of reproducing the paper’s training algorithm, network wiring, initialization and success criteria.

What the original experiments actually tested

The paper evaluates artificial sequence-learning problems, including an embedded Reber grammar, sequences with long delays and distractors, tasks with real-valued information, and order-sensitive inputs. These are targeted tests of delayed credit assignment. They are not language-model perplexity experiments or speech-recognition benchmarks. The overview reports tasks with minimal delays reaching 1,000 steps, but the same delay and architecture do not apply to every experiment. §5 overview

The embedded Reber grammar is especially instructive. An early branching symbol determines the appropriate symbol at the end of an embedded grammatical sequence. The output must predict valid continuations while remembering which outer branch was taken. The authors explicitly note that this task’s delays, sometimes as short as nine steps, do not make it their hardest long-lag problem. Its role includes showing the value of controlling cell output. §5.1

Their setup uses local seven-dimensional input and output representations, randomly generated training and separate test sequences, and a criterion requiring correct predictions on both sets. Table 1 reports, for one three-block configuration with two cells per block and learning rate 0.5, 100% successful trials and a mean of 8,440 sequence presentations until success. The table’s comparison averages for other algorithms are drawn from the cited studies and successful runs, with protocol differences explained in its caption. These counts are training presentations under the paper’s criterion, not wall-clock speedups. §5.1 and Table 1

Experiment 2 then separates noise-free local symbol sequences from more difficult sequences where distracting symbols remove predictable local regularities. For the noise-free case with delay parameter p=100, the paper reports that its one-cell LSTM learned about twice as fast as the neural sequence chunker, while BPTT and RTRL failed under the tested conditions. The authors also explain that a hierarchical chunker can solve that particular regular problem, motivating harder follow-up tasks. That is a much narrower, more useful conclusion than saying recurrent baselines can never learn delayed relationships. §5.2

The strength of this evidence is mechanistic: carefully constructed tasks make a failure mode observable and test an architecture designed to address it. Its limitation is equally specific: success on generated sequences does not establish comparative performance on every real sequence distribution, and the reported topologies, activations and optimization conditions are part of each result.

Original paper, equations and executable Python

The three rows retain the original state equation, cell diagram and recurrent-gradient analysis next to a readable reference. Its inputs and gates are fixed teaching values. It implements no optimizer and claims no learned predictions or recreated benchmark result.

1. Add a gated write to a unit self-loop

Original paper

§4: internal state update

Scalar reference; production cells can form blocks sharing gates.

Math and tensor shapes

\begin{aligned}s_t&=s_{t-1}+i_tg(a_t)\\ g(a)&=2\tanh(a/2)\end{aligned}

Scalar reference; production cells can form blocks sharing gates.

The original 1997 cell does not contain a forget gate. Here the input gate is supplied numerically, while the original learns sigmoid gates from network activations. The candidate scaling follows the paper’s appendix, rather than silently substituting modern tanh.

Python implementation

lstm.py · update
python
def g(x):
    return 2*math.tanh(x/2)  # equivalent to 4*sigmoid(x)-2


def cell_step(state, value, input_gate, output_gate):
    """The unit-weight self-loop adds gated writes; it never multiplies by f."""
    state = state + input_gate*g(value)
    return {'state': state, 'output': output_gate*h(state)}

2. Expose stored state through a separate output gate

Original paper

§4 and Figure 1: gated memory cell

One state and output per time step; original gates lie in [0,1].

Math and tensor shapes

\begin{aligned}y_t&=o_th(s_t)\\ h(s)&=\tanh(s/2)\end{aligned}

One state and output per time step; original gates lie in [0,1].

Closing the output gate hides a value without deleting the internal state. Closing the input gate protects memory against irrelevant future writes. The reference does not learn gate weights or reproduce the original task-specific network topology.

Python implementation

lstm.py · readout
python
def h(state):
    return math.tanh(state/2)  # equivalent to 2*sigmoid(state)-1


def trajectory(inputs, input_gates, output_gates, initial=0.0):
    state, results = initial, []
    for value, write, read in zip(inputs,input_gates,output_gates):
        result=cell_step(state,value,write,read)
        state=result['state']; results.append(result)
    return results

3. Separate the preserved state path from the whole gradient

Original paper

§3.1: products of recurrent derivatives

Isolated direct state path; fixed gates and input writes.

Math and tensor shapes

\begin{aligned}\left.\frac{\partial s_T}{\partial s_0}\right|_{\mathrm{fixed\ writes}}&=1\\ \prod_{t=1}^{T}q&=q^T\end{aligned}

Isolated direct state path; fixed gates and input writes.

A unit derivative belongs to the self-loop path. The total network gradient can include recurrent gate dependencies and a saturated readout. The original algorithm truncates selected paths and propagates local weight sensitivities; it is not identical to unrestricted modern BPTT through an LSTM graph.

Python implementation

lstm.py · gradients
python
def gradient_path(steps, factor=0.8):
    """Compare isolated direct state paths, holding gates/writes fixed."""
    return {'cec': 1.0, 'recurrent': factor**steps}


def read_derivative(state, output_gate):
    """CEC derivative is one; the readout derivative can still saturate."""
    return output_gate*0.5*(1-h(state)**2)

Original source excerpts: Sepp Hochreiter, Jürgen Schmidhuber. Neural Computation 9(8), 1735–1780 (1997), author manuscript. Cropped and rasterized from the hash-pinned author PDF; colors unchanged. Source page, rectangle and image hashes are recorded in the crop manifest. No open license is asserted where none was verified.

Run python examples/lstm.py, or use --json for deterministic trajectory and derivative values. The self-checks verify protected retention and compare the analytic readout derivative with a central finite difference. Download the reference, requirements and provenance notes.

The contribution becomes clearer when its pieces remain separate: an additive internal path makes preservation possible; an input gate protects that path from unwanted writes; an output gate controls when memory influences the rest of the network; and a learning procedure assigns credit across the delay. Later LSTM variants changed important details, but this original arrangement explains why controllable memory became a practical alternative to asking an ordinary recurrent unit to preserve and transform everything through the same dynamics.

If you found this paper review helpful, consider sharing it with others.

Mastodon