The recurrence, written out
At each timestep t the cell takes the current input xt and the previous hidden state ht−1, and produces a new hidden state:
ht = tanh(Wxxt + Whht−1 + b)
Three things are worth noticing. The hidden state is the memory — a fixed-size vector holding everything the cell has decided is worth keeping from the sequence so far. The weights Wx, Wh and b are the same at every timestep; the cell is one small network applied repeatedly, not a chain of different networks. And because the weights do not depend on position, the same cell handles a sequence of 5 tokens or 500.
That weight sharing is what makes the parameter count independent of sequence length — and it is exactly the same trick a convolution uses across space.
The unit that carries state
A recurrent cell is the piece of a recurrent network that runs once per time step. It takes two things — the current input and its own state from the previous step — and produces a new state.
hᵗ = cell(xᵗ, hᵗ₋₁)
For the simplest cell:
hᵗ = tanh(Wₕₕ hᵗ₋₁ + Wₓₕ xᵗ + b)
The state hₜ is a fixed-size vector that summarises everything seen so far. Its size is a hyperparameter — 128, 256, 512 — and it does not grow with the sequence, which is both the cell's defining strength and its defining limit.
The same weights are used at every step. That is parameter sharing along time, and it is what lets one cell handle a sequence of any length with a fixed parameter count. A network with distinct weights per position would need a separate set for every possible length.
Three cells, in order of sophistication
| Cell | State | Gates | Parameters per layer |
|---|
| Simple RNN | h | None | 1 matrix |
| GRU | h | 2 (update, reset) | 3 matrices |
| LSTM | h and c | 3 (forget, input, output) | 4 matrices |
The simple cell overwrites its state at every step. Whatever it knew is repeatedly transformed, and after fifty steps almost nothing survives. Gradients decay along the same path, so it cannot learn long-range dependencies. It is a teaching device rather than a practical choice.
The GRU adds an update gate that interpolates between keeping the old state and replacing it, plus a reset gate controlling how much history the candidate sees. One state, two gates.
The LSTM adds a separate cell state updated additively, with three gates deciding what to drop, what to store and what to expose. Two states, three gates.
The pattern across the three is the same insight applied more thoroughly: give the state an additive path so information and gradients can survive many steps.
Why gating changes everything
For the simple cell, the gradient across one step involves the derivative of tanh times the recurrent weight matrix — a quantity nothing keeps near 1, and usually below it. Over 50 steps the product collapses.
For an LSTM, ∂cₜ/∂cₜ₋₁ is exactly the forget gate. When the network has learned to preserve something, that value is near 1 and the gradient passes back essentially undamped.
That single derivative is the difference between failing at 20 steps and succeeding at 200. And it is structurally the same trick as a residual connection in a deep feed-forward network: an additive path with a derivative of 1.
The practical guidance follows directly: use an LSTM or GRU, never nn.RNN. The extra cost is small and the difference in what can be learned is large.
One interface, three implementations
A recurrent cell is any function that takes a state and an input and returns a new state. Writing the RNN, LSTM and GRU cells against that one signature shows what they share and exactly where they differ.
Things to try
- Feed one token at a time. Press Feed Next Token and watch the hidden state change. Each press mixes new input into the existing memory rather than replacing it.
- Watch old information fade. Keep pressing and follow the contribution of the first token. It does not vanish at once; it is progressively diluted as later tokens are folded in. That gradual decay is exactly why plain recurrent cells struggle with long-range dependencies.
- Reset and compare. Press Reset Memory and feed the same tokens in a different order. The final state differs — unlike a bag-of-words model, order genuinely changes the representation.
- Run it continuously. Press Auto Run and note that no new parameters appear as the sequence lengthens. One cell, applied repeatedly.
Why the simple cell is not enough
The hidden state is overwritten at every step. Information from token 1 survives to token 50 only by passing through 49 successive multiplications by Wh and 49 tanh nonlinearities.
That is the vanishing gradient problem in its original setting. The gradient reaching timestep 1 is a product of 49 terms, and since tanh’s derivative is at most 1 and typically well below it, the product decays geometrically. In practice a simple recurrent cell reliably remembers about 10 timesteps and loses anything much further back.
The fix is to give the cell an explicit, additive memory channel with learned gates controlling what enters and leaves it — which is precisely what LSTM and GRU do.
Common mistakes
- Expecting long-range memory. A vanilla recurrent cell does not have it. If dependencies span more than a handful of tokens, use an LSTM, a GRU, or attention.
- Exploding gradients. If Wh has eigenvalues above 1 the repeated multiplication amplifies instead of decaying, and the loss becomes NaN. Gradient clipping is standard practice for recurrent models for exactly this reason.
- Confusing the hidden state with the output. They coincide in the simplest cell, but in an LSTM the cell state and hidden state are different objects, and mixing them up is a common implementation bug.
- Forgetting to reset state between sequences. Carrying the hidden state from one unrelated example into the next leaks information and quietly corrupts training.
In one line
A recurrent cell applies one small network at every timestep, combining the current token with a hidden state that summarises everything before it, using the same weights throughout. That gives a fixed-size model the ability to read arbitrary-length sequences and to be sensitive to order. Its weakness is that memory is overwritten multiplicatively at each step, so information decays after roughly ten timesteps — the limitation that gated cells were invented to remove.
The cell versus the layer
Frameworks distinguish two levels, and knowing which you want saves confusion.
The cell processes one time step. nn.LSTMCell takes (x, (h, c)) and returns the new states. You write the loop.
The layer processes a whole sequence. nn.LSTM takes a (batch, seq, features) tensor and runs the loop internally, in optimised C or cuDNN.
# the layer - what you almost always want
lstm = nn.LSTM(300, 256, batch_first=True)
out, (h, c) = lstm(x) # x: (batch, seq, 300)
# the cell - when you need per-step control
cell = nn.LSTMCell(300, 256)
h, c = torch.zeros(b, 256), torch.zeros(b, 256)
for t in range(seq_len):
h, c = cell(x[:, t], (h, c))
The layer is dramatically faster — often ten times or more — because it fuses the four gate matrices into one multiplication and uses hardware-optimised kernels. Write the loop only when you need something the layer cannot express: inspecting gates, applying custom logic between steps, scheduled sampling during training.
Two states, two purposes
For an LSTM specifically, the distinction between the states is worth restating because it is the design's whole point.
The cell state is private long-term memory. It is unbounded, updated additively, and passed only to the next time step.
The hidden state is the working output. It is bounded by tanh, filtered by the output gate, and passed to three places: the next layer, the prediction head, and the next step's gate computations.
That separation lets a cell hold information quietly without reporting it. A GRU, having only a hidden state, cannot make the distinction — whatever it remembers is what it outputs.
Where recurrent cells still fit
Transformers displaced them for large-scale language work. The cases where a recurrent cell remains the better engineering choice are specific and real:
- Streaming with bounded memory. One fixed-size state regardless of how much has been processed, where a transformer's KV cache grows with every token.
- On-device inference. Small, fast, no attention kernels required.
- Small datasets. The inductive bias helps where a transformer trained from scratch overfits.
- Classical time series. Numeric sequences of moderate length, where windowing plus a GRU is a strong baseline.
- Inside larger systems, as state trackers or encoders.
State-space models such as Mamba are worth noting: they revisit recurrence with linear scaling in sequence length and are competitive with transformers on long sequences, which suggests the idea is not finished.
Questions people ask
What is the difference between a cell and a layer? The cell handles one step; the layer runs the loop over a whole sequence and is much faster.
Should I ever use nn.RNN? Only to demonstrate the problem it has. Use LSTM or GRU.
How big should the hidden state be? 128–512 typically. Larger means more memory capacity and more parameters.
Can I stack cells? Yes — each layer's outputs are the next layer's inputs. Two layers is the practical limit.
Does the cell see the whole sequence? Not at once. It sees one element at a time, carrying a summary of the rest.
Why can it not be parallelised? Step t needs hₜ₋₁, so the steps are strictly ordered. This is the structural reason transformers won at scale.
Recap in one screen
- A recurrent cell maps the current input and the previous state to a new state, using the same weights every step.
- The state is a fixed-size summary that does not grow with the sequence.
- A simple cell overwrites its state and cannot learn long range; gating adds an additive path that preserves both information and gradients.
- Use the layer (
nn.LSTM) rather than the cell unless you need per-step control. - Still the right tool for streaming, on-device and small-data sequence work.