LSTM Networks

Long short-term memory networks, commonly called LSTMs, are a specialized type of RNN designed to address the vanishing gradient problem and remember information over much longer sequences. While a basic RNN has a single, simple hidden state that gets overwritten at every step, an LSTM introduces a more sophisticated internal structure that can selectively decide what information to keep, what to forget, and what to pass forward.

This added complexity allows LSTMs to maintain relevant context over far longer sequences than a basic RNN ever could. For tasks like language translation, where understanding a word at the end of a long sentence might depend on something mentioned much earlier, this ability to retain long-term context made a meaningful, practical difference, and LSTMs became the standard choice for sequence-based tasks for many years before newer architectures emerged.

How LSTMs Fix the Vanishing Gradient Problem

Step 1: Introduce a Cell State

LSTMs solve the vanishing gradient problem by introducing a separate memory pathway called the cell state, which carries information through the entire sequence.

Step 2: Preserve Important Information

Instead of completely overwriting the memory at every step like a basic RNN, an LSTM carefully controls how much information is:

• Kept
• Updated
• Discarded

Step 3: Maintain Stable Gradient Flow

This controlled memory allows gradients to flow backward through many time steps without shrinking toward zero.

Step 4: Carry Information Across the Sequence

The cell state acts like a conveyor belt, carrying important information through the sequence while making only small, learned updates at each step.

Step 5: Handle Long Sequences

Because of this design, LSTMs can learn long-term dependencies and process long sequences much more effectively than basic RNNs.

Gates in an LSTM Cell

1. Forget Gate

The forget gate decides what portion of the existing memory should be discarded, allowing the network to remove information that is no longer relevant.

2. Input Gate

The input gate decides how much of the new incoming information should be added to the cell state.

3. Output Gate

The output gate decides what part of the current memory should be used to produce the output at the current step.

4. Learn Gate Behavior

Each gate is a small neural network component that learns, through training, how to behave for the specific task.

5. Maintain Selective Memory

This gating mechanism gives LSTMs a selective memory, allowing them to retain important information for long periods while updating or discarding information that is no longer needed.

LSTM vs GRU