Recurrent Neural Networks (RNNs)

Dense and convolutional networks process each input independently; they have no memory of what came before. But language, speech, time-series data, and music are all inherently sequential: the meaning of a word depends on the words before it. Recurrent Neural Networks are designed to handle exactly this kind of ordered, dependent data.

Understanding Sequence Data

Sequence data is any data where order matters. Common examples include:

• Text: the words 'not bad' and 'bad, not good' mean opposite things. Word order is everything.
• Time-series: a stock price at 2 PM depends on what happened at 1 PM. Order carries causal information.
• Audio: speech is a sequence of pressure waves where timing and order encode phonemes, words, and meaning.
• Video: frames are sequences where temporal relationships (motion) carry as much information as content.

What is an RNN?

A Recurrent Neural Network processes sequences by maintaining a hidden state — a fixed-size vector that summarises everything seen so far. At each time step t, the RNN takes the current input x_t and the previous hidden state h_(t-1), combines them, and produces a new hidden state h_t. This hidden state is passed forward to the next step, giving the network a form of memory.
import tensorflow as tf

# SimpleRNN — the most basic recurrent layer

rnn_layer = tf.keras.layers.SimpleRNN(

units=64,           # size of the hidden state vector

activation='tanh',  # tanh is standard for RNNs

return_sequences=False # return only the final hidden state

)

Challenges of Traditional RNNs

Simple RNNs suffer from two related problems that make training on long sequences very difficult:

• Vanishing gradient: during backpropagation through time (BPTT), gradients are multiplied at every time step. Over long sequences (50+ steps), these multiplications cause gradients to shrink exponentially, becoming so small that early time steps learn almost nothing. This was described formally by Hochreiter in 1991 and remained unsolved until LSTM was introduced in 1997.
• Exploding gradient: the opposite problem, gradients can also grow exponentially, causing numerical instability. This is typically addressed with gradient clipping (clipnorm or clipvalue in the optimizer).

Key Limitation

A standard RNN can theoretically look back at all past inputs, but in practice, it can rarely maintain useful information beyond 10–20 time steps. For anything longer, use LSTM or GRU instead.

GRU Networks

Gated Recurrent Unit (GRU), introduced by Cho et al. in 2014, is a streamlined version of LSTM with only two gates: a reset gate and an update gate. GRUs have fewer parameters than LSTMs, train faster, and often perform comparably. As described in the AgentFlow 2025 neural network guide: 'GRU is simpler than LSTM but often just as effective, like having a smarter filing system with fewer drawers.'

Implementing LSTM and GRU in TensorFlow

import tensorflow as tf

# LSTM for sequence classification

lstm_model = tf.keras.Sequential([

tf.keras.layers.Embedding(input_dim=10000, output_dim=64),

# return_sequences=True passes full sequence to next LSTM layer

tf.keras.layers.LSTM(128, return_sequences=True, dropout=0.2),

tf.keras.layers.LSTM(64,  return_sequences=False, dropout=0.2),

tf.keras.layers.Dense(1, activation='sigmoid'),   # binary output

])

# GRU (fewer params, similar performance)
gru_model = tf.keras.Sequential([

tf.keras.layers.Embedding(input_dim=10000, output_dim=64),

tf.keras.layers.GRU(128, return_sequences=True, dropout=0.2),

tf.keras.layers.GRU(64,  return_sequences=False, dropout=0.2),

tf.keras.layers.Dense(1, activation='sigmoid'),

])

# Both compile identically

lstm_model.compile(optimizer='adam', loss='binary_crossentropy',

metrics=['accuracy'])

Text Classification with RNNs

A classic application of LSTMs is sentiment analysis, determining whether a movie review is positive or negative. TensorFlow includes the IMDB dataset (50,000 movie reviews, balanced 50/50 positive/negative), which is the standard benchmark for this task:
import tensorflow as tf

# Load IMDB — vocabulary of 10,000 most common words

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.imdb.load_data(

num_words=10000

)

# Pad sequences to the same length (256 tokens)

x_train = tf.keras.preprocessing.sequence.pad_sequences(x_train, maxlen=256)

x_test  = tf.keras.preprocessing.sequence.pad_sequences(x_test,  maxlen=256)

model = tf.keras.Sequential([

tf.keras.layers.Embedding(10000, 64),

tf.keras.layers.LSTM(64, dropout=0.2, recurrent_dropout=0.2),

tf.keras.layers.Dense(1, activation='sigmoid'),

])

model.compile(optimizer='adam',

loss='binary_crossentropy',

metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5, batch_size=64,

validation_split=0.2)
# Expected test accuracy: ~85-88%

Time-Series Prediction Basics

LSTMs are widely used for forecasting and predicting future values based on past sequences. The key concept is the look-back window: you feed the model the last N time steps and ask it to predict the next value. As stated in the official TensorFlow time-series tutorial, the model architecture and window size should be tuned together for your specific forecasting horizon.

import numpy as np, tensorflow as tf

# Generate synthetic sine wave data

t = np.linspace(0, 100, 1000)

data = np.sin(t)

# Create sliding windows: use 50 steps to predict 1 step ahead

LOOK_BACK = 50

X, y = [], []

for i in range(len(data) - LOOK_BACK):

X.append(data[i:i+LOOK_BACK])

y.append(data[i+LOOK_BACK])

X, y = np.array(X), np.array(y)

X = X.reshape(X.shape[0], X.shape[1], 1)  # (samples, timesteps, features)

model = tf.keras.Sequential([

tf.keras.layers.LSTM(50, return_sequences=False, input_shape=(LOOK_BACK, 1)),

tf.keras.layers.Dense(1),

])

model.compile(optimizer='adam', loss='mse')

model.fit(X, y, epochs=20, batch_size=32, validation_split=0.1)