Natural Language Processing with TensorFlow

Text is unstructured, variable in length, and made of discrete symbols, none of which a neural network can consume directly. This chapter walks through how TensorFlow turns raw text into numbers, and how those numbers flow through architectures ranging from simple sequence models up to a Transformer built entirely from scratch.

Text Tokenization and Embeddings

Tokenization is the process of splitting raw text into smaller units, words, subwords, or characters, and mapping each unit to an integer ID. TextVectorization is TensorFlow's built-in layer for this, and it can be saved directly inside your model so that preprocessing travels with it into production.

import tensorflow as tf

texts = ['TensorFlow makes deep learning accessible',

'Natural language processing is powerful']

vectorizer = tf.keras.layers.TextVectorization(

max_tokens=10000,

output_mode='int',

output_sequence_length=20,

)

vectorizer.adapt(texts)   # builds the vocabulary

sequences = vectorizer(texts)

print(sequences.shape)   # (2, 20)

Once text is integer-encoded, an Embedding layer maps each token ID to a dense vector that captures semantic meaning; words used in similar contexts end up with similar vectors. These embeddings can be learned from scratch alongside your model, or loaded pre-trained from sources like GloVe or Word2Vec.

embedding_layer = tf.keras.layers.Embedding(

input_dim=10000, # vocabulary size

output_dim=128,  # embedding dimension

input_length=20,

)

# Input: (batch, 20) integer IDs -> Output: (batch, 20, 128) dense vectors

Sequence-to-Sequence Models

Sequence-to-sequence (seq2seq) models map an input sequence to an output sequence of a potentially different length, the backbone of machine translation, summarisation, and chatbots. The classic architecture pairs an encoder, which compresses the input into a context vector, with a decoder, which generates the output one token at a time, conditioned on that context.

import tensorflow as tf

latent_dim = 256

# Encoder

encoder_inputs = tf.keras.Input(shape=(None,))

enc_emb = tf.keras.layers.Embedding(10000, latent_dim)(encoder_inputs)

_, state_h, state_c = tf.keras.layers.LSTM(latent_dim, return_state=True)(enc_emb)

encoder_states = [state_h, state_c]

# Decoder

decoder_inputs = tf.keras.Input(shape=(None,))

dec_emb = tf.keras.layers.Embedding(10000, latent_dim)(decoder_inputs)

decoder_lstm = tf.keras.layers.LSTM(latent_dim, return_sequences=True, return_state=True)

decoder_outputs, _, _ = decoder_lstm(dec_emb, initial_state=encoder_states)

decoder_outputs = tf.keras.layers.Dense(10000, activation='softmax')(decoder_outputs)

model = tf.keras.Model([encoder_inputs, decoder_inputs], decoder_outputs)

The encoder's final hidden state acts as a compressed summary of the entire input, handed to the decoder as its starting point. This works reasonably well for short sequences but struggles as inputs grow longer, which is exactly the limitation attention mechanisms were designed to fix.

Building a Transformer from Scratch

The Transformer replaced recurrence entirely with self-attention, letting every token look directly at every other token in the sequence in parallel, regardless of distance. This solves the long-range dependency problem of RNNs and is dramatically more parallelisable on GPUs and TPUs.

The core building block is scaled dot-product attention, computed from three projections of the input: Queries (Q), Keys (K), and Values (V).

import tensorflow as tf

def scaled_dot_product_attention(q, k, v, mask=None):

matmul_qk = tf.matmul(q, k, transpose_b=True)

dk = tf.cast(tf.shape(k)[-1], tf.float32)

scaled_logits = matmul_qk / tf.math.sqrt(dk)

if mask is not None:

scaled_logits += (mask * -1e9)

weights = tf.nn.softmax(scaled_logits, axis=-1)

output = tf.matmul(weights, v)

return output, weights

Multi-head attention runs several of these attention computations in parallel, each with its own learned projections, so different heads can specialise in capturing different kinds of relationships (syntax, position, meaning).

class MultiHeadAttention(tf.keras.layers.Layer):

def __init__(self, d_model, num_heads):

super().__init__()

self.num_heads = num_heads

self.depth = d_model // num_heads

self.wq = tf.keras.layers.Dense(d_model)

self.wk = tf.keras.layers.Dense(d_model)

self.wv = tf.keras.layers.Dense(d_model)

self.dense = tf.keras.layers.Dense(d_model)

def split_heads(self, x, batch_size):

x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))

return tf.transpose(x, perm=[0, 2, 1, 3])

def call(self, q, k, v, mask=None):

batch_size = tf.shape(q)[0]

q = self.split_heads(self.wq(q), batch_size)

k = self.split_heads(self.wk(k), batch_size)

v = self.split_heads(self.wv(v), batch_size)

attn_output, _ = scaled_dot_product_attention(q, k, v, mask)

attn_output = tf.transpose(attn_output, perm=[0, 2, 1, 3])

concat = tf.reshape(attn_output, (batch_size, -1, self.num_heads * self.depth))

return self.dense(concat)

A full Transformer encoder layer wraps multi-head attention and a feed-forward network with residual connections and layer normalisation, then stacks several of these layers to build depth.

Using TensorFlow Hub for NLP

Training a Transformer from scratch is valuable for learning, but in practice, most teams fine-tune a pre-trained model instead. TensorFlow Hub hosts ready-to-use models, including BERT and its many variants, that have already learned rich language representations from billions of words.

import tensorflow_hub as hub

import tensorflow as tf

bert_preprocess = hub.KerasLayer(

'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')

bert_encoder = hubKerasLayer(

'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4')

text_input = tf.keras.layers.Input(shape=(), dtype=tf.string)

encoder_inputs = bert_preprocess(text_input)

outputs = bert_encoder(encoder_inputs)

pooled_output = outputs['pooled_output']   # (batch, 768) sentence embedding

classifier = tf.keras.layers.Dense(1, activation='sigmoid')(pooled_output)

model = tf.keras.Model(text_input, classifier)

This pattern, attaching a small classification head on top of a frozen or lightly fine-tuned pre-trained encoder, is called transfer learning, and it routinely outperforms training from scratch, especially when labelled data is limited.

NLP Applications and Use Cases

Did You Know?

Self-attention's complexity grows quadratically with sequence length, since every token attends to every other token. This is why long-document models use techniques like sparse attention or sliding windows to remain efficient on inputs of thousands of tokens.