Text Preprocessing and Representation

Computers generally require numerical representations of text for machine learning. Text preprocessing prepares language data for analysis.

Common Preprocessing Techniques

Tokenization: Splits text into smaller units called tokens. Tokens may be words, subwords, or characters.

Lowercasing: Converts text to lowercase when capitalization is not important for the task.

Stop-word removal: Removes selected common words, such as the or is, when appropriate. This is not always useful, particularly when negation affects meaning.

Stemming: Reduces words to approximate root forms. For example, playing may become play.

Lemmatization: Converts words into dictionary forms using linguistic information. For example, better may be mapped to good in the appropriate context.

Representing Text as Numbers

Bag of Words: Represents text using word occurrence counts, without preserving word order.

TF-IDF: Gives greater importance to terms that are frequent in a document but relatively uncommon across a collection of documents.

Word Embeddings: Represent words or tokens as dense numerical vectors that can capture semantic relationships.

Transformer-based models use contextual representations, allowing the representation of a word to depend on the surrounding text.

Practical Example: TF-IDF

Install the required library:

pip install scikit-learn
Use TF-IDF to convert sentences into numerical features:
from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
"AI can analyze customer feedback",
"Machine learning helps analyze data",
"Customer feedback helps improve services"
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)
print("Feature names:")
print(vectorizer.get_feature_names_out())
print("Feature matrix shape:", X.shape)
print("TF-IDF matrix:")
print(X.toarray())

Each row represents a document, while each column corresponds to a vocabulary term. The numerical values represent the terms' TF-IDF weights.

TF-IDF is useful for many traditional NLP tasks, although it does not understand language meaning in the same way that contextual language models do.