Training Trains Internship
Student login
← Natural Language Processing learning path
Natural Language Processing · Module 2 of 10

Text preprocessing

Clean and prepare text before modelling.

📖 Key ideas

  1. Tokenisation splits text into words or subwords.
  2. Lowercasing and removing punctuation (for classic models).
  3. Stop words (the, is, a) are often removed in classic NLP.
  4. Stemming chops words (running → run); lemmatisation uses dictionaries (better → good).
  5. Modern Transformers use their own tokenizers and need less cleaning.
💡 Real-life example: Preprocessing is chopping vegetables into the right size before cooking.
TokenisationSplitting text into tokens
Stop wordsVery common words
StemmingCrude suffix removal
LemmatisationReducing to dictionary form