Training Trains Internship
Student login
← Natural Language Processing learning path
Natural Language Processing · Module 7 of 10

Transformers and BERT

Understand the models behind modern NLP.

📖 Key ideas

  1. Transformers use self-attention to relate all words in a sentence.
  2. BERT is an encoder trained with masked language modelling.
  3. GPT-style models are decoders that predict the next token.
  4. Hugging Face pipeline() makes them easy to use.
  5. Fine-tune with your labelled data for best task results.
💡 Real-life example: BERT reads the whole sentence both ways like a careful reader; GPT writes the next word like a storyteller.
BERTBidirectional encoder model
Masked LMPredict hidden words
GPTGenerative decoder model
pipeline()Hugging Face quick-start API