Transformer & Self-Attention: I Built an Encoder From Scratch, Then Caught It Overfitting
This week's topic is one of the most important building blocks of modern NLP: the Transformer and the self-attention mechanism behind it. Instead of just walking through the theory, I decided to build a Transformer Encoder from scratch for Indonesian intent classification. It didn't go as smoothly as planned — the first model overfit almost immediately, and chasing down why turned out to be the most interesting part of this post.
Table of Contents
Guess First
Before reading on: if you train a 6-layer Transformer Encoder (millions of parameters) from scratch, using only 75 training sentences, for a fixed 20 epochs with no validation split — what do you think happens to the training loss?
A) It decreases slowly and settles at a reasonable value
B) It collapses toward zero well before the epochs run out
C) It actually increases because the model is too small
The answer is in the "Plot Twist" section below.
What Are Transformers & Self-Attention
A Transformer is an architecture built to handle sequential data like text, without processing it word by word the way RNNs or LSTMs do. Introduced in "Attention Is All You Need" (Vaswani et al., 2017), its core idea is self-attention: every word in a sentence can directly "look at" every other word, regardless of position, to understand context.
Self-attention runs on three components:
Query (Q) — the representation of the word currently being processed
Key (K) — the representation of every other word in the sequence
Value (V) — the content or context carried by those other words
Since a Transformer processes an entire sequence at once rather than one token at a time, it needs Positional Encoding to keep track of word order. The standard approach uses a sinusoidal formula (sin/cos) that gives each position a unique numerical fingerprint. Multi-Head Attention then runs several of these attention computations in parallel, so the model can capture different kinds of relationships between words at the same time.
The Project: an Indonesian Intent Classifier
To put these ideas into practice, I built an intent classifier — a model that guesses the intent behind a short sentence, across three categories: greeting, sekarang_jam_berapa (asking for the time), and siapa_anda (asking for identity).
The architecture is a TransformerClassifier, encoder-only, built on PyTorch's nn.TransformerEncoder (which already implements multi-head self-attention, a feed-forward network, and residual connections internally), sitting on top of embeddings from an Indonesian BERT tokenizer (cahya/bert-base-indonesian-522M).
Plot Twist: Caught Overfitting
The first training run looked "successful" — loss kept dropping. But a closer look revealed something off:
Loss collapsed to nearly zero by epoch 9 out of 20 — the answer to the guess above is B. That's not a sign of good learning, it's a sign the model memorized the training set. Which makes sense: a 6-layer Transformer with a hidden dimension of 768 has millions of parameters, trained on just 75 sentences, with no validation split to signal when to stop.
So I audited my own system: what caused it, and how do you actually fix it?
Investigating & Fixing It
Three things were tried at once:
Validation split + early stopping — training stops as soon as validation loss stops improving, instead of running a fixed number of epochs
A smaller model (
scratch_small: 2 layers, hidden size 128, higher dropout and weight decay) — to test whether regularization alone was enoughActual transfer learning (
finetuned_bert) — fine-tuning pretrained Indonesian BERT weights, not just borrowing its tokenizer
To avoid depending on one lucky or unlucky data split, all three variants were compared using 5-fold Stratified Cross-Validation:
| Variant | CV Accuracy (mean ± std) |
|---|---|
| scratch_large (baseline, fixed) | 0.9895 ± 0.0211 |
| scratch_small | 0.9789 ± 0.0258 |
| finetuned_bert | 1.0000 ± 0.0000 |
Early stopping and a validation split appeared to help: the exact same scratch_large architecture, once its training procedure changed (validation split + early stopping) and it was evaluated as a 5-fold average instead of a single split, went from a single-split accuracy of roughly 89–95% to an average CV accuracy of 98.95%. Worth noting: part of that jump also comes from the measurement itself — a 5-fold average is inherently more stable than one small split, so this isn't a perfectly isolated comparison. What's clearer is finetuned_bert: consistently ahead with zero variance across every fold, matching the hypothesis that pretrained language representations need far less data to generalize.
Why Did Every Model End Up Near-Perfect?
This is the part worth being honest about. All three variants scored above 97%, including the deliberately shrunk model. That's suspicious on its own.
So the dataset itself got a closer look. Vocabulary overlap between the three classes turned out to be only 8–10% (Jaccard similarity), and the overlapping words were mostly generic function words ("ada", "bisa", "ini", "yang", "kamu", "apa") — not topic-specific keywords. The actual keywords for each class (jam/waktu/pukul for time, siapa/nama/dirimu for identity, hi/halo/selamat for greetings) barely overlap at all. No duplicate rows were found either.
In other words, this task is inherently easy to separate lexically. That's not a bug and not data leakage — it's a property of a small, simple dataset. These results are valid evidence that the methodology (cross-validation, early stopping, transfer learning) works correctly, not proof that the model is generally "great." A larger, more ambiguous dataset would likely show clearer differences between the three variants.
Quiz
1. Why does a Transformer need Positional Encoding while an RNN doesn't? a) A Transformer doesn't have enough parameters b) A Transformer processes all tokens at once, so it has no built-in sense of order c) Positional Encoding is only there to speed up training
Answer: b — an RNN processes tokens sequentially, so order is implicit in how it works, while a Transformer processes everything in parallel and needs explicit position information.
2. In the experiment above, why did scratch_large jump from ~89% to 98.95% with the exact same architecture? a) Because more data was added b) Because the training procedure was fixed (validation split + early stopping), not the architecture c) Because the model was restarted from scratch
Answer: b — the architecture didn't change at all. What changed was the training procedure (validation split + early stopping) and the evaluation procedure (a 5-fold average instead of a single split). This shows the original problem was indeed about training strategy, though part of the numeric jump is also explained by the more stable measurement.
3. Why should near-perfect scores across every model variant raise suspicion instead of being celebrated? a) Because it means the code must be buggy b) Because it can signal an easy or small task rather than a genuinely strong model c) Because accuracy above 95% is always invalid
Answer: b — a perfect score on a small, simple dataset doesn't automatically prove good generalization; the context needs to be checked first.
Summary
[x] A Transformer uses self-attention (Query/Key/Value) to understand context between words globally
[x] Positional Encoding (sinusoidal) is needed because a Transformer doesn't process tokens sequentially
[x] Training without a validation split and early stopping is prone to overfitting, even on small datasets
[x] 5-fold Cross-Validation gives a far more stable performance estimate than a single split
[x] Transfer learning (fine-tuned BERT) won this experiment, but near-perfect scores across every variant still need the dataset's context checked, not taken at face value
Full code, notebook, and experiment results are on GitHub: transformer-self-attention-intent-classifier
