This one is the hands-on half. A model gets trained to tell 102 flower species apart, from daffodils to sunflowers, using the real Oxford Flowers 102 dataset, then wrapped into an API another application can call. Every number in this post comes from an actual GPU run, not an estimate. Full code lives in the GitHub repo.

💡 Haven't read the previous post? This one assumes backbone/head and transfer learning concepts are already familiar. "Quick Recap" boxes throughout should help if not.

Table of Contents

1. Case Study: Oxford Flowers 102

2. Environment Setup

3. Preparing the Dataset

4. Plot Twist: A Closer Look at the Original Script

5. Building the Model with Transfer Learning

6. The Full Training Loop

7. Real Results: Accuracy, Precision, Recall

8. From Model to API with FastAPI

9. Cheat Sheet

10. Test Your Understanding

Case Study: Oxford Flowers 102

The dataset: Oxford Flowers 102, a collection of flower photos across 102 categories. It's a classic image classification benchmark precisely because it has so many classes, several of them visually similar to each other, the same kind of mix-up as ginger, galangal, and turmeric from the previous post, just scaled up to 102 categories.

The backbone: EfficientNet-B1, same as before. No architecture designed from scratch here, just a pretrained backbone with a swapped-out classifier.

Actual samples from the validation set, labeled using the verified cat_to_name.json mapping.

Environment Setup

pip install torch torchvision scikit-learn scipy pandas matplotlib

scikit-learn is new here, needed specifically for evaluation metrics: accuracy, precision, recall.

Preparing the Dataset

Flowers 102 doesn't come as a neat folder-per-class structure like the previous post's examples. Labels live in a separate .mat file (imagelabels.mat). This is a good opportunity to build a custom Dataset, the same pattern as the CSV approach covered earlier, just with a different label source.

import os
import torch
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
from PIL import Image
import scipy.io

class FlowersDataset(Dataset):
    def __init__(self, img_dir, all_files, labels, indices, transform=None):
        self.img_dir = img_dir
        self.files = all_files
        self.labels = labels
        self.indices = indices
        self.transform = transform

    def __len__(self):
        return len(self.indices)

    def __getitem__(self, i):
        idx = self.indices[i]
        img_path = os.path.join(self.img_dir, self.files[idx])
        image = Image.open(img_path).convert("RGB")
        label = int(self.labels[idx])
        if self.transform:
            image = self.transform(image)
        return image, label

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'eval': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

mat_labels = scipy.io.loadmat('imagelabels.mat')
labels = mat_labels['labels'][0] - 1  # shift to 0-indexed

Two details differ from the transforms in the previous post:

• RandomResizedCrop and RandomHorizontalFlip apply only to training data. This augmentation exposes the model to more angle and position variation from the same images, so it doesn't just memorize them.

• The Normalize values [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225] are the R/G/B mean and standard deviation from ImageNet, the large dataset EfficientNet was originally pretrained on. Normalizing with the same numbers keeps the input "speaking the same language" as those pretrained weights.

🎯 Guess First: Why does augmentation only apply to training data, not validation?

Jawaban: Validation needs to evaluate the model under consistent, representative conditions. Randomized crops and flips would make validation scores noisy and hard to compare across epochs. Augmentation's job is helping the model generalize during training, not shaping what it's measured against.

Plot Twist: A Closer Look at the Original Script

Before getting to training, here's an interesting part of putting this post together. The original bootcamp script got read line by line while adapting it, and a few things turned out to need fixing for correct, reproducible results. This is part of the process of verifying code before publishing it, so it's documented here rather than glossed over.

1. File order can drift out of sync with labels. The original script reads image filenames with plain os.listdir(). The problem: os.listdir() doesn't guarantee sorted order across systems, while the label file (imagelabels.mat) assumes files follow the order image_00001.jpg, image_00002.jpg, .... If the order drifts, images and labels can get silently swapped with no error at all. Testing this directly on a small sample confirmed it: plain os.listdir() order really did differ from sorted(). The fix is a one-word wrap: sorted(...).

2. Augmentation leaked into validation. The original script builds one dataset with the training transform, then splits it into train/val afterward. That means validation images inherit random augmentation too, when they should be processed consistently. Fixed by building two separate dataset instances, each with its own transform, joined through the same split indices.

3. The train/val split skipped the dataset's official split. Flowers 102 ships with an official split from its original paper (setid.mat), useful for comparing results against other research. The original script did its own random split instead. The final version uses the official one.

4. pretrained=True is deprecated in newer torchvision versions, replaced with the weights= API.

5. The 102 flower class names. An early draft retyped the name list by hand from memory, and it turned out to contain 104 entries instead of 102. An assert caught it before it went anywhere. The final version fetches the name mapping from a public reference (cat_to_name.json) at runtime instead of relying on manual transcription.

All of these fixes are verified, and the full notebook, including the actual run output, is in the linked GitHub repo.

Building the Model with Transfer Learning

The transfer learning concept from the previous post, applied to a real case now.

🔁 Quick Recap: transfer learning means reusing an already-pretrained backbone (from ImageNet, say) and swapping out just the final classifier layer to match the target classes. No architecture gets designed from scratch.

from torchvision.models import efficientnet_b1, EfficientNet_B1_Weights
import torch.nn as nn

model = efficientnet_b1(weights=EfficientNet_B1_Weights.IMAGENET1K_V1)
num_ftrs = model.classifier[1].in_features

model.classifier = nn.Sequential(
    nn.Dropout(p=0.4, inplace=True),
    nn.Linear(num_ftrs, 102)
)

The pretrained EfficientNet-B1 backbone is used as-is; only the classifier gets swapped to output 102 classes. Out of 6,643,846 total parameters in this model, only 130,662 are actually trained from scratch, just the classifier portion.

The Full Training Loop

Training ran for 15 epochs, logging loss, accuracy, precision, and recall each epoch, with the best checkpoint saved along the way.

import torch.optim as optim
from sklearn.metrics import accuracy_score, precision_score, recall_score
import os

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
checkpoint_dir = "./checkpoints"
os.makedirs(checkpoint_dir, exist_ok=True)

def train_model(model, criterion, optimizer, num_epochs=15):
    best_acc = 0.0

    for epoch in range(num_epochs):
        for phase in ['train', 'val']:
            model.train() if phase == 'train' else model.eval()

            running_loss = 0.0
            all_preds, all_labels = [], []

            for inputs, labels in dataloaders[phase]:
                inputs, labels = inputs.to(device), labels.to(device)
                optimizer.zero_grad()

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    loss = criterion(outputs, labels)
                    _, preds = torch.max(outputs, 1)
                    all_preds.extend(preds.cpu().numpy())
                    all_labels.extend(labels.cpu().numpy())

                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                running_loss += loss.item() * inputs.size(0)

            epoch_acc = accuracy_score(all_labels, all_preds)

            if phase == 'val':
                precision = precision_score(all_labels, all_preds, average='weighted')
                recall = recall_score(all_labels, all_preds, average='weighted')

                if epoch_acc > best_acc:
                    best_acc = epoch_acc
                    torch.save(model.state_dict(), f"{checkpoint_dir}/best.pt")
                torch.save(model.state_dict(), f"{checkpoint_dir}/last.pt")

    return best_acc

train_model(model, criterion, optimizer, num_epochs=15)

Two details that answer questions the previous post might have raised:

• model.train() vs model.eval() is the concrete implementation of "set training mode" mentioned earlier. During the val phase, torch.set_grad_enabled(False) automatically turns off gradient computation.

• Two checkpoints get saved: best.pt for the model with the best validation accuracy seen so far, last.pt for whatever state the model is in at the final epoch.

Real Results: Accuracy, Precision, Recall

Here are the actual numbers from a 15-epoch GPU run (Google Colab, Tesla T4):

Training accuracy climbs fast in the first few epochs (from 15.6% to around 90%), while validation accuracy peaks earlier, at epoch 8, then plateaus and drifts slightly downward through epoch 15. That pattern points to mild overfitting past epoch 8: the model keeps fitting the training data more closely while its performance on unseen data stops improving. That's why the best.pt checkpoint, not last.pt, gets used for final evaluation.

Validation accuracy (orange) peaks at epoch 8, then validation loss starts climbing again even as training loss keeps dropping, a textbook overfitting signature.

The weighted averages above hide a fair amount of variation between classes. Looking at the full 102-class classification report, several classes hit perfect precision and recall, 1.00, including bird of paradise and black-eyed susan. Others are much harder: mallow sits at 0.41 precision, japanese anemone at 0.46 recall. Visual similarity between certain flower classes is a likely factor here, along with the fairly small per-class training sample (around 10 images per class on average, given this dataset's official split).

📌 With 102 classes and uneven performance across them, accuracy alone can be misleading. A model can look good overall while struggling badly on specific classes. Per-class precision and recall catch what the average hides.

From Model to API with FastAPI

The model is trained and the best checkpoint (best.pt) is saved. Next: making it usable in the real world.

pip install fastapi uvicorn pillow torch torchvision

from fastapi import FastAPI, File, UploadFile
from fastapi.responses import JSONResponse
from PIL import Image
import torch
import torch.nn as nn
from torchvision import models, transforms
import io, json, urllib.request

app = FastAPI()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = models.efficientnet_b1(weights=None)
num_ftrs = model.classifier[1].in_features
model.classifier = nn.Sequential(
    nn.Dropout(p=0.4, inplace=True),
    nn.Linear(num_ftrs, 102)
)
model.load_state_dict(torch.load('checkpoints/best.pt', map_location=device))
model = model.to(device)
model.eval()

transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

with urllib.request.urlopen(
    "https://raw.githubusercontent.com/udacity/aipnd-project/master/cat_to_name.json"
) as resp:
    cat_to_name = json.load(resp)
class_names = [cat_to_name[str(i)] for i in range(1, 103)]

@app.post("/predict/")
async def predict(file: UploadFile = File(...)):
    try:
        image = Image.open(io.BytesIO(await file.read())).convert("RGB")
        input_tensor = transform(image).unsqueeze(0).to(device)

        with torch.no_grad():
            outputs = model(input_tensor)
            _, predicted = torch.max(outputs, 1)

        return JSONResponse(content={"predicted_class": class_names[predicted.item()]})
    except Exception as e:
        return JSONResponse(content={"error": str(e)}, status_code=400)

Three things that matter for serving this correctly:

1. The model architecture has to match training exactly, including the number of classes and the classifier structure. A mismatch can make load_state_dict fail outright, or worse, load successfully while producing wrong predictions with no error at all.

2. The transform has to match validation, not training. Use the same consistent Resize and CenterCrop, not random augmentation.

3. model.eval() and torch.no_grad() keep the model in pure inference mode.

Running the server:

uvicorn main:app --reload

curl -X POST "http://localhost:8000/predict/" -F "file=@path_to_your_image.jpg"

{"predicted_class": "sunflower"}

The model is now callable from any application, just send an image to the /predict/ endpoint.

Cheat Sheet

• [ ] Use a custom Dataset when labels don't come in a tidy folder structure

• [ ] Sort file listings manually when labels come from a separate array (like a .mat file)

• [ ] Augmentation applies to training data only, validation stays consistent

• [ ] A dataset's official split beats a custom random split for reproducibility

• [ ] Save two checkpoints: best.pt and last.pt

• [ ] Validation accuracy peaking then declining signals overfitting, not a bug

• [ ] Per-class precision and recall complement accuracy, especially with uneven class performance

• [ ] Serving architecture must match training architecture exactly

• [ ] Serving transform follows validation transform, not training

Test Your Understanding

1. Validation accuracy rises until epoch 8, then plateaus and drifts down through epoch 15. What does that mean?

Jawaban: Mild overfitting. The model keeps improving on training data, but its performance on unseen data stops improving past a certain point. This is why the best checkpoint, not the last one, gets used for deployment.

2. A model has 88% overall accuracy but only 46% recall on one class. What does that tell you?

Jawaban: The model is solid overall but specifically weak on that class, missing a large share of its actual samples. High overall accuracy can mask this when that class makes up a small fraction of the total data.

3. Why does the serving transform need to match validation, not training?

Jawaban: Training transforms include random augmentation meant to help the model generalize, not to represent an image "as-is." At inference time, images need consistent processing, matching exactly how the model was evaluated during validation.

4. Why does file order need manual sorting when labels come from a separate file?

Jawaban: Functions like os.listdir() don't guarantee consistent file ordering across systems. If labels assume files follow a specific numeric order, any mismatch can silently swap images and labels with no error raised.

5. What's the point of a dataset having an official train/val/test split from its original paper?

Jawaban: It makes results comparable to other research or experiments using the same dataset. A custom random split makes training numbers hard to compare across different runs or studies.

The two big questions left open at the end of the previous post, how to actually train the model and how to serve it, are answered here in full, from raw data to a callable API endpoint. Full code and the executed notebook are in the GitHub repo.

Part of an ongoing Computer Vision series.

Sekarang bagian prakteknya. Model dilatih untuk membedakan 102 jenis bunga, dari daffodil sampai sunflower, memakai dataset asli Oxford Flowers 102, lalu dibungkus jadi API yang siap dipakai aplikasi lain. Semua angka di artikel ini berasal dari eksekusi nyata di GPU, bukan estimasi. Kode lengkapnya ada di repo GitHub.

💡 Belum baca artikel sebelumnya? Tulisan ini mengasumsikan konsep backbone/head dan transfer learning sudah familiar. Kotak "Kilas Balik" di beberapa bagian akan membantu kalau belum sempat baca.

Daftar Isi

1. Studi Kasus: Oxford Flowers 102

2. Persiapan Lingkungan

3. Menyiapkan Dataset

4. Plot Twist: Iseng Cek Ulang Skripnya

5. Membangun Model dengan Transfer Learning

6. Training Loop Lengkap

7. Hasil Asli: Accuracy, Precision, Recall

8. Dari Model ke API dengan FastAPI

9. Cheat Sheet

10. Uji Pemahaman Kamu

Studi Kasus: Oxford Flowers 102

Dataset yang dipakai: Oxford Flowers 102, kumpulan foto bunga dengan 102 kategori. Salah satu dataset klasik untuk benchmark image classification, karena jumlah kelasnya banyak dan beberapa jenisnya mirip satu sama lain. Mirip kasus jahe-lengkuas-kunyit di artikel sebelumnya, hanya saja skalanya jadi 102 kelas.

Backbone yang dipakai: EfficientNet-B1, sama seperti yang dibahas di artikel sebelumnya. Arsitektur tidak dirancang dari nol, cukup pakai backbone pretrained dan ganti bagian classifier-nya.

Sampel asli dari validation set, lengkap dengan label yang sudah diverifikasi lewat mapping cat_to_name.json.

Persiapan Lingkungan

pip install torch torchvision scikit-learn scipy pandas matplotlib

scikit-learn ditambahkan khusus untuk menghitung metrik evaluasi: accuracy, precision, recall.

Menyiapkan Dataset

Dataset Flowers 102 tidak datang dalam format folder per kelas yang rapi seperti contoh di artikel sebelumnya. Labelnya ada di file .mat terpisah (imagelabels.mat). Kesempatan bagus untuk praktik bikin custom Dataset, pola yang sama seperti pendekatan CSV yang sudah dibahas, hanya sumber labelnya beda.

import os
import torch
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
from PIL import Image
import scipy.io

class FlowersDataset(Dataset):
    def __init__(self, img_dir, all_files, labels, indices, transform=None):
        self.img_dir = img_dir
        self.files = all_files
        self.labels = labels
        self.indices = indices
        self.transform = transform

    def __len__(self):
        return len(self.indices)

    def __getitem__(self, i):
        idx = self.indices[i]
        img_path = os.path.join(self.img_dir, self.files[idx])
        image = Image.open(img_path).convert("RGB")
        label = int(self.labels[idx])
        if self.transform:
            image = self.transform(image)
        return image, label

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'eval': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

mat_labels = scipy.io.loadmat('imagelabels.mat')
labels = mat_labels['labels'][0] - 1  # ubah ke 0-indexed

Dua detail yang beda dari transform di artikel sebelumnya:

• RandomResizedCrop dan RandomHorizontalFlip dipakai khusus untuk data training. Teknik augmentasi ini membuat model melihat variasi sudut dan posisi yang lebih banyak dari gambar yang sama, supaya tidak mudah menghafal.

• Normalize dengan angka [0.485, 0.456, 0.406] dan [0.229, 0.224, 0.225] adalah rata-rata dan standar deviasi channel R/G/B dari dataset ImageNet, dataset besar yang dipakai melatih EfficientNet sebelumnya. Menormalkan dengan angka yang sama membuat input berbicara "bahasa" yang sama dengan bobot pretrained-nya.

🎯 Coba Tebak Dulu: Kenapa augmentasi hanya diterapkan ke data training, bukan data validation?

Jawaban: Validasi perlu mengevaluasi model dengan kondisi konsisten dan representatif dari data asli. Kalau ikut diacak, skor validasi jadi tidak stabil dan sulit dibandingkan antar-epoch. Augmentasi tugasnya membantu model belajar lebih general saat training, bukan untuk diukur performanya.

Plot Twist: Iseng Cek Ulang Skripnya

Sebelum lanjut ke training, ada bagian menarik dari proses menyusun ulang materi ini. Skrip asli dari sesi bootcamp sempat dicek ulang baris demi baris, dan ternyata ada beberapa hal yang perlu dibetulkan supaya hasilnya benar dan reproducible. Ini bagian dari proses belajar verifikasi kode sebelum dipublikasikan, jadi didokumentasikan di sini alih-alih disembunyikan.

1. Urutan file bisa tidak sinkron dengan label. Skrip asli memakai os.listdir() polos untuk membaca daftar file gambar. Masalahnya, os.listdir() tidak menjamin urutan terurut di semua sistem, padahal file label (imagelabels.mat) mengasumsikan urutan image_00001.jpg, image_00002.jpg, .... Kalau urutannya meleset, gambar dan label bisa tertukar tanpa error apa pun. Diuji langsung dengan data kecil: benar saja, urutan os.listdir() polos beda dari sorted(). Perbaikannya simpel, tinggal bungkus dengan sorted(...).

2. Augmentasi ikut bocor ke data validasi. Skrip asli membuat satu dataset dengan transform training, baru displit ke train/val sesudahnya. Akibatnya validasi ikut kena augmentasi acak, padahal seharusnya diproses secara konsisten. Diperbaiki dengan membuat dua instance dataset terpisah, masing-masing dengan transform sendiri, disatukan lewat indeks split yang sama.

3. Split train/val tidak pakai split resmi. Dataset Flowers 102 sebenarnya sudah punya split resmi dari paper aslinya (setid.mat), yang bisa dipakai supaya hasil bisa dibandingkan dengan riset lain. Skrip asli malah split random sendiri. Versi final memakai split resmi tersebut.

4. pretrained=True sudah deprecated di versi torchvision terbaru, diganti ke API weights=.

5. Nama 102 kelas bunga. Draf pertama sempat mengetik ulang daftar nama secara manual dari ingatan, dan ternyata salah hitung: 104 nama, bukan 102. Ketahuan lewat assert sebelum sempat dipakai kemana-mana. Solusi finalnya mengambil mapping nama langsung dari referensi publik (cat_to_name.json) saat runtime, bukan mengandalkan hasil ketikan.

Semua perbaikan ini sudah diverifikasi, dan notebook lengkapnya (beserta hasil eksekusi asli) ada di repo GitHub yang ditautkan di awal artikel.

Membangun Model dengan Transfer Learning

Konsep transfer learning dari artikel sebelumnya, sekarang diterapkan di kasus nyata.

🔁 Kilas Balik: transfer learning artinya memakai backbone yang sudah pretrained (misalnya dari ImageNet), lalu cukup mengganti bagian classifier terakhirnya sesuai jumlah kelas. Tidak perlu merancang arsitektur dari nol.

from torchvision.models import efficientnet_b1, EfficientNet_B1_Weights
import torch.nn as nn

model = efficientnet_b1(weights=EfficientNet_B1_Weights.IMAGENET1K_V1)
num_ftrs = model.classifier[1].in_features

model.classifier = nn.Sequential(
    nn.Dropout(p=0.4, inplace=True),
    nn.Linear(num_ftrs, 102)
)

Backbone EfficientNet-B1 yang sudah pretrained dipakai apa adanya, hanya bagian classifier yang diganti supaya jumlah output-nya sesuai 102 kelas. Dari total 6.643.846 parameter di model ini, cuma 130.662 yang benar-benar dilatih dari nol (bagian classifier-nya saja).

Training Loop Lengkap

Training berjalan 15 epoch, dengan pencatatan loss, accuracy, precision, dan recall di tiap epoch, plus penyimpanan checkpoint model terbaik.

import torch.optim as optim
from sklearn.metrics import accuracy_score, precision_score, recall_score
import os

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
checkpoint_dir = "./checkpoints"
os.makedirs(checkpoint_dir, exist_ok=True)

def train_model(model, criterion, optimizer, num_epochs=15):
    best_acc = 0.0

    for epoch in range(num_epochs):
        for phase in ['train', 'val']:
            model.train() if phase == 'train' else model.eval()

            running_loss = 0.0
            all_preds, all_labels = [], []

            for inputs, labels in dataloaders[phase]:
                inputs, labels = inputs.to(device), labels.to(device)
                optimizer.zero_grad()

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    loss = criterion(outputs, labels)
                    _, preds = torch.max(outputs, 1)
                    all_preds.extend(preds.cpu().numpy())
                    all_labels.extend(labels.cpu().numpy())

                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                running_loss += loss.item() * inputs.size(0)

            epoch_acc = accuracy_score(all_labels, all_preds)

            if phase == 'val':
                precision = precision_score(all_labels, all_preds, average='weighted')
                recall = recall_score(all_labels, all_preds, average='weighted')

                if epoch_acc > best_acc:
                    best_acc = epoch_acc
                    torch.save(model.state_dict(), f"{checkpoint_dir}/best.pt")
                torch.save(model.state_dict(), f"{checkpoint_dir}/last.pt")

    return best_acc

train_model(model, criterion, optimizer, num_epochs=15)

Dua detail yang menjawab pertanyaan yang mungkin muncul dari artikel sebelumnya:

• model.train() vs model.eval() adalah implementasi konkret dari "set mode training" yang dibahas sebelumnya. Di fase val, torch.set_grad_enabled(False) otomatis mematikan perhitungan gradien.

• Dua file checkpoint disimpan: best.pt untuk model dengan akurasi validasi terbaik sejauh proses training, last.pt untuk kondisi model di epoch paling akhir.

Hasil Asli: Accuracy, Precision, Recall

Berikut angka sebenarnya dari training 15 epoch di GPU (Google Colab, Tesla T4):

Kurva akurasi training naik cepat di beberapa epoch pertama (dari 15,6% ke sekitar 90%), sementara akurasi validasi memuncak lebih awal, di epoch 8, lalu stagnan dan sedikit menurun sampai epoch 15. Pola ini menandakan overfitting ringan setelah epoch 8, model mulai terlalu menyesuaikan diri dengan data training, sementara performa di data yang belum pernah dilihat berhenti membaik. Inilah alasan checkpoint best.pt yang dipakai untuk evaluasi akhir, bukan last.pt.

Val accuracy (oranye) memuncak di epoch 8, lalu val loss mulai naik lagi meski train loss terus turun, tanda klasik overfitting.

Rata-rata weighted di atas menyembunyikan variasi yang cukup besar antar kelas. Dari classification report lengkap (102 kelas), beberapa kelas mencapai precision dan recall sempurna, 1,00, misalnya bird of paradise dan black-eyed susan. Sebagian kelas lain jauh lebih sulit, seperti mallow dengan precision 0,41 dan japanese anemone dengan recall 0,46. Kemungkinan penyebabnya adalah kemiripan visual antar kelas bunga tertentu, ditambah jumlah sampel training yang cukup kecil per kelas (rata-rata sekitar 10 gambar per kelas di split resmi dataset ini).

📌 Dengan 102 kelas dan performa yang tidak merata, akurasi saja bisa menipu. Model bisa terlihat bagus secara keseluruhan padahal buruk di beberapa kelas tertentu. Precision dan recall per kelas membantu menangkap itu.

Dari Model ke API dengan FastAPI

Model sudah dilatih dan checkpoint terbaik (best.pt) sudah tersimpan. Bagian selanjutnya: bagaimana model ini dipakai di dunia nyata.

pip install fastapi uvicorn pillow torch torchvision

from fastapi import FastAPI, File, UploadFile
from fastapi.responses import JSONResponse
from PIL import Image
import torch
import torch.nn as nn
from torchvision import models, transforms
import io, json, urllib.request

app = FastAPI()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = models.efficientnet_b1(weights=None)
num_ftrs = model.classifier[1].in_features
model.classifier = nn.Sequential(
    nn.Dropout(p=0.4, inplace=True),
    nn.Linear(num_ftrs, 102)
)
model.load_state_dict(torch.load('checkpoints/best.pt', map_location=device))
model = model.to(device)
model.eval()

transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

with urllib.request.urlopen(
    "https://raw.githubusercontent.com/udacity/aipnd-project/master/cat_to_name.json"
) as resp:
    cat_to_name = json.load(resp)
class_names = [cat_to_name[str(i)] for i in range(1, 103)]

@app.post("/predict/")
async def predict(file: UploadFile = File(...)):
    try:
        image = Image.open(io.BytesIO(await file.read())).convert("RGB")
        input_tensor = transform(image).unsqueeze(0).to(device)

        with torch.no_grad():
            outputs = model(input_tensor)
            _, predicted = torch.max(outputs, 1)

        return JSONResponse(content={"predicted_class": class_names[predicted.item()]})
    except Exception as e:
        return JSONResponse(content={"error": str(e)}, status_code=400)

Tiga hal yang wajib diperhatikan supaya serving-nya tidak meleset:

1. Arsitektur model harus identik dengan yang dipakai saat training, termasuk jumlah kelas dan struktur classifier. Kalau beda, load_state_dict bisa gagal, atau lebih berbahaya lagi, berhasil dimuat tapi menghasilkan prediksi yang salah tanpa error.

2. Transform harus sama persis dengan yang dipakai saat validasi, bukan training. Pakai Resize dan CenterCrop yang konsisten, bukan augmentasi random.

3. model.eval() dan torch.no_grad() memastikan model dalam mode inference murni.

Menjalankan servernya:

uvicorn main:app --reload

curl -X POST "http://localhost:8000/predict/" -F "file=@path_to_your_image.jpg"

{"predicted_class": "sunflower"}

Model sekarang bisa dipanggil dari aplikasi apa pun, tinggal kirim gambar ke endpoint /predict/.

Cheat Sheet

• [ ] Custom Dataset dipakai saat label tidak datang dalam format folder rapi

• [ ] Urutan file harus di-sort manual kalau labelnya berasal dari array terpisah (mis. file .mat)

• [ ] Augmentasi hanya untuk data training, validasi tetap konsisten

• [ ] Split resmi dari paper dataset lebih baik dipakai daripada split random sendiri

• [ ] Simpan dua checkpoint: best.pt dan last.pt

• [ ] Val accuracy yang memuncak lalu menurun adalah tanda overfitting, bukan bug

• [ ] Precision dan recall per kelas melengkapi accuracy, terutama saat performa antar kelas tidak merata

• [ ] Arsitektur model saat serving harus identik dengan saat training

• [ ] Transform saat serving mengikuti transform validasi, bukan training

Uji Pemahaman Kamu

1. Val accuracy naik sampai epoch 8, lalu stagnan dan sedikit menurun sampai epoch 15. Apa artinya?

Jawaban: Tanda overfitting ringan. Model terus membaik di data training, tapi performanya di data yang belum pernah dilihat berhenti meningkat setelah titik tertentu. Ini alasan checkpoint terbaik (bukan checkpoint terakhir) yang dipakai untuk deployment.

2. Model punya akurasi keseluruhan 88%, tapi recall untuk satu kelas hanya 46%. Apa artinya?

Jawaban: Model bagus secara umum, tapi buruk khusus untuk kelas tersebut. Banyak sampel dari kelas itu gagal terdeteksi dengan benar. Akurasi keseluruhan yang tinggi bisa menyembunyikan masalah ini karena porsi kelas tersebut kecil dari total data.

3. Kenapa transform saat serving harus sama dengan transform validasi, bukan training?

Jawaban: Transform training mengandung augmentasi acak yang tujuannya membuat model belajar lebih general, bukan untuk representasi asli dari gambar. Saat inference, gambar perlu diproses secara konsisten, persis seperti cara model dievaluasi saat validasi.

4. Kenapa urutan file gambar perlu di-sort manual kalau labelnya dari file terpisah?

Jawaban: Fungsi seperti os.listdir() tidak menjamin urutan file yang konsisten di semua sistem. Kalau label diasumsikan mengikuti urutan penomoran file, ketidaksesuaian urutan bisa membuat gambar dan label tertukar tanpa error apa pun.

5. Apa gunanya dataset punya split resmi (train/val/test) dari paper aslinya?

Jawaban: Supaya hasil training bisa dibandingkan secara adil dengan penelitian atau eksperimen lain yang memakai dataset yang sama. Split random sendiri membuat angka hasil training sulit dibandingkan lintas eksperimen.

Dua pertanyaan besar dari akhir artikel sebelumnya, bagaimana cara training dan bagaimana cara membuat model serving, sudah terjawab lengkap di sini, dari data mentah sampai endpoint API yang siap dipanggil. Kode lengkap dan notebook yang sudah dijalankan ada di repo GitHub.

Bagian dari seri belajar Computer Vision.

This post breaks that idea down from scratch, starting with pixels and ending with a CNN ready to train in PyTorch.

💡 How to read this: a few "Guess First" boxes are scattered through the post. Take a guess before checking the answer, it helps the concept stick.

Table of Contents

1. What Image Classification Actually Is

2. Computer Vision vs Image Processing

3. CNN Anatomy: Backbone vs Head

4. Getting Comfortable with PyTorch

5. Building a CNN from Scratch

6. Transfer Learning

7. Preparing a Dataset

8. The Training Loop

9. Cheat Sheet

10. Test Your Understanding

What Image Classification Actually Is

The definition is simple: feed in an image, get back what it's a picture of. Input is an image, output is a class or label.

That simple idea powers a lot of everyday applications: face recognition, disease detection in medical scans, object recognition in self-driving systems, and yes, the grocery app that tells vegetables apart.

Before going further, one thing is worth internalizing: to a computer, an image is just numbers.

• An image is made of pixels, the smallest unit of a picture.

• On a computer, an image becomes a matrix: rows by columns, each cell holding a value for the light intensity at that point.

• A color image is really three matrices stacked together: Red (R), Green (G), Blue (B), forming a single 3-dimensional block of data.

🎯 Guess First: A 100×100 color image is made up of how many matrices?

Jawaban: Three (R, G, B), each 100×100, stacked into a single 100×100×3 block.

Computer Vision vs Image Processing

These two terms get mixed up often, but they serve different goals.

One image processing technique underpins CNNs directly: convolution, sliding a kernel (a small matrix) across an image to produce a specific effect.

A natural question follows: do the numbers inside that kernel have to be set by hand? They don't. Kernel values start out randomly initialized, then get updated automatically through training until the model settles on the best values for its data. A hand-designed kernel is fixed and doesn't adapt; a trained one shapes itself around whatever data it sees.

That's the foundation for this post's main subject: image classification with CNNs (Convolutional Neural Networks).

CNN Anatomy: Backbone vs Head

A CNN works in two phases: look closely first, then decide. These are called the Backbone and the Head.

The Backbone, feature extractor

The backbone learns visual features from an image: edges, corners, textures, patterns, eventually objects. It works progressively. Early layers pick up simple features like lines and basic shapes. Deeper layers capture increasingly complex, abstract features, from object parts to whole objects. As the network goes deeper, the spatial size of the image shrinks while the information packed into it gets denser.

Each convolutional layer contains kernels (filters) that sweep across the image and produce a feature map. Two parameters matter most:

• in_channels, the number of input channels. For the first layer receiving a color image directly, this is 3 (R, G, B).

• out_channels, the number of output channels, equal to the number of filters used. Use 32 filters, get 32 stacked feature maps out.

Multiple filters help because each one learns to look for something different: texture, color, edges. More filters mean the model captures a richer range of visual patterns.

⚠️ A hard rule when stacking layers: a layer's in_channels (or in_features) must match the previous layer's out_channels (or out_features). Skip this and the code throws an error the moment it runs. It's the single most common mistake when designing a CNN architecture for the first time.

A linear layer only accepts a 1-dimensional vector, while an image is a matrix. Force that combination and the image needs to be flattened first. Example: a 28×28 image flattens into a vector of 784 (28 times 28). For a large image, say 1000×1000 pixels, that flattened vector balloons and becomes impractical.

Convolutional layers are useful precisely because they accept matrix input directly. By the time the gradual extraction process is done, the representation is already compact and information-dense before it gets flattened.

There's no need to design a backbone from scratch either. Proven architectures already exist and can be reused directly: ResNet, VGG, EfficientNet.

The Head, decision maker

Once the backbone finishes extracting features, the result (a vector, after flattening) moves into the head for classification. The head is typically one or more fully connected (FC) layers.

Every neuron in an FC layer connects to every neuron in the layer before it, hence "fully connected." The final output layer needs exactly as many neurons as there are classes. Classifying digits 0 through 9 means 10 output neurons; a 15-category vegetable dataset means 15.

The last layer usually applies softmax, converting raw outputs into a probability for each class. The class with the highest probability becomes the model's prediction.

🎯 Guess First: A model classifies animals into 3 classes: cat, dog, panda. Its softmax output is [0.15, 0.15, 0.70]. What does it predict?

Jawaban: Panda, since it has the highest probability of the three.

In short: the backbone turns a raw image into a feature representation through convolution, pooling, and non-linear activation. The head takes that representation and picks the most likely class based on softmax probabilities.

A raw image gets distilled into shrinking feature maps, flattened into a vector, then classified through FC layers into a final prediction.

Getting Comfortable with PyTorch

PyTorch is a tool for building and training deep learning models. Its API feels a lot like NumPy, so prior NumPy experience speeds things up considerably. The difference is that PyTorch plugs directly into hardware accelerators: CUDA GPUs, Apple Silicon, or plain CPU.

A few basics worth knowing upfront:

• Default data types: integers become int64, decimals become float32.

• A tensor's default device is CPU. Moving it to GPU is a single call: .to('cuda').

• Autograd is PyTorch's mechanism for computing gradients automatically, the core of training. That computation happens automatically; there's no manual gradient math involved.

import torch

x = torch.randn(2, 2)
x = x.to('cuda')

a = torch.ones(2, 3)
b = torch.zeros(2, 3)
c = a + b

Building a CNN from Scratch

Linear Layer

import torch.nn as nn

layer = nn.Linear(in_features=5, out_features=10)

This layer takes 5 input values and produces 10 output values. Because it's fully connected, there are 5 times 10 connections behind the scenes.

A Custom Model

Building a custom architecture means writing a class that extends nn.Module. This is mandatory: PyTorch is designed so that any model recognized this way can actually go through training.

import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.block1 = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3),
            nn.ReLU(),
        )
        self.block2 = nn.Sequential(
            nn.Conv2d(in_channels=10, out_channels=10, kernel_size=3),
            nn.ReLU(),
        )
        self.flatten = nn.Flatten()
        self.classifier = nn.Linear(in_features=10 * 24 * 24, out_features=num_classes)

    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.flatten(x)
        x = self.classifier(x)
        return x

Three things worth double-checking:

1. Block two's in_channels (10) has to match block one's out_channels (10).

2. in_features on the classifier has to match the flattened output size. For a 28×28 input with no padding, each Conv2d with kernel size 3 shrinks each side by 2 pixels: 28 becomes 26, then 26 becomes 24. That leaves a 24×24 map, so in_features = 10 times 24 times 24. When the math gets confusing, printing the output shape after each block is the fastest way to check.

3. The final layer's out_features has to match the number of classes in the dataset.

Transfer Learning

Rather than designing an architecture from zero, a pretrained, already-proven backbone can be reused directly. torchvision's model zoo has plenty: ResNet, VGG, EfficientNet.

The pattern: load the backbone, then override its final classifier layer to match the target number of classes. Layer naming differs across architectures.

import torch.nn as nn
from torchvision import models

# EfficientNet-B1
model = models.efficientnet_b1(weights="IMAGENET1K_V1")
model.classifier[1] = nn.Linear(in_features=1280, out_features=15)

# ResNet-50
model = models.resnet50(weights="IMAGENET1K_V1")
model.fc = nn.Linear(in_features=2048, out_features=15)

These architectures already extract general visual features well. The remaining work is teaching the model to recognize the specific classes in the target dataset.

Preparing a Dataset

Two common approaches exist for organizing data before training.

Folder per Class

dataset/
├── train/
│   ├── class1/
│   │   ├── image1.jpg
│   │   └── image2.jpg
│   └── class2/
│       ├── image3.jpg
│       └── image4.jpg
├── val/
│   ├── class1/
│   └── class2/

Each image sits in the folder matching its class. This structure is recognized directly by built-in framework utilities: ImageFolder in PyTorch, ImageDataGenerator in TensorFlow.

from torchvision import datasets, transforms
from torch.utils.data import DataLoader

transform = transforms.Compose([
    transforms.Resize((128, 128)),
    transforms.ToTensor()
])

train_dataset = datasets.ImageFolder(root='dataset/train', transform=transform)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

The upside is simplicity and broad framework support. The downside shows up with random filenames or extra metadata needs, like multi-label data.

CSV File

The alternative is a CSV listing image paths alongside their labels.

image_path,class_label
dataset/images/image1.jpg,0
dataset/images/image2.jpg,1
dataset/images/image3.jpg,0

import pandas as pd
from PIL import Image
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms

class CustomDataset(Dataset):
    def __init__(self, csv_file, transform=None):
        self.data = pd.read_csv(csv_file)
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        img_path = self.data.iloc[idx, 0]
        image = Image.open(img_path)
        label = int(self.data.iloc[idx, 1])
        if self.transform:
            image = self.transform(image)
        return image, label

transform = transforms.Compose([
    transforms.Resize((128, 128)),
    transforms.ToTensor()
])

train_dataset = CustomDataset(csv_file='dataset/train_labels.csv', transform=transform)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

This is more flexible for datasets carrying extra metadata or scattered across an unorganized directory structure. The tradeoff is custom code, since no framework ships a built-in loader for this shape.

📌 Which one to pick? Clean, simple, already-organized-by-category data: use folders. Complex data with extra metadata or messy file layouts: use CSV.

Dataset vs DataLoader

A Dataset holds all the data, images and labels together. Pulling one sample from it returns one image plus one label. Feeding an entire dataset to a model at once is usually impossible given GPU memory limits.

A DataLoader splits the dataset into batches, small chunks processed incrementally. DataLoader(batch_size=128) produces batches shaped [128, 3, 64, 64]: 128 images, 3 RGB channels, 64×64 pixels, plus 128 matching labels.

The Training Loop

Each epoch (one full pass through the training data) runs through the following steps.

model.train()
train_loss, train_acc = 0, 0

for images, labels in train_loader:
    images, labels = images.to(device), labels.to(device)

    outputs = model(images)
    loss = criterion(outputs, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    train_loss += loss.item()
    train_acc += (outputs.argmax(1) == labels).float().mean().item()

train_loss /= len(train_loader)
train_acc /= len(train_loader)

The test/validation step follows the same shape, minus loss.backward() and optimizer.step(). That step is pure evaluation, not learning.

The loss function for multi-class problems: Cross Entropy Loss.

Finally, save the model whenever it improves, so it can be reused later without retraining from zero.

if test_acc > best_acc:
    best_acc = test_acc
    torch.save(model.state_dict(), 'best_model.pth')

Cheat Sheet

• [ ] A CNN is a Backbone (feature extraction) plus a Head (classification)

• [ ] A layer's in_channels/in_features must match the previous layer's out_channels/out_features

• [ ] out_channels equals the number of filters used

• [ ] The final output layer needs one neuron per class

• [ ] Linear layers need 1D input, so images must be flattened first

• [ ] Convolutional layers accept matrix input directly

• [ ] Kernel values come from training, not manual design

• [ ] A Dataset holds the data; a DataLoader splits it into batches

• [ ] Transfer learning reuses a pretrained backbone, overriding just the classifier

• [ ] The multi-class loss function of choice: Cross Entropy Loss

Test Your Understanding

1. Why can't an image go straight into a linear layer?

Jawaban: A linear layer only accepts a 1-dimensional vector, while an image is a matrix. The fix is to flatten it first, or use a convolutional layer instead, which is built to accept matrix input directly.

2. If a convolutional layer uses 64 filters, what's its out_channels?

Jawaban: 64 — out_channels always equals the number of filters used.

3. A dataset has 20 fruit classes. How many neurons should the final output layer have?

Jawaban: 20 — the number of output neurons must match the number of classes.

4. What's the difference between the training step and the test/validation step in a training loop?

Jawaban: They follow the same shape: forward pass, compute loss. The difference is the test/validation step skips optimization (backpropagation and parameter updates), since that step is purely for evaluation, not learning.

5. When does a CSV-based approach make more sense than folder-per-class?

Jawaban: When the dataset carries extra information (metadata, multi-label) or the images live scattered across a directory structure that isn't neatly organized by class. CSV trades a bit of custom code for more flexibility.

The next post picks up from here: training a real model to tell 102 flower species apart, all the way through to an API another app can call.

Part of an ongoing Computer Vision series.

Artikel ini membongkar konsepnya dari nol: mulai dari apa itu piksel, sampai model CNN siap dilatih dengan PyTorch.

💡 Cara baca artikel ini: ada beberapa kotak "Coba Tebak Dulu" di sepanjang tulisan. Tebak dulu sebelum lihat jawabannya, biar lebih nempel.

Daftar Isi

1. Apa Itu Image Classification, Sebenarnya?

2. Computer Vision vs Image Processing

3. Anatomi CNN: Backbone vs Head

4. Kenalan Sama PyTorch

5. Membangun CNN dari Nol

6. Transfer Learning

7. Menyiapkan Dataset

8. Training Loop

9. Cheat Sheet

10. Uji Pemahaman Kamu

Apa Itu Image Classification, Sebenarnya?

Definisinya sederhana. Kasih gambar, sistem menjawab gambar itu tentang apa. Input berupa gambar, output berupa kelas atau label.

Dari definisi sesederhana itu, lahir banyak aplikasi yang dipakai sehari-hari: face recognition, deteksi penyakit dari gambar medis, pengenalan objek pada kendaraan otonom, sampai aplikasi belanja yang mengenali jenis sayur dari foto.

Sebelum masuk lebih dalam, ada satu hal dasar yang perlu dipahami. Bagi komputer, gambar itu sekadar angka.

• Gambar terdiri dari piksel, bagian terkecil dari sebuah gambar.

• Di komputer, gambar direpresentasikan sebagai matriks: baris dikali kolom, tiap sel berisi angka intensitas cahaya di titik itu.

• Gambar berwarna adalah gabungan tiga matriks: Merah (R), Hijau (G), Biru (B), ditumpuk jadi satu struktur 3 dimensi.

🎯 Coba Tebak Dulu: Kalau sebuah gambar berwarna berukuran 100×100 piksel, ada berapa banyak matriks yang menyusunnya?

Jawaban: 3 matriks (R, G, B), masing-masing berukuran 100×100. Ditumpuk jadi satu blok data berukuran 100×100×3.

Computer Vision vs Image Processing

Dua istilah ini sering ketuker, padahal tujuannya beda.

Salah satu teknik image processing yang jadi fondasi CNN adalah convolution: proses menggeser sebuah kernel (matriks kecil) ke seluruh bagian gambar untuk menghasilkan efek tertentu.

Pertanyaan yang sering muncul: apakah angka-angka di kernel itu harus ditentukan manual satu per satu? Jawabannya tidak. Nilai kernel di-initialize secara random, lalu diperbarui otomatis lewat proses training sampai model menemukan sendiri kernel terbaik untuk datanya. Kernel yang ditentukan manual sifatnya kaku dan tidak adaptif. Lewat training, kemampuan model dalam mengekstrak fitur menyesuaikan dengan data yang dipelajari.

Inilah yang mendasari topik utama artikel ini: image classification dengan CNN (Convolutional Neural Network).

Anatomi CNN: Backbone vs Head

CNN bekerja lewat dua fase: melihat detail dulu, baru memutuskan. Dua fase ini disebut Backbone dan Head.

Backbone, si pengekstrak fitur

Backbone bertugas mempelajari fitur visual dari gambar: tepi, sudut, tekstur, pola, sampai objek. Cara kerjanya bertahap. Layer awal menangkap fitur sederhana seperti garis dan bentuk dasar. Layer yang lebih dalam menangkap fitur yang makin kompleks, dari bagian objek sampai objek utuh. Seiring makin dalam jaringannya, ukuran spasial gambar makin mengecil, sementara informasinya makin padat.

Di dalam tiap convolutional layer ada kernel (filter) yang menyapu gambar dan menghasilkan feature map. Dua parameter kuncinya:

• in_channels, jumlah channel input. Untuk layer pertama yang langsung menerima gambar berwarna, nilainya 3 (R, G, B).

• out_channels, jumlah channel output, sama dengan jumlah filter yang dipakai. Pakai 32 filter, hasilnya 32 feature map yang ditumpuk.

Filter yang banyak berguna karena tiap filter belajar mencari hal yang berbeda. Ada yang fokus ke tekstur, ada yang ke warna, ada yang ke tepi objek. Semakin banyak filter, semakin kaya aspek visual yang bisa ditangkap model.

⚠️ Aturan wajib saat menyusun layer: in_channels (atau in_features) suatu layer harus sama dengan out_channels (atau out_features) layer sebelumnya. Kalau tidak, siap-siap ketemu error saat kode dijalankan. Ini kesalahan paling umum pemula saat pertama kali merancang arsitektur CNN sendiri.

Linear layer hanya menerima input berbentuk vektor 1 dimensi, sementara gambar berbentuk matriks. Kalau tetap mau pakai linear layer, gambar harus di-flatten dulu. Contoh: gambar 28×28 di-flatten jadi vektor berukuran 784 (28 dikali 28). Untuk gambar besar, katakanlah 1000×1000 piksel, hasil flatten-nya jadi raksasa dan tidak efisien.

Convolutional layer istimewa karena bisa langsung menerima input berbentuk matriks. Lewat proses ekstraksi bertahap, ukurannya sudah mengecil dan padat informasi sebelum akhirnya di-flatten.

Tidak perlu merancang backbone dari nol juga. Sudah ada arsitektur populer yang terbukti bagus dan tinggal dipakai: ResNet, VGG, EfficientNet.

Head, si pengambil keputusan

Setelah backbone selesai mengekstrak fitur, hasilnya (berupa vektor setelah di-flatten) diteruskan ke head untuk diklasifikasikan. Head biasanya berupa satu atau beberapa fully connected (FC) layer.

Tiap neuron di FC layer terhubung ke semua neuron di layer sebelumnya, makanya disebut fully connected. Layer output terakhir wajib punya jumlah neuron sama dengan jumlah kelas. Klasifikasi digit 0 sampai 9 berarti 10 neuron output, dataset 15 jenis sayuran berarti 15 neuron output.

Layer terakhir biasanya pakai softmax, yang mengubah nilai output jadi probabilitas tiap kelas. Kelas dengan probabilitas tertinggi jadi jawaban model.

🎯 Coba Tebak Dulu: Model dilatih untuk klasifikasi hewan dengan 3 kelas: kucing, anjing, panda. Output softmax-nya [0.15, 0.15, 0.70]. Model memprediksi gambar itu apa?

Jawaban: Panda, karena nilai probabilitasnya paling tinggi dibanding kucing dan anjing.

Alur singkatnya: backbone mengubah gambar mentah jadi representasi fitur lewat konvolusi, pooling, dan aktivasi non-linear. Head menerima fitur itu dan memutuskan kelas paling sesuai berdasarkan probabilitas softmax.

Gambar mentah diekstrak jadi feature map yang mengecil, di-flatten jadi vektor, lalu diklasifikasikan lewat FC layer sampai keluar prediksi.

Kenalan Sama PyTorch

PyTorch adalah tools untuk membangun dan melatih model deep learning. Konsepnya mirip NumPy, jadi kalau sudah familiar dengan NumPy, proses belajarnya jauh lebih cepat. Bedanya, PyTorch sudah terintegrasi dengan akselerator hardware: CUDA GPU, Apple Silicon, atau tetap CPU.

Beberapa hal dasar yang perlu diketahui:

• Data type default: bilangan bulat jadi int64, bilangan desimal jadi float32.

• Device default sebuah tensor adalah CPU. Pindah ke GPU tinggal panggil .to('cuda').

• Autograd adalah mekanisme PyTorch untuk menghitung gradien secara otomatis, inti dari proses training. Perhitungan gradien ini sudah otomatis dilakukan PyTorch, tidak perlu dihitung manual.

import torch

x = torch.randn(2, 2)
x = x.to('cuda')

a = torch.ones(2, 3)
b = torch.zeros(2, 3)
c = a + b

Membangun CNN dari Nol

Linear Layer

import torch.nn as nn

layer = nn.Linear(in_features=5, out_features=10)

Layer ini menerima 5 nilai input dan mengeluarkan 10 nilai output. Karena fully connected, ada 5 dikali 10 koneksi di baliknya.

Custom Model

Untuk membangun arsitektur sendiri, dibuat sebuah class yang meng-extend nn.Module. Ini wajib. PyTorch didesain agar setiap model yang dibuat dikenali sebagai model deep learning, supaya proses training bisa dijalankan terhadapnya.

import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.block1 = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3),
            nn.ReLU(),
        )
        self.block2 = nn.Sequential(
            nn.Conv2d(in_channels=10, out_channels=10, kernel_size=3),
            nn.ReLU(),
        )
        self.flatten = nn.Flatten()
        self.classifier = nn.Linear(in_features=10 * 24 * 24, out_features=num_classes)

    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.flatten(x)
        x = self.classifier(x)
        return x

Tiga hal yang perlu diperhatikan:

1. in_channels block kedua (10) harus sama dengan out_channels block pertama (10).

2. in_features pada classifier harus cocok dengan ukuran hasil flatten. Untuk input 28×28 tanpa padding, tiap Conv2d dengan kernel 3 mengecilkan sisi gambar sebanyak 2 piksel: 28 jadi 26, lalu 26 jadi 24. Jadi ukuran akhirnya 24×24, dan in_features = 10 dikali 24 dikali 24. Kalau bingung menghitung manual, cara praktisnya adalah mencetak (print) shape output di tiap block untuk mengecek ukurannya langsung.

3. out_features di layer terakhir harus sama dengan jumlah kelas dataset.

Transfer Learning

Daripada merancang arsitektur dari nol, backbone yang sudah pretrained dan terbukti bagus bisa langsung dipakai. Tersedia lewat model zoo torchvision. Beberapa nama besar: ResNet, VGG, EfficientNet.

Caranya: load backbone-nya, lalu override layer classifier terakhir sesuai jumlah kelas. Penamaan layer berbeda-beda tiap arsitektur.

import torch.nn as nn
from torchvision import models

# EfficientNet-B1
model = models.efficientnet_b1(weights="IMAGENET1K_V1")
model.classifier[1] = nn.Linear(in_features=1280, out_features=15)

# ResNet-50
model = models.resnet50(weights="IMAGENET1K_V1")
model.fc = nn.Linear(in_features=2048, out_features=15)

Arsitektur seperti ini sudah terbukti bagus mengekstrak fitur visual secara umum. Fokusnya jadi tinggal mengajari model mengenali kelas-kelas spesifik di dataset yang dipakai.

Menyiapkan Dataset

Ada dua pendekatan populer untuk menyusun data sebelum training.

Folder per Kelas

dataset/
├── train/
│   ├── class1/
│   │   ├── image1.jpg
│   │   └── image2.jpg
│   └── class2/
│       ├── image3.jpg
│       └── image4.jpg
├── val/
│   ├── class1/
│   └── class2/

Tiap gambar diletakkan di folder sesuai kelasnya. Pendekatan ini didukung langsung oleh fungsi bawaan framework: ImageFolder di PyTorch, ImageDataGenerator di TensorFlow.

from torchvision import datasets, transforms
from torch.utils.data import DataLoader

transform = transforms.Compose([
    transforms.Resize((128, 128)),
    transforms.ToTensor()
])

train_dataset = datasets.ImageFolder(root='dataset/train', transform=transform)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

Kelebihannya: gampang dipahami, banyak framework yang mendukung langsung. Kekurangannya: kurang fleksibel kalau nama file acak atau butuh info tambahan (misalnya multilabel).

File CSV

Alternatifnya, pakai file CSV berisi path gambar dan labelnya.

image_path,class_label
dataset/images/image1.jpg,0
dataset/images/image2.jpg,1
dataset/images/image3.jpg,0

import pandas as pd
from PIL import Image
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms

class CustomDataset(Dataset):
    def __init__(self, csv_file, transform=None):
        self.data = pd.read_csv(csv_file)
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        img_path = self.data.iloc[idx, 0]
        image = Image.open(img_path)
        label = int(self.data.iloc[idx, 1])
        if self.transform:
            image = self.transform(image)
        return image, label

transform = transforms.Compose([
    transforms.Resize((128, 128)),
    transforms.ToTensor()
])

train_dataset = CustomDataset(csv_file='dataset/train_labels.csv', transform=transform)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

Lebih fleksibel, terutama untuk dataset dengan metadata tambahan atau file yang tersebar di direktori yang tidak terorganisir. Konsekuensinya, butuh kode custom karena tidak ada dukungan bawaan langsung dari framework.

📌 Kapan pakai yang mana? Dataset sudah rapi per kategori dan sederhana, pakai folder. Dataset kompleks dengan metadata tambahan atau file berantakan, pakai CSV.

Dataset vs DataLoader

Dataset menampung semua data, gambar dan label sekaligus. Ambil satu sampel dari dataset, hasilnya satu gambar plus satu label. Memberi seluruh dataset sekaligus ke model biasanya tidak memungkinkan karena keterbatasan memori GPU.

DataLoader memecah dataset jadi batch, kelompok kecil yang diproses bertahap. DataLoader(batch_size=128) menghasilkan batch berbentuk [128, 3, 64, 64]: 128 gambar, 3 channel RGB, ukuran 64×64, beserta 128 labelnya.

Training Loop

Setiap epoch (satu putaran penuh lewat seluruh data training) melewati tahapan berikut.

model.train()
train_loss, train_acc = 0, 0

for images, labels in train_loader:
    images, labels = images.to(device), labels.to(device)

    outputs = model(images)
    loss = criterion(outputs, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    train_loss += loss.item()
    train_acc += (outputs.argmax(1) == labels).float().mean().item()

train_loss /= len(train_loader)
train_acc /= len(train_loader)

Tahap test/validation alurnya mirip, hanya saja tanpa loss.backward() dan optimizer.step(). Tahap ini murni evaluasi, bukan belajar.

Loss function untuk kasus multi-kelas: Cross Entropy Loss.

Terakhir, simpan model kalau performanya membaik, supaya bisa dipakai lagi nanti tanpa training ulang dari nol.

if test_acc > best_acc:
    best_acc = test_acc
    torch.save(model.state_dict(), 'best_model.pth')

Cheat Sheet

• [ ] CNN terdiri dari Backbone (ekstraksi fitur) dan Head (klasifikasi)

• [ ] in_channels/in_features layer saat ini harus sama dengan out_channels/out_features layer sebelumnya

• [ ] out_channels sama dengan jumlah filter yang dipakai

• [ ] Output layer terakhir punya neuron sebanyak jumlah kelas

• [ ] Linear layer butuh input 1D, gambar harus di-flatten dulu

• [ ] Convolutional layer bisa langsung menerima input matriks

• [ ] Nilai kernel tidak ditentukan manual, didapat lewat training

• [ ] Dataset menampung data, DataLoader memecahnya jadi batch

• [ ] Transfer learning memakai backbone pretrained, cukup override classifier-nya

• [ ] Loss function untuk multi-kelas: Cross Entropy Loss

Uji Pemahaman Kamu

1. Kenapa gambar tidak bisa langsung dimasukkan ke linear layer?

Jawaban: Linear layer hanya menerima input berbentuk vektor 1 dimensi, sedangkan gambar berbentuk matriks. Solusinya di-flatten dulu, atau langsung pakai convolutional layer yang memang didesain menerima input matriks.

2. Kalau sebuah convolutional layer memakai 64 filter, berapa out_channels-nya?

Jawaban: 64 — out_channels selalu sama dengan jumlah filter yang digunakan.

3. Dataset kamu punya 20 kelas gambar buah. Berapa jumlah neuron di output layer terakhir?

Jawaban: 20 — jumlah neuron output harus sama dengan jumlah kelas.

4. Apa bedanya training step dan test/validation step dalam training loop?

Jawaban: Alurnya mirip: forward pass, hitung loss. Bedanya test/validation step tidak melakukan optimisasi (backpropagation dan update parameter), karena tahap ini murni untuk mengevaluasi, bukan melatih model.

5. Kapan sebaiknya pakai pendekatan CSV dibanding folder per kelas?

Jawaban: Saat dataset punya informasi tambahan (metadata, multilabel) atau gambarnya tersebar di direktori yang tidak terorganisir rapi per kelas. CSV memberi fleksibilitas lebih meski butuh sedikit kode custom.

Artikel berikutnya melanjutkan dari sini: melatih model asli untuk membedakan 102 jenis bunga, lengkap sampai jadi API yang bisa dipanggil aplikasi lain.

Bagian dari seri belajar Computer Vision.

🧠 TL;DR — A digital image is just a bunch of numbers arranged in a matrix. Once that clicks, everything else in Computer Vision — grayscale, convolution, even CNNs — starts making a lot more sense. In this post we'll break the concepts down step by step, code along with OpenCV, then wrap it all into an API with FastAPI.

Ever wondered how your phone knows there's a face in a photo? Or how a CCTV system can "get suspicious" when someone walks into a restricted area? It all starts with one deceptively simple idea: a computer doesn't actually "see" an image the way we do — it just sees numbers.

Once you understand how a computer "sees" those numbers, the door into Computer Vision opens up a lot wider. Let's break it down piece by piece, with a few checkpoints along the way where you can pause and test yourself. 👇

📋 What You'll Learn

• What Computer Vision Actually Is

• Anatomy of a Digital Image

• Image Processing vs Computer Vision — What's the Difference?

• Where Computer Vision Gets Used

• Hands-On: 5 Basic Operations with OpenCV

• Bonus: Advanced Edge Detection (Sobel, Laplacian, Canny+Blur)

• From Manual Kernels to CNNs

• Wrapping It Into an API with FastAPI

• Bonus: Raw Image Endpoints & Real Deployment Proof

• Cheat Sheet & Mini Quiz

💡 If you're publishing this on Hashnode, turn on the "Table of Contents" toggle in the post settings — Hashnode auto-generates working jump-links from the headings in this article.

What Computer Vision Actually Is

The classic definition of AI: a system that can think and act like a human. Now, if that intelligence is pointed at language, that's NLP. If it's pointed at sight, that's Computer Vision.

💡 Computer Vision is the branch of AI that lets computers understand and interpret visual information from images or video — detecting objects, recognizing faces, even making decisions based on what it "sees."

But before a computer can "think" about an image, it needs a more basic foundation first: Image Processing. Think of it like learning a language — you can't write poetry before you know the alphabet. Image processing is that alphabet.

🧩 Quick check #1: If NLP is linguistic intelligence, what kind of intelligence is Computer Vision?

(try answering in your head before scrolling down 👇)

✅ Answer: Visual intelligence — the ability to understand information from images/video, not text.

Anatomy of a Digital Image

Here's the part people most often skip past: a digital image is a matrix of numbers.

Every image is made up of pixels — the smallest unit of an image — and each pixel holds a value representing light intensity at that point. The darker it is, the closer that value gets to 0. The brighter it is, the higher the value.

Grayscale vs RGB

Color is really just a combination of three numbers. For example:

• R=255, G=0, B=0 → bright red 🔴

• R=G=125, B=0 → yellowish 🟡

• R=0, G=0, B=0 → pure black ⚫

That's also why every color picker in design software shows three RGB sliders — those literally are the pixel numbers you're adjusting.

Resolution & Color Depth

A resolution of 1920 × 1080 means 1920 pixels wide and 1080 pixels tall. If it's a color image, the total number of values the computer has to store is:

1920 × 1080 × 3 channels = 6,220,800 numbers

...for just one image. That's why high-resolution images are "heavier" to process — more detail means more numbers to crunch.

Meanwhile, color depth determines how many possible color levels there are: 8-bit = 256 levels (0–255), 24-bit RGB = roughly 16 million possible color combinations.

🧮 Try it yourself: How many total numbers represent a 640×480 grayscale image?

(pause, do the math, then scroll)

✅ Answer: 640 × 480 × 1 channel = 307,200 numbers. Compare that to the color version, which needs 3× as many — 921,600 numbers! That's why converting to grayscale is a common trick for speeding up processing.

Image Dimensions: 2D vs 3D

One more characteristic that tends to get mentioned later: images also have a "dimensionality" to how they're represented.

• 2D images — grayscale, just a single layer of pixel matrix.

• 3D images — color images, where the RGB channels are stacked into one data structure (a typical shape looks like 224 × 224 × 3).

You'll run into this term a lot once you get into deep learning — especially when people talk about a model's "input shape."

Image Processing vs Computer Vision — What's the Difference?

These two terms often get used interchangeably, but they're actually different — even though they're closely related.

The simple version: image processing turns an image into another image; computer vision turns an image into understanding. And usually, computer vision needs image processing as a first step before it can "understand" what's in the image.

Image processing itself covers a lot of ground — filtering, smoothing, contrast enhancement, segmentation, color transformation — and it's used widely across fields: medicine, photography, surveillance, and as the foundation for nearly every AI application that deals with visual data.

Where Computer Vision Gets Used

This isn't just theory — computer vision is already deployed across plenty of industries, often without you noticing:

🎓 Education

• Exam proctoring — detecting cheating during online exams. The system watches student behavior via webcam and flags suspicious activity like talking to someone else, looking away repeatedly, or leaving the camera frame for too long.

• Handwriting recognition — automatically grading handwritten exam answers or assignments.

🏦 Banking / Administrative

• Signature and handwriting verification on documents, for authenticity checks.

🏥 Healthcare

• Disease detection in medical imaging — scanning MRIs, CT scans, or X-rays to automatically detect things like cancer, heart abnormalities, or tumors with high accuracy.

• Microscopic image analysis — examining cells or tissue samples to spot abnormalities such as infections or pathological changes.

🏭 Manufacturing

• Automated quality inspection (visual inspection) — detecting defects, cracks, or size mismatches on the production line (a classic example: checking phone screens for defects) with speed and precision well beyond the human eye.

🔒 Security / Surveillance

• Smart surveillance — CCTV systems (including in smart cities or mining sites) detecting suspicious activity or behavior, like someone entering a restricted area.

• PPE compliance checks — automatically recognizing whether workers are wearing helmets, masks, gloves, and other required gear.

📄 Others

• OCR for document processing, and face recognition, which is now standard in plenty of everyday apps.

Almost every one of these use cases starts from the same place: processing image pixels until the computer can "read" the pattern in them.

Hands-On: 5 Basic Operations with OpenCV

Enough theory — let's write some code. We'll use OpenCV, one of the most popular Python libraries for image processing, so we don't have to implement algorithms from scratch.

⚠️ Gotcha to remember: OpenCV reads images in BGR format, not RGB! Also, coordinate (0,0) is at the top-left corner, not the center like a regular Cartesian system.

1️⃣ Image Cropping

Cropping is really just NumPy array slicing — since the image OpenCV reads is literally a 2D array.

import cv2

image = cv2.imread('input_image.jpg')

# [y_start:y_end, x_start:x_end] — rows first, then columns
cropped_image = image[50:200, 100:300]

cv2.imwrite('cropped_image.jpg', cropped_image)

2️⃣ Grayscale

import cv2

image = cv2.imread('input_image.jpg')
grayscale_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.imwrite('grayscale_image.jpg', grayscale_image)

Real output from the code above — the original color image converted to black and white, keeping only light intensity.

3️⃣ Channel Split

Really handy once you get into semantic segmentation — each channel can represent a mask for a different object class.

import cv2

image = cv2.imread('input_image.jpg')
blue_channel, green_channel, red_channel = cv2.split(image)

cv2.imwrite('red_channel.jpg', red_channel)
cv2.imwrite('green_channel.jpg', green_channel)
cv2.imwrite('blue_channel.jpg', blue_channel)

4️⃣ Convolution — The Heart of Image Processing

This is the single most important concept in this whole article. Picture a kernel (a small matrix, say 5×5) sliding slowly across the image. At every position, the kernel's values get multiplied with the pixels underneath, then summed into one output value.

Different kernels = different effects:

• An averaging kernel → blur 🌫️

• Kernel [[0,-1,0],[-1,4,-1],[0,-1,0]] → edge detection ✏️

• Identity kernel → image stays unchanged

import cv2
import numpy as np

image = cv2.imread('input_image.jpg')

kernel = np.ones((5, 5), np.float32) / 25  # blur kernel
convolved_image = cv2.filter2D(image, -1, kernel)

cv2.imwrite('convolved_image.jpg', convolved_image)

🧪 Try it yourself: Swap the kernel above for the edge-detection kernel [[0,-1,0],[-1,4,-1],[0,-1,0]]. What happens?

✅ Answer: The result shows the edges of the objects in the photo, instead of a blurred whole image. That's because this kernel "highlights" the intensity differences between neighboring pixels — exactly where an object's edges are.

This isn't a simulation — it's the actual output of the edge-detection kernel above, applied to a Spider-Man image. Notice how the kernel "pops out" the lines of the costume.

5️⃣ Line Detection (Canny + Hough Transform)

import cv2
import numpy as np

image = cv2.imread('input_image.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

edges = cv2.Canny(gray, 50, 150, apertureSize=3)
lines = cv2.HoughLines(edges, 1, np.pi / 180, 200)

if lines is not None:
    for rho, theta in lines[:, 0]:
        a, b = np.cos(theta), np.sin(theta)
        x0, y0 = a * rho, b * rho
        x1, y1 = int(x0 + 1000 * (-b)), int(y0 + 1000 * (a))
        x2, y2 = int(x0 - 1000 * (-b)), int(y0 - 1000 * (a))
        cv2.line(image, (x1, y1), (x2, y2), (0, 0, 255), 2)

cv2.imwrite('line_detected_image.jpg', image)

The red lines are the output of the Hough Transform — detecting straight-line patterns from the edges Canny found earlier.

How it works: Canny finds the edges first, then Hough Transform looks for straight-line patterns among those edge points — kind of like how you'd connect dots into a line yourself.

💡 Practical note: if the detected lines look too short or odd on a large-resolution image, the line length (1000 in the code above) can be made dynamic based on the image dimensions: line_length = max(img_height, img_width). The Hough Transform threshold can also be lowered (say, from 200 to 80) if too few lines are being detected.

Bonus: Advanced Edge Detection — Sobel, Laplacian & Canny + Gaussian Blur

Canny isn't the only way to detect edges. A few other variations that are commonly used:

Canny, reading straight from grayscale

import cv2

img = cv2.imread('input_image.jpg', cv2.IMREAD_GRAYSCALE)
edges = cv2.Canny(img, 100, 200)
cv2.imwrite('canny_edge_detection.jpg', edges)

Sobel Operator — computes intensity gradients separately along the horizontal and vertical axes, then combines them into one magnitude.

sobelx = cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=3)
sobely = cv2.Sobel(img, cv2.CV_64F, 0, 1, ksize=3)

gradient_magnitude = cv2.magnitude(sobelx, sobely)
gradient_magnitude = cv2.convertScaleAbs(gradient_magnitude)
cv2.imwrite('sobel_edge_detection.jpg', gradient_magnitude)

Laplacian Operator — uses the second derivative, sensitive to intensity changes in every direction at once (not just horizontal/vertical like Sobel).

laplacian = cv2.Laplacian(img, cv2.CV_64F)
laplacian_abs = cv2.convertScaleAbs(laplacian)
cv2.imwrite('laplacian_detection.jpg', laplacian_abs)

Gaussian Blur + Canny — smooth the image first before running Canny, so it doesn't pick up false edges caused by noise.

blur = cv2.GaussianBlur(img, (5, 5), 1.4)
edges = cv2.Canny(blur, threshold1=100, threshold2=200)
cv2.imwrite('canny_edge_detection_blurred.jpg', edges)

From Manual Kernels to CNNs: The Leap That Changes Everything

Now here's the important question: how do we know what kernel values are "right" for a given task?

The answer: we don't have to define them manually. This is where Convolutional Neural Networks (CNNs) come in — instead of a kernel with fixed values, a CNN learns the best kernel values through training, adapting to whatever data it's given.

🚀 This is exactly why CNNs are so much more powerful than manual convolution: their kernels are adaptive, not hardcoded.

Architecturally, a CNN is still a regular neural network at its core — just with a series of convolutional layers added, whose job is to find the best possible values to fill those filters so that feature extraction from the image is maximized.

CNNs are a branch of deep learning built specifically for image data — a different "family" from RNNs, Transformers, or LSTMs, which are typically used for sequential data like text. If you're already familiar with those architectures from the NLP side, think of a CNN as their "cousin," purpose-built for visual data.

CNNs are a big topic on their own that we'll dig into more later — but you now know the foundation they're built on: convolution, which you just got hands-on with above.

Wrapping It Into an API with FastAPI

Finally, all five operations above can be turned into a service via an API. The idea: receive an image → process it in memory (no disk writes) → return the result as base64 — since an API endpoint can't directly return a raw image file.

Installation

pip install fastapi uvicorn pillow opencv-python

Setup & Helper Functions

from fastapi import FastAPI, UploadFile, File
from PIL import Image, ImageOps
import io, base64, cv2
import numpy as np

app = FastAPI()

def pil_image_to_base64(image: Image.Image):
    buffered = io.BytesIO()
    image.save(buffered, format="JPEG")
    return base64.b64encode(buffered.getvalue()).decode("utf-8")

def cv2_image_to_base64(image):
    _, buffer = cv2.imencode('.jpg', image)
    return base64.b64encode(buffer).decode("utf-8")

With those two helper functions, each operation just needs one endpoint:

/crop/ — crop the image to given coordinates

@app.post("/crop/")
async def crop_image(file: UploadFile = File(...), x: int = 0,
                      y: int = 0, width: int = 100, height: int = 100):
    image = Image.open(io.BytesIO(await file.read()))
    cropped_image = image.crop((x, y, x + width, y + height))
    return {"image_base64": pil_image_to_base64(cropped_image)}

/grayscale/ — convert to grayscale

@app.post("/grayscale/")
async def grayscale_image(file: UploadFile = File(...)):
    image = Image.open(io.BytesIO(await file.read()))
    grayscale_image = ImageOps.grayscale(image)
    return {"image_base64": pil_image_to_base64(grayscale_image)}

/channel_split/ — extract a single color channel

@app.post("/channel_split/")
async def channel_split(file: UploadFile = File(...), channel: str = "red"):
    image = cv2.imdecode(np.frombuffer(await file.read(), np.uint8),
                          cv2.IMREAD_COLOR)
    (blue, green, red) = cv2.split(image)
    if channel == "red":
        return {"image_base64": cv2_image_to_base64(red)}
    elif channel == "green":
        return {"image_base64": cv2_image_to_base64(green)}
    else:
        return {"image_base64": cv2_image_to_base64(blue)}

/convolution/ — apply a blur filter

@app.post("/convolution/")
async def convolution(file: UploadFile = File(...)):
    image = cv2.imdecode(np.frombuffer(await file.read(),
                          np.uint8), cv2.IMREAD_COLOR)
    kernel = np.ones((5, 5), np.float32) / 25
    convolved_image = cv2.filter2D(image, -1, kernel)
    return {"image_base64": cv2_image_to_base64(convolved_image)}

/line_detection/ — detect lines via Canny + Hough Transform

@app.post("/line_detection/")
async def line_detection(file: UploadFile = File(...)):
    image = cv2.imdecode(np.frombuffer(await file.read(),
                          np.uint8), cv2.IMREAD_COLOR)
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    edges = cv2.Canny(gray, 50, 150, apertureSize=3)
    lines = cv2.HoughLines(edges, 1, np.pi / 180, 200)

    if lines is not None:
        for rho, theta in lines[:, 0]:
            a, b = np.cos(theta), np.sin(theta)
            x0, y0 = a * rho, b * rho
            x1, y1 = int(x0 + 1000 * (-b)), int(y0 + 1000 * (a))
            x2, y2 = int(x0 - 1000 * (-b)), int(y0 - 1000 * (a))
            cv2.line(image, (x1, y1), (x2, y2), (0, 0, 255), 2)

    return {"image_base64": cv2_image_to_base64(image)}

Five endpoints, five image-processing operations you've learned from the ground up — and now ready to be called by anyone over HTTP. 🎉

Bonus: Raw Image Response Endpoints (Not Base64)

The five endpoints above return JSON containing base64 — a great format when the API is called from another application (web/mobile/backend), but not exactly pleasant to look at directly in a browser or Swagger UI, since it just shows up as a wall of text.

The fix is simple: build a version of each endpoint whose response is the raw image file itself (media_type="image/jpeg"), instead of JSON. Just add two helper functions to convert to raw bytes (not base64), then wrap the result in FastAPI's Response:

from fastapi.responses import Response

def pil_image_to_bytes(image: Image.Image) -> bytes:
    buffered = io.BytesIO()
    image.save(buffered, format="JPEG")
    return buffered.getvalue()

def cv2_image_to_bytes(image) -> bytes:
    _, buffer = cv2.imencode('.jpg', image)
    return buffer.tobytes()

Then build an "-image" version of each endpoint. Here's grayscale, for example:

@app.post("/grayscale-image/")
async def grayscale_image_raw(file: UploadFile = File(...)):
    image = Image.open(io.BytesIO(await file.read()))
    grayscale_image = ImageOps.grayscale(image)
    return Response(content=pil_image_to_bytes(grayscale_image), media_type="image/jpeg")

The same pattern repeats for /crop-image/, /channel_split-image/, /convolution-image/, and /line_detection-image/ — the logic is identical to the base64 versions, only the return line changes.

💡 Now your API has two "flavors" of endpoints: JSON+base64 for other programs to call, and raw image for quick testing or dropping straight into an <img src="..."> tag.

Real Proof: Deploying & Testing with ngrok

Every piece of code above has already been validated to work (tested through FastAPI's TestClient, with real requests hitting the actual endpoint functions, not just eyeballed). But beyond that — two of these endpoints (/grayscale/ and /grayscale-image/) were also tested through an actual live public deployment: the server ran on Google Colab, was exposed to the internet with ngrok, and was hit from the outside through the Swagger UI (/docs) that FastAPI generates automatically. Here's the proof:

The /grayscale-image/ endpoint tested with Spiderman.jpg, with the result rendered directly as an image in the response panel. Notice the response headers: content-type: image/jpeg, server: uvicorn, and ngrok-agent-ips — proof the request genuinely went through an ngrok tunnel from the internet, not a local simulation.

If you want to try deploying your own version from Google Colab (free), here's the gist:

1. Install nest-asyncio and pyngrok.

2. Sign up for a free ngrok account and grab your authtoken from the ngrok dashboard.

3. Run uvicorn inside an asyncio.create_task() (not a plain uvicorn.run() — Colab already has its own event loop, which will conflict if you call it directly).

4. ngrok will hand you a public URL (https://xxxx.ngrok-free.dev) that tunnels to localhost:8000 on your Colab instance.

(The finer details — including a few common gotchas like port conflicts and a SystemExit that can actually crash your Colab kernel if the old server isn't shut down properly — might get their own post, since there's too much to unpack here.)

Cheat Sheet & Mini Quiz

Before you close this tab, let's consolidate everything you've learned:

🎯 Quick Quiz — Test Your Understanding

Try answering these three questions in your head before checking the answer under each one.

1. Why does OpenCV sometimes make image colors look "off" when displayed directly with another library?

✅ Because OpenCV reads images in BGR format, not RGB. If you display it directly with a library that assumes RGB (like matplotlib), the red and blue channels get swapped. The fix: convert it first with cv2.COLOR_BGR2RGB.

2. What's the most fundamental difference between image processing and computer vision?

✅ Image processing: image in, image out (manipulated). Computer vision: image/video in, interpretation out (labels, coordinates, descriptions).

3. Why is a CNN considered "better" than convolution with a manual (predefined) kernel?

✅ Because a CNN's kernel values are learned automatically through training and become adaptive to the data — whereas a manual kernel is fixed and requires a lot of trial-and-error to fit a specific case.

📖 Glossary of Key Terms

Bookmark this section for a quick reference whenever you forget a term:

• Pixel — the smallest unit of a digital image; its value represents light intensity at that point.

• Channel — a single matrix layer within an image (e.g., R, G, or B); the number of channels determines an image's color dimensionality.

• Resolution — the number of pixels wide × tall in an image.

• Color Depth — the number of bits used to represent each pixel's value (determines the number of possible levels/colors).

• Kernel / Filter — a small matrix slid across an image to perform a convolution operation.

• Convolution — a multiply-and-sum operation between a kernel and the overlapping image area, slid across the whole image.

• Edge Detection — a technique for detecting the edges/boundaries of objects in an image.

• Hough Transform — a method for extracting lines from edge-detection output, based on connectivity patterns between edge points.

• Semantic Segmentation — a computer vision technique for assigning a class label to every pixel in an image.

• CNN (Convolutional Neural Network) — a deep learning architecture for image data, where kernel/filter values are learned automatically through training rather than defined manually.

• Base64 — an encoding format for representing binary data (like an image) as text, commonly used when sending images through an API/JSON response.

• ngrok — a tunneling service that makes a local server (e.g., on Colab) accessible via a temporary public URL, without needing an actual cloud deployment.

Wrapping Up

Computer Vision looks intimidating from the outside, but once you've got the fundamentals down — pixels, matrices, and convolution — every advanced topic (CNNs, object detection, segmentation) turns out to be built on that same foundation. These same concepts will keep showing up all the way through to modern deep learning.

If you try any of the code above, share your results in the comments — I'd love to see what kernels you experimented with! 👇

Full source code (notebook, README, all verified working) is open on GitHub: github.com/arielshakaramiro/computer-vision-image-processing

This article is part of a learning-notes series on Computer Vision & Image Processing. Follow along for the next post in the series on Convolutional Neural Networks (CNNs).

Image credit: The Spider-Man illustration used as the example image throughout this article was sourced from Pinterest, used purely for demonstrating image-processing techniques. Spider-Man is a trademark/copyright of Marvel/Sony.

Tags: computer-vision opencv python machine-learning fastapi image-processing ai deep-learning

🧠 TL;DR — Gambar digital cuma kumpulan angka dalam bentuk matriks. Begitu kamu paham itu, semua konsep di Computer Vision — dari grayscale, convolution, sampai CNN — jadi jauh lebih masuk akal. Di post ini kita bongkar konsepnya pelan-pelan, sambil coding bareng pakai OpenCV, terus kita bungkus jadi API pakai FastAPI.

Pernah kepikiran nggak, gimana caranya HP kamu bisa tahu ada wajah di foto? Atau gimana caranya CCTV bisa "curiga" kalau ada orang masuk area terlarang? Semua itu berawal dari satu ide sederhana yang sering diremehkan orang: komputer sebenarnya nggak "melihat" gambar seperti kita — dia cuma melihat angka.

Begitu kamu paham cara komputer "melihat" angka ini, pintu masuk ke dunia Computer Vision jadi jauh lebih terang. Yuk kita bongkar satu-satu, plus ada beberapa titik di mana kamu bisa berhenti sejenak buat nge-test pemahaman kamu sendiri. 👇

📋 Yang Akan Kamu Pelajari

• Apa Itu Computer Vision, Sebenarnya?

• Anatomi Sebuah Gambar Digital

• Image Processing vs Computer Vision — Bedanya Apa?

• Di Mana Computer Vision Dipakai?

• Praktik: 5 Operasi Dasar dengan OpenCV

• Bonus: Edge Detection Lanjutan (Sobel, Laplacian, Canny+Blur)

• Dari Kernel Manual ke CNN

• Bungkus Jadi API dengan FastAPI

• Bonus: Endpoint Versi Gambar Langsung & Bukti Deploy Nyata

• Cheat Sheet & Kuis Mini

💡 Kalau kamu publish di Hashnode, aktifkan toggle "Table of Contents" di pengaturan artikel (bagian Post Settings saat publish) — Hashnode otomatis bikin navigasi loncat-ke-section yang beneran jalan, dari heading yang ada di artikel ini.

Apa Itu Computer Vision, Sebenarnya?

Definisi klasik AI: sistem yang bisa berpikir dan bertindak layaknya manusia. Nah, kalau kecerdasan itu diarahkan ke bahasa, itu namanya NLP. Kalau diarahkan ke penglihatan, itu namanya Computer Vision.

💡 Computer Vision adalah cabang AI yang membuat komputer bisa memahami dan menafsirkan informasi visual dari gambar atau video — mendeteksi objek, mengenali wajah, bahkan mengambil keputusan dari apa yang "dilihatnya".

Tapi sebelum komputer bisa "mikir" soal gambar, dia butuh fondasi yang lebih dasar dulu: Image Processing. Ibarat belajar bahasa, kamu nggak bisa langsung nulis puisi sebelum kenal huruf. Image processing itu "huruf"-nya.

🧩 Quick check #1: Kalau NLP itu kecerdasan linguistik, Computer Vision itu kecerdasan apa?

(coba jawab dulu di kepala kamu sebelum baca baris berikutnya 👇)

✅ Jawaban: Kecerdasan visual — kemampuan memahami informasi dari gambar/video, bukan teks.

Anatomi Sebuah Gambar Digital

Ini bagian paling penting yang sering dilewatkan orang: gambar digital itu matriks angka.

Setiap gambar tersusun dari piksel — satuan terkecil sebuah gambar — dan tiap piksel punya nilai yang menyatakan intensitas cahaya di titik itu. Makin gelap, nilainya makin mendekati 0. Makin terang, nilainya makin tinggi.

Grayscale vs RGB

Warna itu sebenarnya cuma hasil kombinasi tiga angka. Contoh:

• R=255, G=0, B=0 → merah menyala 🔴

• R=G=125, B=0 → kekuningan 🟡

• R=0, G=0, B=0 → hitam pekat ⚫

Ini juga alasan kenapa color picker di software desain selalu nunjukkin tiga slider RGB — itu literally angka-angka piksel yang lagi kamu atur.

Resolusi & Color Depth

Resolusi 1920 × 1080 artinya ada 1920 piksel di lebar dan 1080 di tinggi. Kalau gambarnya berwarna, total angka yang harus disimpan komputer adalah:

1920 × 1080 × 3 channel = 6.220.800 angka

...hanya untuk satu gambar. Ini kenapa gambar resolusi tinggi lebih "berat" diproses — makin detail, makin banyak angka yang harus dihitung.

Sementara color depth menentukan berapa banyak level warna yang mungkin: 8-bit = 256 level (0–255), 24-bit RGB = sekitar 16 juta warna kombinasi.

🧮 Coba hitung sendiri: Berapa total angka yang direpresentasikan gambar grayscale 640×480?

(pause dulu, hitung manual, baru scroll)

✅ Jawaban: 640 × 480 × 1 channel = 307.200 angka. Bandingkan dengan versi berwarnanya yang butuh 3× lebih banyak — 921.600 angka! Itu sebabnya konversi ke grayscale sering dipakai buat mempercepat pemrosesan.

Dimensi Gambar: 2D vs 3D

Satu karakteristik lagi yang sering disebut belakangan: gambar juga punya "dimensi" representasi.

• Gambar 2D — gray scale, cuma satu lapisan matriks piksel.

• Gambar 3D — gambar berwarna, di mana channel-channel RGB ditumpuk jadi satu struktur data (contoh bentuknya: array 224 × 224 × 3).

Istilah ini bakal sering muncul lagi pas kamu belajar deep learning — terutama waktu ngomongin "input shape" di sebuah model.

Image Processing vs Computer Vision — Bedanya Apa?

Dua istilah ini sering dianggap sama padahal beda — meski saling terkait erat.

Gampangnya: image processing mengubah gambar jadi gambar lain, computer vision mengubah gambar jadi pemahaman. Dan biasanya, computer vision butuh image processing sebagai langkah awal sebelum bisa "mengerti" apa yang ada di gambar.

Operasi image processing sendiri macam-macam bentuknya — filtering, penghalusan gambar, peningkatan kontras, segmentasi, sampai transformasi warna — dan dipakai luas di banyak bidang: dunia medis, fotografi, pengawasan (surveillance), sampai jadi fondasi hampir semua aplikasi AI yang berurusan sama visual.

Di Mana Computer Vision Dipakai?

Ini bukan teori kosong — computer vision udah dipakai di banyak industri, sering tanpa kamu sadari:

🎓 Education

• Pengawasan ujian — mendeteksi kecurangan lewat webcam: siswa ngobrol sama orang lain, sering menoleh, atau keluar dari frame kamera dalam waktu lama otomatis kena flag sebagai peringatan.

• Pengenalan tulisan tangan — menilai jawaban ujian atau tugas tulisan tangan siswa secara otomatis.

🏦 Banking / Administratif

• Verifikasi tanda tangan dan tulisan tangan pada dokumen untuk keperluan validasi keaslian.

🏥 Healthcare

• Deteksi penyakit pada citra medis — memindai MRI, CT scan, atau X-ray untuk mendeteksi kanker, kelainan jantung, atau tumor secara otomatis dengan akurasi tinggi.

• Analisis citra mikroskopis — memeriksa sel/jaringan untuk menemukan abnormalitas seperti infeksi atau perubahan patologis.

🏭 Manufaktur

• Inspeksi kualitas otomatis (visual inspection) — mendeteksi cacat, retak, atau ketidaksesuaian ukuran di lini produksi (contoh klasik: cek cacat pada layar HP) dengan kecepatan dan ketelitian yang jauh di atas mata manusia.

🔒 Security / Surveillance

• Pengawasan keamanan — sistem CCTV pintar (termasuk di kota pintar atau area tambang) mendeteksi aktivitas atau perilaku mencurigakan, misalnya orang yang masuk ke area terlarang.

• Pengecekan kelengkapan APD pekerja — otomatis mengenali apakah pekerja sudah pakai helm, masker, sarung tangan, dan perlengkapan wajib lainnya.

📄 Lainnya

• OCR untuk pemrosesan dokumen, dan face recognition yang sekarang udah jadi standar di banyak aplikasi sehari-hari.

Hampir semua use case di atas dimulai dari hal yang sama: memproses piksel gambar sampai komputer bisa "membaca" polanya.

Praktik: 5 Operasi Dasar dengan OpenCV

Cukup teori — sekarang kita coding. Kita akan pakai OpenCV, salah satu library Python paling populer untuk image processing, biar nggak perlu implementasi algoritma dari nol.

⚠️ Gotcha yang wajib diingat: OpenCV membaca gambar dalam format BGR, bukan RGB! Titik (0,0) juga ada di pojok kiri atas, bukan di tengah seperti koordinat kartesius biasa.

1️⃣ Image Cropping

Cropping itu sebenarnya cuma slicing NumPy array — karena gambar yang dibaca OpenCV memang berbentuk array 2D.

import cv2

image = cv2.imread('input_image.jpg')

# [y_start:y_end, x_start:x_end] — baris dulu, baru kolom
cropped_image = image[50:200, 100:300]

cv2.imwrite('cropped_image.jpg', cropped_image)

2️⃣ Grayscale

import cv2

image = cv2.imread('input_image.jpg')
grayscale_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.imwrite('grayscale_image.jpg', grayscale_image)

Hasil nyata dari kode di atas — asli berwarna diubah jadi hitam-putih, cuma nyimpen intensitas cahaya.

3️⃣ Channel Split

Berguna banget saat masuk ke semantic segmentation — tiap channel bisa merepresentasikan mask kelas objek yang berbeda.

import cv2

image = cv2.imread('input_image.jpg')
blue_channel, green_channel, red_channel = cv2.split(image)

cv2.imwrite('red_channel.jpg', red_channel)
cv2.imwrite('green_channel.jpg', green_channel)
cv2.imwrite('blue_channel.jpg', blue_channel)

4️⃣ Convolution — Jantungnya Image Processing

Ini konsep paling penting di seluruh artikel ini. Bayangkan sebuah kernel (matriks kecil, misal 5×5) yang digeser pelan-pelan di atas gambar. Di setiap posisi, nilai kernel dikalikan dengan piksel di bawahnya, lalu dijumlahkan jadi satu nilai output.

Kernel yang berbeda = efek yang berbeda:

• Kernel isi rata-rata → blur 🌫️

• Kernel [[0,-1,0],[-1,4,-1],[0,-1,0]] → edge detection ✏️

• Kernel identitas → gambar nggak berubah

import cv2
import numpy as np

image = cv2.imread('input_image.jpg')

kernel = np.ones((5, 5), np.float32) / 25  # kernel blur
convolved_image = cv2.filter2D(image, -1, kernel)

cv2.imwrite('convolved_image.jpg', convolved_image)

🧪 Coba sendiri: Ganti kernel di atas dengan kernel edge detection [[0,-1,0],[-1,4,-1],[0,-1,0]]. Apa yang terjadi?

✅ Jawaban: Gambar hasilnya akan menampilkan garis-garis tepi (edge) dari objek di foto, bukan lagi gambar utuh yang blur. Ini karena kernel tersebut "menonjolkan" perbedaan intensitas antar piksel bertetangga — persis di mana tepi objek berada.

Ini bukan simulasi — ini hasil beneran dari kernel edge detection di atas, diterapkan ke gambar Spider-Man. Lihat gimana kernelnya "menonjolkan" garis-garis kostumnya.

5️⃣ Line Detection (Canny + Hough Transform)

import cv2
import numpy as np

image = cv2.imread('input_image.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

edges = cv2.Canny(gray, 50, 150, apertureSize=3)
lines = cv2.HoughLines(edges, 1, np.pi / 180, 200)

if lines is not None:
    for rho, theta in lines[:, 0]:
        a, b = np.cos(theta), np.sin(theta)
        x0, y0 = a * rho, b * rho
        x1, y1 = int(x0 + 1000 * (-b)), int(y0 + 1000 * (a))
        x2, y2 = int(x0 - 1000 * (-b)), int(y0 - 1000 * (a))
        cv2.line(image, (x1, y1), (x2, y2), (0, 0, 255), 2)

cv2.imwrite('line_detected_image.jpg', image)

Garis-garis merah adalah hasil Hough Transform — mendeteksi pola garis lurus dari tepi-tepi yang ditemukan Canny sebelumnya.

Cara kerjanya: Canny dulu cari tepi-tepinya, lalu Hough Transform mencari pola garis dari titik-titik tepi tersebut — mirip cara kamu menghubungkan titik-titik jadi garis lurus.

💡 Catatan praktik: kalau garis hasil deteksi kelihatan terlalu pendek/aneh di gambar resolusi besar, panjang garis (1000 di kode atas) bisa dibuat dinamis mengikuti dimensi gambar: line_length = max(img_height, img_width). Threshold Hough Transform juga bisa diturunkan (mis. dari 200 ke 80) kalau garis yang kedeteksi kurang banyak.

Bonus: Edge Detection Lanjutan — Sobel, Laplacian & Canny + Gaussian Blur

Canny bukan satu-satunya cara deteksi tepi. Beberapa variasi lain yang sering dipakai:

Canny langsung dari grayscale

import cv2

img = cv2.imread('input_image.jpg', cv2.IMREAD_GRAYSCALE)
edges = cv2.Canny(img, 100, 200)
cv2.imwrite('canny_edge_detection.jpg', edges)

Sobel Operator — gradien intensitas dihitung terpisah di sumbu horizontal & vertikal, lalu digabung jadi satu magnitude.

sobelx = cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=3)
sobely = cv2.Sobel(img, cv2.CV_64F, 0, 1, ksize=3)

gradient_magnitude = cv2.magnitude(sobelx, sobely)
gradient_magnitude = cv2.convertScaleAbs(gradient_magnitude)
cv2.imwrite('sobel_edge_detection.jpg', gradient_magnitude)

Laplacian Operator — pakai turunan kedua, sensitif ke perubahan intensitas di segala arah sekaligus (bukan cuma horizontal/vertikal kayak Sobel).

laplacian = cv2.Laplacian(img, cv2.CV_64F)
laplacian_abs = cv2.convertScaleAbs(laplacian)
cv2.imwrite('laplacian_detection.jpg', laplacian_abs)

Gaussian Blur + Canny — smoothing dulu sebelum Canny, biar nggak nangkep tepi-tepi palsu akibat noise.

blur = cv2.GaussianBlur(img, (5, 5), 1.4)
edges = cv2.Canny(blur, threshold1=100, threshold2=200)
cv2.imwrite('canny_edge_detection_blurred.jpg', edges)

Dari Kernel Manual ke CNN: Lompatan yang Mengubah Segalanya

Sekarang pertanyaan pentingnya: gimana kita tahu nilai kernel yang "benar" untuk tugas tertentu?

Jawabannya: kita nggak perlu menentukannya secara manual. Di sinilah Convolutional Neural Network (CNN) masuk — alih-alih kernel yang nilainya tetap (fixed), CNN mempelajari nilai kernel terbaik lewat proses training, menyesuaikan dengan data yang diberikan.

🚀 Inilah kenapa CNN jauh lebih powerful dibanding operasi convolution manual: kernelnya adaptif, bukan hardcoded.

Dari sisi arsitektur, basic-nya CNN tetap neural network biasa — cuma ditambah serangkaian convolutional layer yang tugasnya nyari nilai-nilai terbaik buat mengisi filter, supaya ekstraksi fitur dari gambar jadi maksimal.

CNN sendiri adalah turunan dari deep learning yang khusus dipakai untuk data gambar — beda "keluarga" dengan RNN, Transformer, atau LSTM yang biasa dipakai buat data sekuensial seperti teks. Kalau kamu sudah kenal arsitektur-arsitektur itu dari sisi NLP, CNN adalah versi "sepupunya" yang didesain khusus buat data visual.

CNN adalah topik besar tersendiri yang akan kita bahas lebih dalam nanti — tapi sekarang kamu udah tahu fondasinya: convolution, yang barusan kita praktikkan sendiri di atas.

Bungkus Jadi API dengan FastAPI

Terakhir, kelima operasi di atas bisa langsung "dijual" sebagai layanan lewat API. Idenya: terima gambar → proses di memori (tanpa simpan ke disk) → kembalikan hasilnya dalam format base64 — karena endpoint API nggak bisa langsung ngembaliin file gambar mentah.

Instalasi

pip install fastapi uvicorn pillow opencv-python

Setup & Helper Function

from fastapi import FastAPI, UploadFile, File
from PIL import Image, ImageOps
import io, base64, cv2
import numpy as np

app = FastAPI()

def pil_image_to_base64(image: Image.Image):
    buffered = io.BytesIO()
    image.save(buffered, format="JPEG")
    return base64.b64encode(buffered.getvalue()).decode("utf-8")

def cv2_image_to_base64(image):
    _, buffer = cv2.imencode('.jpg', image)
    return base64.b64encode(buffer).decode("utf-8")

Dari dua helper function itu, tinggal bikin satu endpoint per operasi:

/crop/ — potong gambar sesuai koordinat

@app.post("/crop/")
async def crop_image(file: UploadFile = File(...), x: int = 0,
                      y: int = 0, width: int = 100, height: int = 100):
    image = Image.open(io.BytesIO(await file.read()))
    cropped_image = image.crop((x, y, x + width, y + height))
    return {"image_base64": pil_image_to_base64(cropped_image)}

/grayscale/ — konversi ke grayscale

@app.post("/grayscale/")
async def grayscale_image(file: UploadFile = File(...)):
    image = Image.open(io.BytesIO(await file.read()))
    grayscale_image = ImageOps.grayscale(image)
    return {"image_base64": pil_image_to_base64(grayscale_image)}

/channel_split/ — ambil salah satu channel warna

@app.post("/channel_split/")
async def channel_split(file: UploadFile = File(...), channel: str = "red"):
    image = cv2.imdecode(np.frombuffer(await file.read(), np.uint8),
                          cv2.IMREAD_COLOR)
    (blue, green, red) = cv2.split(image)
    if channel == "red":
        return {"image_base64": cv2_image_to_base64(red)}
    elif channel == "green":
        return {"image_base64": cv2_image_to_base64(green)}
    else:
        return {"image_base64": cv2_image_to_base64(blue)}

/convolution/ — terapkan filter blur

@app.post("/convolution/")
async def convolution(file: UploadFile = File(...)):
    image = cv2.imdecode(np.frombuffer(await file.read(),
                          np.uint8), cv2.IMREAD_COLOR)
    kernel = np.ones((5, 5), np.float32) / 25
    convolved_image = cv2.filter2D(image, -1, kernel)
    return {"image_base64": cv2_image_to_base64(convolved_image)}

/line_detection/ — deteksi garis via Canny + Hough Transform

@app.post("/line_detection/")
async def line_detection(file: UploadFile = File(...)):
    image = cv2.imdecode(np.frombuffer(await file.read(),
                          np.uint8), cv2.IMREAD_COLOR)
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    edges = cv2.Canny(gray, 50, 150, apertureSize=3)
    lines = cv2.HoughLines(edges, 1, np.pi / 180, 200)

    if lines is not None:
        for rho, theta in lines[:, 0]:
            a, b = np.cos(theta), np.sin(theta)
            x0, y0 = a * rho, b * rho
            x1, y1 = int(x0 + 1000 * (-b)), int(y0 + 1000 * (a))
            x2, y2 = int(x0 - 1000 * (-b)), int(y0 - 1000 * (a))
            cv2.line(image, (x1, y1), (x2, y2), (0, 0, 255), 2)

    return {"image_base64": cv2_image_to_base64(image)}

Lima endpoint, lima operasi image processing yang udah kamu pelajari dari awal — dan sekarang siap dipanggil orang lain lewat HTTP request. 🎉

Bonus: Endpoint Versi "Gambar Langsung" (Bukan Base64)

Kelima endpoint di atas balikin JSON berisi base64 — format ini paling cocok kalau API-nya dipanggil dari aplikasi lain (web/mobile/backend), tapi kurang enak dilihat langsung di browser atau Swagger UI, karena cuma nongol sebagai teks panjang.

Solusinya gampang: bikin versi endpoint yang response-nya langsung file gambar (media_type="image/jpeg"), bukan JSON. Cukup tambah dua helper function buat convert ke bytes mentah (bukan base64), lalu bungkus pakai Response dari FastAPI:

from fastapi.responses import Response

def pil_image_to_bytes(image: Image.Image) -> bytes:
    buffered = io.BytesIO()
    image.save(buffered, format="JPEG")
    return buffered.getvalue()

def cv2_image_to_bytes(image) -> bytes:
    _, buffer = cv2.imencode('.jpg', image)
    return buffer.tobytes()

Lalu bikin versi "-image" dari tiap endpoint. Contoh buat grayscale:

@app.post("/grayscale-image/")
async def grayscale_image_raw(file: UploadFile = File(...)):
    image = Image.open(io.BytesIO(await file.read()))
    grayscale_image = ImageOps.grayscale(image)
    return Response(content=pil_image_to_bytes(grayscale_image), media_type="image/jpeg")

Pola yang sama tinggal diulang buat /crop-image/, /channel_split-image/, /convolution-image/, dan /line_detection-image/ — logikanya identik dengan versi base64, cuma baris return-nya yang beda.

💡 Sekarang API kamu punya dua "rasa" endpoint: yang JSON+base64 buat dipanggil program lain, dan yang gambar langsung buat testing cepat atau dipakai langsung di <img src="...">.

Bukti Nyata: Deploy & Testing Pakai ngrok

Semua kode di atas sudah divalidasi jalan (dites lewat TestClient FastAPI, request beneran sampai ke fungsi endpoint-nya, bukan cuma dibaca sekilas). Tapi lebih dari itu — dua endpoint di antaranya (/grayscale/ dan /grayscale-image/) sudah dites langsung lewat deployment publik sungguhan: server jalan di Google Colab, di-expose ke internet pakai ngrok, dan diakses dari luar lewat Swagger UI (/docs) yang otomatis dibikin FastAPI. Ini buktinya:

Endpoint /grayscale-image/ dites pakai Spiderman.jpg, hasilnya langsung dirender sebagai gambar di response panel. Perhatikan response header: content-type: image/jpeg, server: uvicorn, dan ngrok-agent-ips — bukti request-nya beneran lewat tunnel ngrok dari internet, bukan simulasi lokal.

Kalau kamu mau coba deploy versi kamu sendiri dari Google Colab (gratis), garis besarnya:

1. Install nest-asyncio dan pyngrok.

2. Daftar akun ngrok gratis, ambil authtoken dari dashboard ngrok.

3. Jalankan uvicorn di dalam asyncio.create_task() (bukan uvicorn.run() biasa — Colab udah punya event loop sendiri yang bakal bentrok kalau dipanggil langsung).

4. ngrok bakal kasih URL publik (https://xxxx.ngrok-free.dev) yang nge-tunnel ke localhost:8000 di Colab kamu.

(Detail lengkapnya — termasuk beberapa jebakan umum kayak port conflict dan SystemExit yang bisa bikin kernel Colab crash kalau server lama nggak dimatikan dengan benar — mungkin saya bahas terpisah di post lain, karena cukup panjang buat dijelasin di sini.)

Cheat Sheet & Kuis Mini

Sebelum kamu tutup tab ini, yuk konsolidasi semua yang udah dipelajari:

🎯 Kuis Cepat — Tes Pemahaman Kamu

Coba jawab dulu ketiga pertanyaan ini di kepala kamu sebelum baca jawabannya di bawah masing-masing soal.

1. Kenapa OpenCV kadang bikin warna gambar "aneh" kalau langsung ditampilkan pakai library lain?

✅ Karena OpenCV membaca gambar dalam format BGR, bukan RGB. Kalau kamu tampilkan langsung pakai library yang mengasumsikan RGB (misalnya matplotlib), channel merah dan birunya akan tertukar. Solusinya: konversi dulu pakai cv2.COLOR_BGR2RGB.

2. Apa perbedaan paling mendasar antara image processing dan computer vision?

✅ Image processing: input gambar, output gambar (yang sudah dimanipulasi). Computer vision: input gambar/video, output interpretasi (label, koordinat, deskripsi).

3. Kenapa CNN dianggap "lebih baik" daripada convolution dengan kernel manual (predefined)?

✅ Karena nilai kernel di CNN dipelajari otomatis lewat training dan jadi adaptif terhadap data — sementara kernel manual bersifat tetap (fixed) dan butuh banyak trial-and-error untuk cocok dengan kasus tertentu.

📖 Glosarium Istilah Penting

Simpan bagian ini buat referensi cepat kalau nanti lupa istilah:

• Piksel — satuan terkecil dari gambar digital; nilainya menyatakan intensitas cahaya pada titik tersebut.

• Channel — satu lapisan matriks pada gambar (mis. R, G, atau B); jumlah channel menentukan dimensi warna gambar.

• Resolusi — jumlah piksel lebar × tinggi pada suatu gambar.

• Color Depth — jumlah bit yang dipakai untuk merepresentasikan nilai tiap piksel (menentukan jumlah level/warna yang mungkin).

• Kernel / Filter — matriks kecil yang digeser di atas gambar untuk melakukan operasi convolution.

• Convolution — operasi perkalian & penjumlahan antara kernel dan area gambar yang bertumpukan, digeser ke seluruh gambar.

• Edge Detection — teknik mendeteksi tepi/garis batas objek pada gambar.

• Hough Transform — metode untuk mengekstrak garis dari hasil edge detection berdasarkan pola keterhubungan titik-titik tepi.

• Semantic Segmentation — teknik computer vision untuk memberi label kelas pada setiap piksel gambar.

• CNN (Convolutional Neural Network) — arsitektur deep learning untuk data gambar, di mana nilai kernel/filter dipelajari otomatis lewat training, bukan ditentukan manual.

• Base64 — format encoding untuk merepresentasikan data biner (seperti gambar) sebagai teks, umum dipakai saat mengirim gambar lewat response API/JSON.

• ngrok — layanan tunneling yang bikin server lokal (mis. di Colab) bisa diakses lewat URL publik sementara, tanpa perlu deploy ke cloud beneran.

Penutup

Computer Vision kelihatannya rumit dari luar, tapi begitu kamu pegang fondasinya — piksel, matriks, dan convolution — semua topik lanjutan (CNN, object detection, segmentation) jadi bangunan yang berdiri di atas dasar yang sama. Konsep yang sama ini juga yang bakal kamu pakai terus sampai ke deep learning modern sekalipun.

Kalau kamu coba salah satu kode di atas, share hasilnya di kolom komentar ya — penasaran lihat kernel eksperimen kamu! 👇

Kode lengkapnya (notebook, README, semua sudah dites jalan) tersedia terbuka di GitHub: github.com/arielshakaramiro/computer-vision-image-processing

Artikel ini bagian dari seri catatan belajar Computer Vision & Image Processing. Follow untuk nggak ketinggalan seri lanjutannya soal Convolutional Neural Network (CNN).

Sumber gambar: Ilustrasi Spider-Man yang dipakai sebagai contoh gambar di sepanjang artikel ini diambil dari Pinterest, dipakai murni untuk keperluan demonstrasi teknik image processing. Karakter Spider-Man adalah hak cipta Marvel/Sony.

Tags: computer-vision opencv python machine-learning fastapi image-processing ai deep-learning

The full project (3 notebooks + results documentation) is on GitHub.

Table of Contents

• Why Run Both Approaches?

• Guess First

• Case Study 1: Code Generation From Scratch

• Shipping It as an API

• Case Study 2: Fine-Tuning an Indonesian GPT-2

• Plot Twist: Two Numbers, Two Different Stories

• Side-by-Side Comparison

• Quick Quiz

• Summary

Why Run Both Approaches?

Training from scratch means the model starts from random weights — it knows nothing about language or code yet. It needs a lot of data and a lot of training steps before it produces anything coherent.

Fine-tuning means starting from a model that's already been pretrained, then adapting it to a new task or domain with a much smaller dataset.

These two approaches carry genuinely different trade-offs, and the clearest way to understand them isn't through theory — it's running both yourself and looking closely at what actually comes out.

Guess First

Before reading further, take a guess:

If a GPT-2 model is trained from scratch for 196 steps, and a pretrained GPT-2 model is fine-tuned for 250 steps on a tiny dataset (just 50 sentences) — which one do you think produces better text? Does a lower loss always mean a better model, or is there a catch?

Hold onto your answer. We'll circle back in the Plot Twist section.

Case Study 1: Code Generation From Scratch

The first notebook trains a GPT-2 architecture (124M parameters) entirely from random weights to complete Python code snippets, using a filtered subset of the CodeParrot dataset (data-science-related code: numpy, pandas, sklearn, and similar).

The pipeline:

1. Load the pre-filtered dataset (huggingface-course/codeparrot-ds-train) — 50,000 samples

2. Tokenize with a code-specific tokenizer

3. Build an empty GPT-2 config and initialize the model with random weights

4. Train for 1 epoch (196 steps)

5. Test by asking the model to complete code from a comment

Final training loss: 6.43. When asked to complete this prompt:

# create some data
x = np.random.randn(100)
y = np.random.randn(100)

# create dataframe from x and y

instead of continuing with pandas code, the model loops on open-source license boilerplate:

# the of the data
# with the terms of the
# This is the terms of the terms of the Free Software Foundation...
# but WITHOUT ANY WARRANTIES OR CONDITIONS OF ANY WITHOUT ANY WARRANTY...

At first glance this looks like a failure. Look closer and it actually makes sense: a huge number of GitHub Python files open with a long, repeated license comment block. For a model starting from random weights, that pattern is simply easier to pick up early than the deeper structure of real Python code.

Shipping It as an API

The model from notebook 1 gets wrapped in a FastAPI endpoint and exposed through ngrok. What's interesting is that the exact same model produces noticeably different output once sampling parameters (top_k=50, top_p=0.9, temperature=0.8) are turned on — instead of looping on license text, it starts producing more code-like tokens (import, def __init__, class, assert), even though the result still isn't valid Python.

In other words, decoding strategy visibly shapes how "good" an output looks, even when the underlying model's actual competence hasn't changed at all.

Case Study 2: Fine-Tuning an Indonesian GPT-2

The third notebook runs in two parts. The first part is pure exploration of existing Indonesian models — fill-mask with Indonesian BERT and RoBERTa, then text generation with a pretrained Indonesian GPT-2 (no fine-tuning yet). Both fill-mask models confidently predict the preposition "di" for "Ibu ku sedang bekerja [MASK] supermarket" — BERT at 94.96% confidence, RoBERTa at 64.92%. A simple sanity check confirming these models behave as expected out of the box.

The second part fine-tunes cahya/gpt2-small-indonesian-522M on a dummy dataset of 50 sentences about dinosaur facts, written in several different styles (encyclopedic, casual slang, children's storytelling). After 10 epochs, the loss drops to 1.27 — far lower than notebook 1's.

Prompted with "Trex merupakan" ("T-Rex is"), it generates:

Trex merupakan dinosaurus karnivora pertama yang diketahui memiliki adaptasi unik untuk hidup dan berburu di dalam air. Bukti yang dapat diandalkan untuk hidup dan berburu di dalam air menunjukkan bahwa T-Rex memiliki adaptasi unik untuk hidup dan berburu di dalam air...

On a first read, this sounds genuinely fluent. Which is exactly why it's worth a closer look.

Plot Twist: Two Numbers, Two Different Stories

Out of curiosity, I compared that generated sentence against the training data itself, and found this:

Training sample (about Spinosaurus): "Spinosaurus adalah dinosaurus karnivora pertama yang diketahui memiliki adaptasi unik untuk hidup dan berburu di dalam air."

Generated output (about T-Rex): "Trex merupakan dinosaurus karnivora pertama yang diketahui memiliki adaptasi unik untuk hidup dan berburu di dalam air."

Nearly identical, with just the subject swapped. With only 50 training examples repeated over 10 epochs fine-tuning an already-pretrained model, this is likely a textbook sign of overfitting: the model is stitching together memorized fragments rather than generating genuinely new text.

One methodological detail makes this even clearer: the notebook uses the same dataset for both training and evaluation (eval_dataset=tokenized_dataset, identical to train_dataset). That means the reported eval loss can't actually detect overfitting at all — a proper held-out validation split would be needed to measure that honestly.

So a dramatically lower loss and more fluent-sounding output in notebook 3 doesn't necessarily mean a better model; it could just as easily be a sign of memorizing a tiny dataset. Meanwhile, the higher loss and messier output in notebook 1 is simply what from-scratch training looks like on a limited compute budget, rather than evidence that the pipeline failed.

Side-by-Side Comparison

Quick Quiz

Q1: Does a lower loss always mean a better model? Not necessarily. A low loss on a very small dataset (like the 50 sentences in notebook 3) can just as easily mean the model memorized the data rather than learning general language patterns.

Q2: What happens when eval_dataset equals train_dataset? The evaluation metric stops being informative for detecting overfitting, since the model is being evaluated on the exact same data it was trained on.

Q3: Why do notebook 1 and the API in notebook 2 produce different-looking output from the same model? Because they use different decoding strategies — notebook 1 uses default/deterministic decoding, while the notebook 2 API applies top_k/top_p/temperature, which changes how the next token gets sampled.

Summary

• [ ] Training from scratch needs far more data and steps than used here to produce coherent output

• [ ] A low loss on a small dataset can signal overfitting, not success

• [ ] Always use a separate validation split if you actually want to measure generalization

• [ ] Decoding strategy (top_k, top_p, temperature) can change perceived output quality even when the model itself hasn't improved

• [ ] Document results as they actually are, imperfections included — that's usually where the real lesson is

All three notebooks, full results, and more detail on both findings above are in the GitHub repo.

Proyek lengkapnya (3 notebook + dokumentasi hasil) ada di GitHub.

Daftar Isi

• Kenapa Dua Pendekatan Sekaligus?

• Coba Tebak Dulu

• Case Study 1: Code Generation dari Nol

• Deploy Jadi API

• Case Study 2: Fine-Tuning GPT-2 Bahasa Indonesia

• Plot Twist: Dua Angka, Dua Cerita Berbeda

• Tabel Perbandingan

• Kuis Singkat

• Ringkasan

Kenapa Dua Pendekatan Sekaligus?

Training dari nol artinya model mulai dari bobot acak, tidak tahu apa-apa soal bahasa atau kode. Butuh data besar dan banyak langkah training sebelum menghasilkan sesuatu yang masuk akal.

Fine-tuning artinya mulai dari model yang sudah dilatih sebelumnya (pretrained), lalu disesuaikan ke tugas atau domain baru dengan data yang jauh lebih sedikit.

Dua pendekatan ini punya trade-off yang benar-benar berbeda, dan cara paling jelas untuk memahaminya bukan dari teori, tapi dari menjalankan keduanya sendiri dan melihat apa yang sebenarnya keluar.

Coba Tebak Dulu

Sebelum baca lanjut, coba tebak dulu:

Kalau GPT-2 dilatih dari nol selama 196 steps, dan GPT-2 pretrained di-fine-tune selama 250 steps pada dataset kecil (cuma 50 kalimat) — mana yang menurutmu menghasilkan teks lebih baik? Loss lebih rendah itu selalu berarti model lebih bagus, atau ada jebakannya?

Simpan dulu jawabanmu. Nanti kita cocokkan di bagian Plot Twist.

Case Study 1: Code Generation dari Nol

Notebook pertama melatih arsitektur GPT-2 (124 juta parameter) dari bobot acak sepenuhnya untuk menyelesaikan potongan kode Python, memakai dataset CodeParrot yang sudah difilter untuk konten data science (numpy, pandas, sklearn, dst).

Alurnya:

1. Load dataset yang sudah pre-filtered (huggingface-course/codeparrot-ds-train) — 50.000 sampel

2. Tokenisasi dengan tokenizer khusus kode

3. Bangun konfigurasi GPT-2 kosong, inisialisasi model dengan bobot acak

4. Training 1 epoch (196 steps)

5. Test: minta model melanjutkan kode dari komentar

Hasil loss akhir: 6.43. Waktu tes minta model melanjutkan:

# create some data
x = np.random.randn(100)
y = np.random.randn(100)

# create dataframe from x and y

Modelnya bukan melanjutkan kode pandas seperti yang diharapkan, malah muter-muter di teks lisensi open-source:

# the of the data
# with the terms of the
# This is the terms of the terms of the Free Software Foundation...
# but WITHOUT ANY WARRANTIES OR CONDITIONS OF ANY WITHOUT ANY WARRANTY...

Awalnya kelihatan seperti kegagalan. Tapi kalau dipikir ulang, ini sebenarnya masuk akal: banyak file kode di GitHub diawali blok komentar lisensi yang panjang dan berulang di ribuan repo berbeda. Untuk model yang baru mulai belajar dari bobot acak, pola itu jauh lebih "mudah dihafal" duluan dibanding memahami struktur kode Python yang sesungguhnya.

Deploy Jadi API

Model dari notebook 1 lalu dibungkus jadi endpoint REST pakai FastAPI, di-expose lewat ngrok. Menariknya, model yang sama persis menghasilkan output yang terasa berbeda ketika parameter sampling (top_k=50, top_p=0.9, temperature=0.8) diaktifkan — bukan lagi muter-muter teks lisensi, tapi mulai muncul token yang lebih mirip kode (import, def __init__, class, assert), meski tetap belum jadi kode Python yang valid.

Jadi strategi decoding ternyata ikut menentukan kesan kualitas output, bahkan untuk model yang levelnya sama-sama belum matang.

Case Study 2: Fine-Tuning GPT-2 Bahasa Indonesia

Notebook ketiga jalan dua bagian. Bagian pertama cuma eksplorasi model Indonesia yang sudah ada — fill-mask pakai BERT dan RoBERTa Indonesia, lalu text generation pakai GPT-2 Indonesia pretrained (belum di-fine-tune apapun). Kedua model fill-mask sama-sama percaya diri menebak preposisi "di" untuk kalimat "Ibu ku sedang bekerja [MASK] supermarket" — BERT dengan confidence 94.96%, RoBERTa 64.92%. Sanity check sederhana yang menunjukkan model-model ini memang jalan sesuai ekspektasi.

Bagian kedua, iseng coba fine-tune cahya/gpt2-small-indonesian-522M pada dataset dummy berisi 50 kalimat fakta dinosaurus dalam berbagai gaya bahasa (ensiklopedia, gaul, cerita anak). Training 10 epoch, loss turun ke 1.27 — jauh lebih rendah dari notebook 1.

Waktu diminta melanjutkan prompt "Trex merupakan", hasilnya:

Trex merupakan dinosaurus karnivora pertama yang diketahui memiliki adaptasi unik untuk hidup dan berburu di dalam air. Bukti yang dapat diandalkan untuk hidup dan berburu di dalam air menunjukkan bahwa T-Rex memiliki adaptasi unik untuk hidup dan berburu di dalam air...

Kelihatan fasih banget di sekilas baca. Tapi ini yang jadi bahan renungan di bagian berikutnya.

Plot Twist: Dua Angka, Dua Cerita Berbeda

Iseng saya bandingkan kalimat generate di atas dengan data training-nya sendiri, dan ketemu ini:

Data training (soal Spinosaurus): "Spinosaurus adalah dinosaurus karnivora pertama yang diketahui memiliki adaptasi unik untuk hidup dan berburu di dalam air*."*

Hasil generate (soal T-Rex): "Trex merupakan dinosaurus karnivora pertama yang diketahui memiliki adaptasi unik untuk hidup dan berburu di dalam air*."*

Hampir sama persis, cuma subjeknya diganti. Dengan cuma 50 contoh dilatih 10 epoch pada model yang sudah pretrained, ini kemungkinan besar tanda overfitting: modelnya menjahit ulang potongan kalimat yang dihafal, alih-alih benar-benar men-generate kalimat baru.

Ada satu detail metodologi yang bikin ini makin jelas: notebook ini memakai dataset yang sama untuk training dan evaluasi (eval_dataset=tokenized_dataset, identik dengan train_dataset). Artinya loss evaluasi yang tercatat sebenarnya tidak bisa dipakai untuk mendeteksi overfitting sama sekali — butuh split validasi terpisah untuk mengukur ini secara jujur.

Jadi loss rendah dan output fasih di notebook 3 belum tentu tanda model yang lebih baik; bisa jadi cuma tanda menghafal dataset kecil. Sebaliknya, loss tinggi dan output berantakan di notebook 1 tetap konsekuensi wajar dari training dari nol dengan compute terbatas, bukan tanda pipeline-nya gagal.

Tabel Perbandingan

Kuis Singkat

Q1: Loss yang lebih rendah selalu berarti model yang lebih baik? Tidak selalu. Loss rendah pada dataset yang sangat kecil (seperti 50 kalimat di notebook 3) bisa berarti model cuma menghafal, bukan belajar pola bahasa secara umum.

Q2: Kalau eval_dataset sama dengan train_dataset, apa dampaknya? Metrik evaluasi jadi tidak informatif untuk mendeteksi overfitting, karena model dievaluasi pakai data yang sama persis dengan yang dipakai untuk belajar.

Q3: Kenapa output notebook 1 dan API di notebook 2 beda, padahal model-nya sama? Karena strategi decoding beda — notebook 1 pakai decoding default/deterministik, sementara API di notebook 2 pakai top_k/top_p/temperature, yang mengubah cara token berikutnya dipilih.

Ringkasan

• [ ] Training dari nol butuh data + steps jauh lebih banyak dari yang dipakai di sini untuk hasil yang koheren

• [ ] Loss rendah pada dataset kecil bisa jadi tanda overfitting, bukan keberhasilan

• [ ] Selalu pakai validation split terpisah kalau mau benar-benar mengukur generalisasi

• [ ] Strategi decoding (top_k, top_p, temperature) bisa mengubah kesan kualitas output meski model-nya sama

• [ ] Dokumentasikan hasil apa adanya — termasuk yang belum sempurna — karena di situ justru pelajarannya

Semua notebook, hasil lengkap, dan detail lebih jauh soal dua temuan di atas ada di repo GitHub-nya.

Table of Contents

• Guess First

• What Are Transformers & Self-Attention

• The Project: an Indonesian Intent Classifier

• Plot Twist: Caught Overfitting

• Investigating & Fixing It

• Why Did Every Model End Up Near-Perfect?

• Quiz

• Summary

Guess First

Before reading on: if you train a 6-layer Transformer Encoder (millions of parameters) from scratch, using only 75 training sentences, for a fixed 20 epochs with no validation split — what do you think happens to the training loss?

• A) It decreases slowly and settles at a reasonable value

• B) It collapses toward zero well before the epochs run out

• C) It actually increases because the model is too small

The answer is in the "Plot Twist" section below.

What Are Transformers & Self-Attention

A Transformer is an architecture built to handle sequential data like text, without processing it word by word the way RNNs or LSTMs do. Introduced in "Attention Is All You Need" (Vaswani et al., 2017), its core idea is self-attention: every word in a sentence can directly "look at" every other word, regardless of position, to understand context.

Self-attention runs on three components:

• Query (Q) — the representation of the word currently being processed

• Key (K) — the representation of every other word in the sequence

• Value (V) — the content or context carried by those other words

Since a Transformer processes an entire sequence at once rather than one token at a time, it needs Positional Encoding to keep track of word order. The standard approach uses a sinusoidal formula (sin/cos) that gives each position a unique numerical fingerprint. Multi-Head Attention then runs several of these attention computations in parallel, so the model can capture different kinds of relationships between words at the same time.

The Project: an Indonesian Intent Classifier

To put these ideas into practice, I built an intent classifier — a model that guesses the intent behind a short sentence, across three categories: greeting, sekarang_jam_berapa (asking for the time), and siapa_anda (asking for identity).

The architecture is a TransformerClassifier, encoder-only, built on PyTorch's nn.TransformerEncoder (which already implements multi-head self-attention, a feed-forward network, and residual connections internally), sitting on top of embeddings from an Indonesian BERT tokenizer (cahya/bert-base-indonesian-522M).

Plot Twist: Caught Overfitting

The first training run looked "successful" — loss kept dropping. But a closer look revealed something off:

Loss collapsed to nearly zero by epoch 9 out of 20 — the answer to the guess above is B. That's not a sign of good learning, it's a sign the model memorized the training set. Which makes sense: a 6-layer Transformer with a hidden dimension of 768 has millions of parameters, trained on just 75 sentences, with no validation split to signal when to stop.

So I audited my own system: what caused it, and how do you actually fix it?

Investigating & Fixing It

Three things were tried at once:

1. Validation split + early stopping — training stops as soon as validation loss stops improving, instead of running a fixed number of epochs

2. A smaller model (scratch_small: 2 layers, hidden size 128, higher dropout and weight decay) — to test whether regularization alone was enough

3. Actual transfer learning (finetuned_bert) — fine-tuning pretrained Indonesian BERT weights, not just borrowing its tokenizer

To avoid depending on one lucky or unlucky data split, all three variants were compared using 5-fold Stratified Cross-Validation:

Early stopping and a validation split appeared to help: the exact same scratch_large architecture, once its training procedure changed (validation split + early stopping) and it was evaluated as a 5-fold average instead of a single split, went from a single-split accuracy of roughly 89–95% to an average CV accuracy of 98.95%. Worth noting: part of that jump also comes from the measurement itself — a 5-fold average is inherently more stable than one small split, so this isn't a perfectly isolated comparison. What's clearer is finetuned_bert: consistently ahead with zero variance across every fold, matching the hypothesis that pretrained language representations need far less data to generalize.

Why Did Every Model End Up Near-Perfect?

This is the part worth being honest about. All three variants scored above 97%, including the deliberately shrunk model. That's suspicious on its own.

So the dataset itself got a closer look. Vocabulary overlap between the three classes turned out to be only 8–10% (Jaccard similarity), and the overlapping words were mostly generic function words ("ada", "bisa", "ini", "yang", "kamu", "apa") — not topic-specific keywords. The actual keywords for each class (jam/waktu/pukul for time, siapa/nama/dirimu for identity, hi/halo/selamat for greetings) barely overlap at all. No duplicate rows were found either.

In other words, this task is inherently easy to separate lexically. That's not a bug and not data leakage — it's a property of a small, simple dataset. These results are valid evidence that the methodology (cross-validation, early stopping, transfer learning) works correctly, not proof that the model is generally "great." A larger, more ambiguous dataset would likely show clearer differences between the three variants.

Quiz

1. Why does a Transformer need Positional Encoding while an RNN doesn't? a) A Transformer doesn't have enough parameters b) A Transformer processes all tokens at once, so it has no built-in sense of order c) Positional Encoding is only there to speed up training

Answer: b — an RNN processes tokens sequentially, so order is implicit in how it works, while a Transformer processes everything in parallel and needs explicit position information.

2. In the experiment above, why did scratch_large jump from ~89% to 98.95% with the exact same architecture? a) Because more data was added b) Because the training procedure was fixed (validation split + early stopping), not the architecture c) Because the model was restarted from scratch

Answer: b — the architecture didn't change at all. What changed was the training procedure (validation split + early stopping) and the evaluation procedure (a 5-fold average instead of a single split). This shows the original problem was indeed about training strategy, though part of the numeric jump is also explained by the more stable measurement.

3. Why should near-perfect scores across every model variant raise suspicion instead of being celebrated? a) Because it means the code must be buggy b) Because it can signal an easy or small task rather than a genuinely strong model c) Because accuracy above 95% is always invalid

Answer: b — a perfect score on a small, simple dataset doesn't automatically prove good generalization; the context needs to be checked first.

Summary

• [x] A Transformer uses self-attention (Query/Key/Value) to understand context between words globally

• [x] Positional Encoding (sinusoidal) is needed because a Transformer doesn't process tokens sequentially

• [x] Training without a validation split and early stopping is prone to overfitting, even on small datasets

• [x] 5-fold Cross-Validation gives a far more stable performance estimate than a single split

• [x] Transfer learning (fine-tuned BERT) won this experiment, but near-perfect scores across every variant still need the dataset's context checked, not taken at face value

Full code, notebook, and experiment results are on GitHub: transformer-self-attention-intent-classifier

Daftar Isi

• Coba Tebak Dulu

• Apa Itu Transformer & Self-Attention

• Proyek: Intent Classifier Bahasa Indonesia

• Plot Twist: Overfitting Ketauan

• Investigasi & Perbaikan

• Kenapa Semua Model Jadi Nyaris Sempurna?

• Quiz

• Ringkasan

Coba Tebak Dulu

Sebelum lanjut baca: kalau kamu latih Transformer Encoder 6-layer (jutaan parameter) dari nol, cuma pakai 75 kalimat data latih, selama 20 epoch tetap tanpa validation split — kira-kira apa yang terjadi ke training loss-nya?

• A) Turun pelan dan stabil di angka wajar

• B) Turun drastis ke hampir nol jauh sebelum epoch-nya habis

• C) Malah naik karena modelnya kekecilan

Jawabannya ada di bagian "Plot Twist" di bawah.

Apa Itu Transformer & Self-Attention

Transformer adalah arsitektur yang dirancang buat menangani data berurutan seperti teks, tanpa harus memprosesnya kata demi kata seperti RNN atau LSTM. Diperkenalkan lewat paper "Attention Is All You Need" (Vaswani dkk., 2017), inti dari Transformer ada di mekanisme self-attention: setiap kata dalam kalimat bisa langsung "melihat" kata lain di posisi mana pun untuk memahami konteks.

Tiga komponen kunci di balik self-attention:

• Query (Q) — representasi kata yang sedang diproses

• Key (K) — representasi kata lain dalam urutan

• Value (V) — isi atau konteks dari kata-kata lain

Karena semua token diproses sekaligus (bukan satu per satu), Transformer butuh Positional Encoding supaya tetap tahu urutan kata. Biasanya dipakai formula sinusoidal (sin/cos) yang memberi tiap posisi "sidik jari" numerik yang unik. Lalu Multi-Head Attention menjalankan beberapa proses attention ini secara paralel, supaya model menangkap berbagai jenis hubungan antar kata dalam satu waktu.

Proyek: Intent Classifier Bahasa Indonesia

Untuk mempraktikkan konsep di atas, aku bangun sebuah intent classifier — model yang menebak maksud dari kalimat pendek, di antara tiga kategori: greeting (sapaan), sekarang_jam_berapa (nanya waktu), dan siapa_anda (nanya identitas).

Arsitekturnya: TransformerClassifier encoder-only, memakai nn.TransformerEncoder bawaan PyTorch (yang di dalamnya sudah mengimplementasikan multi-head self-attention, feed-forward network, dan residual connection) di atas embedding dari tokenizer BERT Bahasa Indonesia (cahya/bert-base-indonesian-522M).

Plot Twist: Overfitting Ketauan

Training pertama kelihatan "berhasil" — loss-nya turun terus. Tapi begitu dilihat lebih detail, ada yang janggal:

Loss-nya turun ke hampir nol sejak epoch ke-9 dari 20 — jawaban "Coba Tebak Dulu" di atas adalah B. Ini bukan tanda model belajar dengan baik, ini tanda model menghafal training set-nya. Masuk akal juga: Transformer 6-layer dengan hidden dimension 768 itu jutaan parameter, dilatih cuma dari 75 kalimat, tanpa validation split yang bisa kasih tahu kapan harus berhenti.

Jadi aku audit sistem aku sendiri: apa penyebabnya, dan gimana cara membenahinya?

Investigasi & Perbaikan

Tiga hal dicoba sekaligus:

1. Validation split + early stopping — training berhenti begitu validation loss berhenti membaik, bukan di epoch tetap

2. Model yang lebih kecil (scratch_small: 2 layer, hidden 128, dropout & weight decay lebih tinggi) — untuk menguji apakah regularisasi saja sudah cukup

3. Transfer learning beneran (finetuned_bert) — fine-tune bobot BERT Indonesia yang sudah pretrained, bukan cuma memakai tokenizer-nya

Supaya hasilnya tidak bergantung pada satu split data yang kebetulan mudah atau sulit, ketiga varian dibandingkan pakai 5-fold Stratified Cross-Validation:

Early stopping dan validation split kelihatan berdampak: model scratch_large dengan arsitektur yang sama persis, begitu cara trainingnya diubah (validation split + early stopping) dan dievaluasi lewat rata-rata 5 fold (bukan cuma satu split), akurasinya naik dari sekitar 89–95% (single split) ke rata-rata CV 98,95%. Perlu dicatat, sebagian kenaikan ini juga berasal dari cara mengukurnya — rata-rata 5 fold secara alami lebih stabil dibanding satu split kecil, jadi ini bukan perbandingan yang sepenuhnya terisolasi ke satu variabel. Yang lebih jelas kelihatan adalah finetuned_bert: unggul konsisten dengan variance nol di semua fold — sejalan dengan hipotesis bahwa representasi bahasa yang sudah dipelajari sebelumnya butuh lebih sedikit data untuk generalisasi.

Kenapa Semua Model Jadi Nyaris Sempurna?

Ini bagian yang paling penting untuk disampaikan jujur. Ketiga varian sama-sama mendapat skor di atas 97%, termasuk model yang sengaja dikecilkan. Itu mencurigakan.

Jadi datasetnya dicek langsung. Ternyata: overlap kosakata antar tiga kelas cuma 8–10% (Jaccard similarity), dan kata-kata yang overlap itu kebanyakan kata fungsi umum ("ada", "bisa", "ini", "yang", "kamu", "apa") — bukan kata kunci yang jadi ciri khas topik. Kata konten tiap kelas (jam/waktu/pukul vs siapa/nama/dirimu vs hi/halo/selamat) nyaris tidak beririsan sama sekali. Tidak ada duplikasi data juga.

Artinya, task ini secara alami mudah dipisahkan secara leksikal — bukan bug, bukan data leakage, tapi karakteristik dataset yang kecil dan sederhana. Hasil ini bukti valid bahwa metodologinya (cross-validation, early stopping, transfer learning) bekerja dengan benar, bukan klaim bahwa modelnya "hebat" secara umum. Dataset yang lebih besar dan ambigu kemungkinan akan menunjukkan perbedaan yang lebih jelas antar ketiga model.

Quiz

1. Kenapa Transformer butuh Positional Encoding, sementara RNN tidak? a) Transformer tidak punya cukup parameter b) Transformer memproses semua token sekaligus, jadi tidak otomatis tahu urutannya c) Positional Encoding cuma untuk mempercepat training

Jawaban: b — RNN memproses token secara berurutan sehingga urutan implisit dari cara kerjanya, sementara Transformer memproses semua sekaligus sehingga butuh informasi posisi eksplisit.

2. Di eksperimen di atas, kenapa scratch_large bisa naik dari sekitar 89% ke 98,95% padahal arsitekturnya sama persis? a) Karena datanya ditambah b) Karena cara training-nya diperbaiki (validation split + early stopping), bukan arsitekturnya c) Karena modelnya direstart dari awal

Jawaban: b — arsitekturnya tidak berubah sama sekali. Yang berubah adalah cara training (validation split + early stopping) sekaligus cara evaluasi (rata-rata 5 fold, bukan satu split). Ini menunjukkan masalah awal memang lebih ke soal strategi training, meskipun sebagian kenaikan angkanya juga dipengaruhi cara pengukuran yang lebih stabil.

3. Kenapa skor yang nyaris sempurna di semua varian model justru perlu dicurigai, bukan langsung dirayakan? a) Karena itu artinya kodenya pasti ada bug b) Karena itu bisa jadi tanda task atau dataset-nya terlalu mudah atau kecil, bukan bukti model yang benar-benar hebat c) Karena akurasi di atas 95% selalu tidak valid

Jawaban: b — skor sempurna di dataset kecil dan sederhana tidak otomatis membuktikan generalisasi yang baik; konteksnya perlu dicek dulu.

Ringkasan

• [x] Transformer memakai self-attention (Query/Key/Value) untuk memahami konteks antar kata secara global

• [x] Positional Encoding (sinusoidal) dibutuhkan karena Transformer tidak memproses token secara berurutan

• [x] Training tanpa validation split dan early stopping rawan overfitting, bahkan di dataset kecil

• [x] 5-fold Cross-Validation memberi estimasi performa yang jauh lebih stabil dibanding satu split

• [x] Transfer learning (fine-tuned BERT) menang di eksperimen ini, tapi skor nyaris sempurna di semua varian tetap perlu dicek konteks dataset-nya, bukan langsung dipercaya begitu saja

Kode lengkap, notebook, dan hasil eksperimennya ada di GitHub: transformer-self-attention-intent-classifier

This note walks through building a spam classifier by fine-tuning BERT: why BERT fits this task, and code that's actually run and evaluated, not just described.

Table of Contents

• The Problem: What Does Spam Look Like?

• Why BERT Instead of Bag-of-Words?

• BERT's Architecture in 3 Blocks

• Why Is BERT Already "Smart" Before Fine-tuning?

• Fine-tuning for Spam Classification

• Dataset & Practical Pipeline

• Evaluation: Why Not Just Accuracy

• Plot Twist: What Does the Official Bootcamp Notebook Actually Do?

• Inference Demo

• Quick Quiz

The Problem: What Does Spam Look Like?

We need a rule that survives the sheer variety of language and spam tricks. SPAM examples usually push urgency, dangle money or prizes, and include a link or number:

"Loan approved in 5 minutes, message us now."

HAM (not spam), on the other hand, has clear context and doesn't push you toward a click, link, or transfer:

"Can you send me the revised file, please?"

💭 Guess First: Why would a simple rule like "flag any message containing the word 'free' or 'click'" fail often?

Answer: because the same word shows up in perfectly normal contexts too — "free shipping" is fine, but "free iPhone, click here" is a different story. The model needs to understand context, not just spot keywords.

Why BERT Instead of Bag-of-Words?

This is what sets BERT apart from classic approaches like Bag-of-Words or TF-IDF:

BERT (an encoder-only Transformer) reads words with left-and-right context at once, through self-attention. The word "click" ends up "paying more attention" to words like "link," "prize," "now," making spam patterns easier to catch.

BERT's Architecture in 3 Blocks

1. Tokenizer (WordPiece) — text is split into subwords. Example: "playing" → "play" + "##ing".

2. Embedding Layer — three embeddings are summed:

Final embedding = Token Embedding + Position Embedding + Segment Embedding

1. Encoder Stack + Task Head — 12 encoder layers (BERT-Base). For classification, the [CLS] vector is taken and passed through a dense layer to produce the label.

Inside each encoder layer: Multi-Head Self-Attention (every token "looks at" every other token), a Feed-Forward Network (processes each token's features), and Residual + LayerNorm (keeps training stable).

Why Is BERT Already "Smart" Before Fine-tuning?

Before fine-tuning, BERT is already pretrained on massive amounts of text through two tasks (the classic setup — some derivative models like RoBERTa drop the second one):

• Masked Language Modeling (MLM): some tokens are hidden ([MASK]), and the model predicts them → learns left and right context simultaneously.

• Next Sentence Prediction (NSP): the model predicts whether sentence B follows sentence A → learns relationships between sentences.

This is why fine-tuning for a specific task like spam detection can work with relatively little data. BERT already comes in with a head start on "understanding language."

Fine-tuning for Spam Classification

The fine-tuning setup is simple:

Text (message) → BERT (encoder) → Spam? (0/1)

What gets trained: the classification head (dense layer) is always trained, and typically the entire BERT model is updated too (end-to-end), using cross-entropy loss for the two classes.

Starting hyperparameters:

learning_rate = 2e-5
batch_size = 16
epochs = 3
max_length = 128

Dataset & Practical Pipeline

For the implementation, we use the SMS Spam Collection v.1 (Almeida et al., 2011). The official release contains 5,574 English SMS messages; the GitHub mirror used here contains 5,572 rows (two fewer than the official release, likely a mirror-side parsing difference).

Verified statistics (computed directly from the data, not assumed):

Stratified 80/10/10 split:

The classes are imbalanced (~87:13). This is why plain accuracy isn't the metric to trust, covered next.

Tokenization and training setup (matching the notebook exactly):

MODEL_NAME = "bert-base-uncased"
MAX_LENGTH = 128

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

def tokenize(texts):
    return tokenizer(
        list(texts),
        padding="max_length",
        truncation=True,
        max_length=MAX_LENGTH,
        return_tensors="pt",
    )

model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME, num_labels=2)

Evaluation: Why Not Just Accuracy

With an imbalanced dataset (87% ham, 13% spam), a model that always guesses "ham" already scores 87% accuracy, while completely failing to catch spam. That's why we read:

• Precision (spam): of everything predicted as spam, how much was actually spam?

• Recall (spam): of the actual spam, how much did the model catch?

• F1: the harmonic mean of precision and recall.

Low recall → a lot of spam slips through. Low precision → normal messages get wrongly flagged as spam.

Out of 517 test messages: 450 ham correctly predicted, 2 ham wrongly flagged as spam, 2 spam wrongly flagged as ham, 63 spam correctly predicted.

Plot Twist: What Does the Official Bootcamp Notebook Actually Do?

While digging further into the bootcamp material, I found the official reference notebook, and its implementation turns out to be different from the theory above. Instead of full fine-tuning, it freezes every BERT parameter:

for param in bert.parameters():
    param.requires_grad = False

Only a custom classification head (Linear 768→512→2) sitting on top of the frozen BERT gets trained. That's feature extraction, not the full fine-tuning described earlier. The same notebook also builds a Naive Bayes baseline for comparison, and the result is a bit surprising: on their test set, Naive Bayes (F1 spam 0.97) slightly outperforms the frozen-BERT model (F1 spam 0.94).

BERT isn't failing here. The SMS Spam Collection dataset is relatively small and the patterns are fairly obvious: keywords like "free," "win," "claim" are already highly informative on their own, which lets a simpler model stay competitive (this reading hasn't been checked against actual feature importance, so treat it as a plausible explanation rather than a settled conclusion). The steadier takeaway: BERT's performance depends on dataset size, pattern complexity, and whether its parameters are actually being fine-tuned or just used as a frozen feature extractor.

For a fair comparison (same test set, not just numbers borrowed from another source), all three approaches — Naive Bayes, frozen-BERT, and full fine-tuning — were re-run in this notebook on the identical data split:

Full fine-tuning does come out ahead on every metric for this split. But frozen-BERT has perfect recall (1.0000, meaning no spam slipped through at all), at the cost of the lowest precision of the three (more ham messages wrongly flagged as spam). There's a genuine trade-off here, depending on what matters more: catching every piece of spam, or avoiding false alarms on normal messages.

Inference Demo

Once trained, here's how the model gets tested on new text:

def predict_spam(texts, model=model, tokenizer=tokenizer):
    model.eval()
    enc = tokenizer(texts, padding=True, truncation=True, max_length=MAX_LENGTH, return_tensors="pt")
    with torch.no_grad():
        logits = model(**enc).logits
        probs = torch.softmax(logits, dim=1)[:, 1]
    return [
        {"text": t, "label": "SPAM" if p >= 0.5 else "HAM", "p_spam": round(p.item(), 4)}
        for t, p in zip(texts, probs)
    ]

Sample inputs tested, with actual results:

The third row is worth talking about. That text was written to sound like loan-scam spam (urgency, a push to make contact fast), but the model predicted HAM with high confidence. A likely explanation: the SMS Spam Collection dataset dates to around 2011, before "WhatsApp" was a common term in English-language spam, so this phrasing probably falls outside the patterns the model ever saw during training. A model with 99% accuracy can still miss on a sentence whose pattern isn't in its training data.

Quick Quiz

Q1. Why does BERT outperform TF-IDF on spam messages with highly varied phrasing? A: Because BERT captures context (through self-attention) instead of just counting word frequency.

Q2. Why isn't accuracy alone enough for an imbalanced spam dataset (87% ham, 13% spam)? A: Because a model that always predicts "ham" already scores high accuracy without actually detecting any spam. Precision and recall are needed to see how it performs on the minority class.

Q3. What's the difference between using BERT as a "frozen feature extractor" and "full fine-tuning"? A: With frozen feature extraction, every BERT parameter is frozen (requires_grad=False) and only a custom classification head is trained. With full fine-tuning, all of BERT's parameters get updated along with the head. Frozen is faster and cheaper to train, but not always more accurate.

Q4. The model above scored 99.23% accuracy on the test set, yet misclassified "URGENT loan approved in 5 minutes, WhatsApp us now!" as HAM. Why would that happen? A: High test-set accuracy only measures performance on data whose patterns resemble the training data. That sentence uses a term ("WhatsApp") likely absent from the training set (dating to around 2011), putting it outside the distribution the model ever learned from. High accuracy doesn't guarantee generalization to new patterns.

Summary Checklist

• [x] Understand the flow: data → tokenization → fine-tuning → evaluation

• [x] Understand BERT's input: token, position, segment embeddings

• [x] Know how to train BERT for binary classification

• [x] Can read precision, recall, F1, and a confusion matrix

• [x] Can test predictions on new text

• [x] Understand the difference between BERT as a frozen feature extractor and full fine-tuning

• [x] Understand why high test-set accuracy doesn't guarantee correctness on every new pattern

Full code available on GitHub

Catatan ini membahas cara membangun spam classifier dengan fine-tuning BERT: alasan BERT cocok untuk tugas ini, sampai kode yang benar-benar dijalankan dan dievaluasi.

Daftar Isi

• Masalah: Spam Itu Seperti Apa?

• Kenapa BERT, Bukan Bag-of-Words?

• Arsitektur BERT dalam 3 Blok

• Kenapa BERT Sudah "Pintar" Sebelum Dilatih?

• Fine-tuning untuk Klasifikasi Spam

• Dataset & Pipeline Praktik

• Evaluasi: Kenapa Bukan Cuma Accuracy

• Plot Twist: Apa Kata Notebook Resmi Bootcamp?

• Demo Inferensi

• Quiz Singkat

Masalah: Spam Itu Seperti Apa?

Kita butuh aturan yang tahan variasi bahasa dan trik spam. Contoh SPAM biasanya punya ajakan cepat, iming-iming hadiah/uang, ada link atau nomor, dan nadanya mendesak:

"Pinjaman cair 5 menit, WA sekarang."

Sedangkan HAM (bukan spam) konteksnya jelas, dan tidak memaksa klik/link/transfer:

"Tolong kirim file revisi ya."

💭 Coba Tebak Dulu: Menurutmu, kenapa cara sederhana seperti "cek apakah ada kata 'gratis' atau 'klik'" gampang salah tebak?

Jawaban: karena kata yang sama bisa muncul di konteks normal — "gratis ongkir" itu wajar, tapi "gratis iPhone klik link" beda cerita. Model butuh memahami konteks, bukan cuma mendeteksi kata kunci.

Kenapa BERT, Bukan Bag-of-Words?

Ini yang bikin BERT beda dari pendekatan klasik seperti Bag-of-Words atau TF-IDF:

BERT (encoder-only Transformer) membaca kata dengan konteks kiri-kanan sekaligus lewat mekanisme self-attention. Kata "klik" akan lebih "memperhatikan" kata "link", "hadiah", "sekarang", sehingga pola spam lebih mudah terdeteksi.

Arsitektur BERT dalam 3 Blok

1. Tokenizer (WordPiece) — teks dipecah jadi subword. Contoh: "playing" → "play" + "##ing".

2. Embedding Layer — tiga embedding dijumlahkan:

Embedding final = Token Embedding + Position Embedding + Segment Embedding

1. Encoder Stack + Task Head — 12 layer encoder (BERT-Base). Untuk klasifikasi, vektor [CLS] diambil lalu dilewatkan ke Dense layer untuk menghasilkan label.

Di dalam tiap layer encoder ada Multi-Head Self-Attention (tiap token "melihat" token lain), Feed-Forward Network (memproses fitur tiap token), dan Residual + LayerNorm (menstabilkan training).

Kenapa BERT Sudah "Pintar" Sebelum Dilatih?

Sebelum fine-tuning, BERT sudah dilatih lebih dulu (pretraining) di data teks yang sangat besar, lewat dua tugas (versi klasik — beberapa model turunan seperti RoBERTa membuang tugas kedua):

• Masked Language Modeling (MLM): sebagian token disembunyikan ([MASK]), model menebak token yang hilang → belajar konteks kiri & kanan sekaligus.

• Next Sentence Prediction (NSP): model menebak apakah kalimat B lanjutan kalimat A → belajar hubungan antar kalimat.

Ini kenapa fine-tuning untuk tugas spesifik seperti spam classifier bisa jalan dengan data yang relatif sedikit. BERT sudah punya modal "paham bahasa" duluan.

Fine-tuning untuk Klasifikasi Spam

Arsitektur saat fine-tuning sederhana:

Teks (pesan) → BERT (encoder) → Spam? (0/1)

Yang dilatih: kepala klasifikasi (Dense) pasti dilatih, dan biasanya seluruh parameter BERT ikut di-update (end-to-end), dengan loss cross-entropy untuk 2 kelas.

Hyperparameter awal yang dipakai:

learning_rate = 2e-5
batch_size = 16
epochs = 3
max_length = 128

Dataset & Pipeline Praktik

Untuk implementasi, dipakai SMS Spam Collection v.1 (Almeida et al., 2011), dataset benchmark publik. Rilis resmi berisi 5.574 pesan SMS berbahasa Inggris; mirror GitHub yang dipakai di sini berisi 5.572 baris (selisih 2 baris dari rilis resmi, kemungkinan perbedaan parsing pada mirror).

Statistik terverifikasi (dihitung langsung dari data, bukan asumsi):

Split stratified 80/10/10:

Kelas tidak seimbang (~87:13). Ini alasan kenapa metrik selain accuracy penting dibaca di bagian berikutnya.

Kode tokenisasi dan setup training (persis seperti di notebook):

MODEL_NAME = "bert-base-uncased"
MAX_LENGTH = 128

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

def tokenize(texts):
    return tokenizer(
        list(texts),
        padding="max_length",
        truncation=True,
        max_length=MAX_LENGTH,
        return_tensors="pt",
    )

model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME, num_labels=2)

Evaluasi: Kenapa Bukan Cuma Accuracy

Dengan kelas tidak seimbang (87% ham, 13% spam), model yang asal tebak "semua ham" saja sudah dapat accuracy 87%, padahal gagal total mendeteksi spam. Karena itu kita baca:

• Precision (spam): dari yang diprediksi spam, berapa yang benar-benar spam?

• Recall (spam): dari spam asli, berapa yang berhasil ditangkap?

• F1: rata-rata harmonik precision & recall.

Recall rendah → banyak spam lolos. Precision rendah → pesan normal ikut ketandai spam (false alarm).

Dari 517 pesan di test set: 450 ham diprediksi benar, 2 ham salah ditandai spam, 2 spam salah ditandai ham, 63 spam diprediksi benar.

Plot Twist: Apa Kata Notebook Resmi Bootcamp?

Sambil menelusuri materi bootcamp lebih jauh, ada notebook referensi resmi yang ternyata implementasinya beda dari teori di atas. Alih-alih fine-tuning penuh, notebook itu membekukan seluruh parameter BERT:

for param in bert.parameters():
    param.requires_grad = False

Yang dilatih cuma head klasifikasi custom (Linear 768→512→2) di atas BERT yang beku. Ini namanya feature extraction, bukan fine-tuning penuh seperti yang dijelaskan di bagian sebelumnya. Notebook itu juga bikin baseline Naive Bayes untuk perbandingan, dan hasilnya cukup mengejutkan: pada test set mereka, Naive Bayes (F1 spam 0,97) sedikit mengungguli BERT-frozen (F1 spam 0,94).

BERT tidak otomatis gagal di sini. Dataset SMS Spam Collection relatif kecil dan polanya cukup jelas: kata kunci seperti "free", "win", "claim" sudah sangat informatif secara statistik, sehingga model sederhana bisa ikut bersaing (interpretasi ini belum diuji langsung lewat feature importance, jadi dianggap sebagai dugaan yang masuk akal, bukan kesimpulan final). Pelajaran yang lebih pasti: performa BERT bergantung pada ukuran data, kompleksitas pola, dan apakah parameter BERT-nya benar-benar ikut di-fine-tune atau cuma dipakai sebagai ekstraktor fitur beku.

Supaya perbandingan adil (test set yang sama, bukan sekadar comot angka dari sumber lain), ketiga pendekatan — Naive Bayes, BERT-frozen, dan BERT full fine-tuning — dijalankan ulang di notebook ini pada split data yang identik:

Hasilnya: fine-tuning penuh memang unggul di semua metrik pada split ini. Tapi frozen-BERT punya recall sempurna (1,0000, artinya tidak ada satu pun spam yang lolos), meski dengan precision paling rendah di antara ketiganya (lebih banyak pesan ham yang salah ditandai spam). Ada trade-off yang berbeda di sini, tergantung mana yang lebih penting: menangkap semua spam, atau menghindari false alarm ke pesan normal.

Demo Inferensi

Setelah model dilatih, begini cara mengujinya ke teks baru:

def predict_spam(texts, model=model, tokenizer=tokenizer):
    model.eval()
    enc = tokenizer(texts, padding=True, truncation=True, max_length=MAX_LENGTH, return_tensors="pt")
    with torch.no_grad():
        logits = model(**enc).logits
        probs = torch.softmax(logits, dim=1)[:, 1]
    return [
        {"text": t, "label": "SPAM" if p >= 0.5 else "HAM", "p_spam": round(p.item(), 4)}
        for t, p in zip(texts, probs)
    ]

Contoh input yang diuji, dengan hasil aktual:

Baris ketiga menarik untuk dibahas. Teks itu sengaja ditulis mirip spam pinjaman (urgensi, ajakan kontak cepat), tapi modelnya memprediksi HAM dengan keyakinan tinggi. Dugaan penyebabnya: dataset SMS Spam Collection berasal dari sekitar 2011, sebelum "WhatsApp" jadi istilah umum di teks spam berbahasa Inggris, jadi frasa ini kemungkinan besar berada di luar pola yang pernah dilihat model saat training. Model dengan accuracy 99% pun tetap bisa salah pada kalimat yang polanya tidak ada di data latihnya.

Quiz Singkat

Q1. Kenapa BERT lebih unggul dari TF-IDF untuk mendeteksi spam yang kalimatnya bervariasi? A: Karena BERT memahami konteks kata (lewat self-attention), bukan cuma menghitung frekuensi kemunculan kata.

Q2. Kenapa accuracy saja tidak cukup untuk dataset spam yang tidak seimbang (87% ham, 13% spam)? A: Karena model yang selalu menebak "ham" pun sudah dapat accuracy tinggi tanpa benar-benar mendeteksi spam. Precision & recall dibutuhkan untuk melihat performa di kelas minoritas.

Q3. Apa beda "BERT sebagai feature extractor (frozen)" dengan "fine-tuning penuh"? A: Pada frozen, seluruh parameter BERT dibekukan (requires_grad=False) dan hanya head klasifikasi custom yang dilatih. Pada fine-tuning penuh, seluruh parameter BERT ikut ter-update bersama head-nya. Frozen lebih cepat & hemat komputasi, tapi tidak selalu lebih akurat.

Q4. Model di atas dapat accuracy 99,23% di test set, tapi salah memprediksi "URGENT loan approved in 5 minutes, WhatsApp us now!" sebagai HAM. Kenapa ini bisa terjadi? A: Accuracy tinggi di test set hanya mengukur performa pada data yang polanya mirip dengan data training. Kalimat itu memakai istilah ("WhatsApp") yang kemungkinan tidak ada di dataset training (sekitar 2011), jadi berada di luar distribusi yang pernah dipelajari model. Accuracy tinggi tidak menjamin generalisasi ke pola baru.

Checklist Ringkasan

• [x] Paham alur: data → tokenisasi → fine-tuning → evaluasi

• [x] Paham input BERT: token, posisi, segment embedding

• [x] Tahu cara melatih BERT untuk klasifikasi 2 kelas

• [x] Bisa membaca precision, recall, F1, confusion matrix

• [x] Bisa menguji prediksi pada teks baru

• [x] Paham beda BERT sebagai feature extractor (frozen) vs fine-tuning penuh

• [x] Paham kenapa accuracy tinggi di test set tidak menjamin model benar di semua pola baru

Kode lengkap tersedia di GitHub

TL;DR: Word embedding merepresentasikan kata sebagai vektor supaya makna kata ikut terbawa. Dipadukan dengan Bidirectional LSTM dan pretrained GloVe, model ini mencapai akurasi 87.75% di test set IMDB — lalu dibungkus jadi REST API yang bisa dipanggil dari luar.

Daftar Isi

• Apa itu Word Embedding?

• Kenapa Pakai Pretrained Embedding (GloVe)?

• Arsitektur: Embedding + Bidirectional LSTM

• Training & Hasil

• Uji Coba Prediksi

• Bonus: Deploy Jadi REST API

• Checklist Pemahaman

• Kuis Singkat

Apa itu Word Embedding?

🤔 Coba Tebak Dulu: kalau komputer cuma paham angka, gimana caranya kata "bagus" dan "keren" bisa "dikenali" mirip maknanya oleh model?

Jawabannya: lewat **vektor**. Setiap kata dipetakan ke sebuah titik di ruang berdimensi tinggi (misalnya 100 dimensi). Kata-kata dengan makna atau konteks pemakaian yang mirip akan punya vektor yang saling berdekatan. Jadi "bagus" dan "keren" akan punya posisi yang relatif dekat, sementara "bagus" dan "sepatu" akan jauh. Word embedding adalah teknik merepresentasikan kata sebagai **vektor angka berdimensi tetap**, sedemikian rupa sehingga kata-kata dengan makna atau konteks pemakaian yang mirip punya representasi vektor yang berdekatan secara matematis (diukur pakai cosine similarity atau euclidean distance).

Ini beda jauh dari cara representasi kata yang lebih sederhana seperti one-hot encoding (yang dipakai di praktik RNN name-classifier pada tulisan sebelumnya) — di one-hot encoding, semua kata berjarak sama satu sama lain, tidak ada informasi makna yang terbawa. Word embedding menangkap hubungan semantik itu.

Kenapa Pakai Pretrained Embedding (GloVe)?

Melatih embedding dari nol butuh dataset sangat besar supaya vektornya benar-benar merepresentasikan makna kata dengan baik. Solusinya: pakai pretrained embedding yang sudah dilatih di dataset raksasa oleh pihak lain — di tulisan ini saya pakai GloVe (Global Vectors for Word Representation), spesifiknya varian glove.6B.100d: dilatih dari 6 miliar token, menghasilkan vektor 100 dimensi per kata.

Keuntungannya:

• Model tidak perlu belajar makna kata dari nol — cukup belajar cara memakai makna itu untuk tugas spesifik (dalam kasus ini: sentiment analysis).

• Berguna terutama kalau dataset training kita sendiri relatif kecil.

Arsitektur: Embedding + Bidirectional LSTM

Alur datanya:

Kalimat → Tokenisasi → Embedding Layer (GloVe) → Bidirectional LSTM → Dropout → Linear → Skor Sentimen

Beberapa poin penting:

• Embedding layer diinisialisasi dengan bobot GloVe yang sudah dilatih sebelumnya (bukan mulai dari acak).

• Bidirectional LSTM — LSTM dijalankan dua arah: dari kata pertama ke terakhir, dan dari kata terakhir ke pertama. Hidden state dari kedua arah digabung, sehingga model punya konteks penuh dari kedua sisi kalimat sebelum membuat keputusan.

• Dipakai 2 layer LSTM bertumpuk, dengan dropout 0.5 untuk mengurangi overfitting.

• Output akhir cuma 1 angka (lewat sigmoid) — mendekati 0 berarti negatif, mendekati 1 berarti positif. | Komponen | Nilai | |---|---| | Vocabulary size | 25.002 | | Embedding dimension | 100 | | Hidden dimension | 256 | | LSTM layers | 2, bidirectional | | Dropout | 0.5 | | Trainable parameters | 4.810.857 |

Training & Hasil

Dataset: IMDB Movie Reviews (50.000 review film, label positif/negatif). Dilatih 5 epoch dengan Adam optimizer dan BCEWithLogitsLoss.

Hasil terverifikasi (langsung dari output training):

Hasil di test set: Loss 0.303, Accuracy 87.75% — cukup solid untuk model yang "hanya" 5 epoch training, berkat bantuan pretrained embedding tadi.

Uji Coba Prediksi

Model cukup percaya diri di dua contoh ekstrem ini — masuk akal, karena kata "terrible" dan "great" adalah sinyal sentimen yang kuat dan sering muncul di data training.

Bonus: Deploy Jadi REST API

Model yang sudah jadi nggak ada gunanya kalau cuma nangkring di notebook. Jadi saya bungkus jadi REST API pakai FastAPI, lalu di-expose ke internet lewat ngrok (supaya bisa diakses dari luar Colab tanpa perlu server sendiri).

Endpoint-nya sederhana:

• GET / — health check

• POST /predict/ — kirim {"sentence": "..."}, dapat balik skor + label (very positive / positive / neutral / negative) Contoh response nyata dari API yang jalan:

{"sentence": "This film is great", "sentiment": "very positive", "score": 0.9846353530883789}

Skornya identik dengan hasil prediksi langsung di notebook training — masuk akal, karena API cuma memuat ulang bobot model yang sama.

⚠️ Catatan keamanan: kalau kamu bikin setup serupa, jangan pernah hardcode ngrok authtoken langsung di kode yang bakal di-share atau di-push ke repo publik. Simpan sebagai secret/environment variable.

Notebook lengkap (training + deployment) tersedia di GitHub: sentiment-analysis-lstm-glove-arielshakaramiro.

Konteks modern: GloVe + Bidirectional LSTM adalah pendekatan yang bagus untuk memahami fondasi word embedding dan sequential modeling, tapi untuk sentiment analysis produksi saat ini, model berbasis Transformer (BERT, RoBERTa, atau versi yang lebih ringan seperti DistilBERT) umumnya memberi akurasi lebih tinggi karena punya representasi kontekstual yang lebih kaya — kata yang sama bisa punya makna berbeda tergantung konteks kalimatnya, sesuatu yang GloVe (representasi statis per kata) tidak bisa tangkap.

Checklist Pemahaman

• [ ] Bisa menjelaskan bedanya word embedding dengan one-hot encoding

• [ ] Paham kenapa pretrained embedding (GloVe) berguna, terutama untuk dataset kecil

• [ ] Bisa menjelaskan kenapa bidirectional LSTM punya konteks lebih lengkap dibanding LSTM satu arah

• [ ] Paham alur dasar men-deploy model PyTorch jadi REST API

Kuis Singkat

1. Apa keuntungan utama word embedding dibanding one-hot encoding? Word embedding menangkap hubungan semantik antar kata (kata dengan makna mirip punya vektor yang berdekatan), sementara one-hot encoding memperlakukan semua kata sebagai entitas yang sama-sama berjarak, tanpa informasi makna.

2. Kenapa memakai LSTM bidirectional, bukan satu arah saja? Karena makna sebuah kata dalam kalimat kadang baru jelas setelah melihat kata-kata sesudahnya. LSTM bidirectional memproses kalimat dari dua arah sekaligus, sehingga setiap hidden state punya konteks dari kata sebelum dan sesudahnya.

3. Kenapa pakai pretrained embedding seperti GloVe alih-alih melatih embedding sendiri dari nol? Karena melatih embedding yang benar-benar menangkap makna kata butuh dataset sangat besar. GloVe sudah dilatih dari miliaran token, jadi model kita tinggal memanfaatkan representasi makna yang sudah matang, alih-alih belajar dari nol dengan dataset yang jauh lebih kecil.

Bagian dari seri "AI Notes & Engineering" — catatan belajar AI Engineering saya.

TL;DR: Word embeddings represent words as vectors so meaning carries over into the model. Paired with a Bidirectional LSTM and pretrained GloVe vectors, this model reaches 87.75% accuracy on the IMDB test set — then gets wrapped into a callable REST API.

Table of Contents

• What is Word Embedding?

• Why Use Pretrained Embeddings (GloVe)?

• Architecture: Embedding + Bidirectional LSTM

• Training & Results

• Testing Predictions

• Bonus: Deploying as a REST API

• Understanding Checklist

• Quick Quiz

What is Word Embedding?

🤔 Guess First: if a computer only understands numbers, how can it "recognize" that words like "great" and "awesome" carry similar meaning?

The answer: **vectors**. Each word gets mapped to a point in a high-dimensional space (say, 100 dimensions). Words with similar meaning or usage context end up with vectors that sit close together. So "great" and "awesome" land near each other, while "great" and "shoe" end up far apart. Word embedding is a technique for representing words as **fixed-dimensional numeric vectors**, arranged so that words with similar meaning or usage context have vectors that are mathematically close (measured via cosine similarity or Euclidean distance).

This is a big step up from simpler representations like one-hot encoding (which we used in the RNN name-classifier from the previous post) — with one-hot encoding, every word is equally distant from every other word, carrying no information about meaning at all. Word embeddings capture that semantic relationship.

Why Use Pretrained Embeddings (GloVe)?

Training embeddings from scratch requires a massive dataset for the vectors to genuinely capture word meaning. The workaround: use pretrained embeddings already trained on a huge corpus by someone else — here I used GloVe (Global Vectors for Word Representation), specifically glove.6B.100d: trained on 6 billion tokens, producing a 100-dimensional vector per word.

The benefits:

• The model doesn't need to learn word meaning from scratch — it just needs to learn how to use that meaning for the specific task (here, sentiment analysis).

• Especially useful when your own training dataset is relatively small.

Architecture: Embedding + Bidirectional LSTM

The data flow:

Sentence → Tokenization → Embedding Layer (GloVe) → Bidirectional LSTM → Dropout → Linear → Sentiment Score

A few key points:

• The embedding layer is initialized with pretrained GloVe weights instead of random values.

• Bidirectional LSTM — the LSTM runs in both directions: first word to last, and last word to first. Hidden states from both directions are concatenated, giving the model full context from both sides of the sentence before it makes a decision.

• Two stacked LSTM layers, with 0.5 dropout to reduce overfitting.

• The final output is a single number (via sigmoid) — close to 0 means negative, close to 1 means positive. | Component | Value | |---|---| | Vocabulary size | 25,002 | | Embedding dimension | 100 | | Hidden dimension | 256 | | LSTM layers | 2, bidirectional | | Dropout | 0.5 | | Trainable parameters | 4,810,857 |

Training & Results

Dataset: IMDB Movie Reviews (50,000 reviews, positive/negative labels). Trained for 5 epochs with the Adam optimizer and BCEWithLogitsLoss.

Verified results (pulled directly from the training run):

Test set results: Loss 0.303, Accuracy 87.75% — a solid result for just 5 epochs of training, thanks to the pretrained embeddings doing a lot of the heavy lifting.

Testing Predictions

The model is fairly confident on these two clear-cut examples — which makes sense, since "terrible" and "great" are strong sentiment signals that show up frequently in the training data.

Bonus: Deploying as a REST API

A trained model sitting in a notebook isn't much use to anyone. So I wrapped it in a REST API using FastAPI, then exposed it to the internet via ngrok (so it's reachable from outside Colab without needing a dedicated server).

The endpoints are simple:

• GET / — health check

• POST /predict/ — send {"sentence": "..."}, get back a score and a label (very positive / positive / neutral / negative) A real response from the live API:

{"sentence": "This film is great", "sentiment": "very positive", "score": 0.9846353530883789}

The score matches the direct prediction from the training notebook exactly — makes sense, since the API is just loading the same trained weights.

⚠️ Security note: if you set up something similar, never hardcode your ngrok authtoken in code that will be shared or pushed to a public repo. Store it as a secret or environment variable instead.

The full notebook (training + deployment) is available on GitHub: sentiment-analysis-lstm-glove-arielshakaramiro.

Modern context: GloVe + Bidirectional LSTM is a great way to understand the fundamentals of word embeddings and sequential modeling, but for production sentiment analysis today, Transformer-based models (BERT, RoBERTa, or lighter variants like DistilBERT) generally achieve higher accuracy thanks to richer contextual representations — the same word can carry different meaning depending on sentence context, something GloVe (a static, per-word representation) can't capture.

Understanding Checklist

• [ ] Can explain the difference between word embedding and one-hot encoding

• [ ] Understand why pretrained embeddings (GloVe) are useful, especially for small datasets

• [ ] Can explain why a bidirectional LSTM captures more context than a one-directional LSTM

• [ ] Understand the basic flow of deploying a PyTorch model as a REST API

Quick Quiz

1. What's the main advantage of word embeddings over one-hot encoding? Word embeddings capture semantic relationships between words (similar-meaning words get vectors that are close together), while one-hot encoding treats every word as equally distant from every other word, with no meaning encoded at all.

2. Why use a bidirectional LSTM instead of a one-directional one? Because a word's meaning in a sentence sometimes only becomes clear after seeing the words that follow it. A bidirectional LSTM processes the sentence in both directions at once, so every hidden state carries context from both before and after that point.

3. Why use a pretrained embedding like GloVe instead of training your own from scratch? Because training embeddings that genuinely capture word meaning requires a massive dataset. GloVe was already trained on billions of tokens, so the model gets to build on a mature semantic representation instead of learning meaning from scratch on a much smaller dataset.

Part of the "AI Notes & Engineering" series — my personal AI engineering study notes.

TL;DR: RNNs process data sequentially by "remembering" previous steps through a hidden state. Their main weakness (vanishing gradients) is addressed by more advanced variants: LSTM and GRU. The final section puts this into practice with a name-origin classifier.

Table of Contents

• What is an RNN?

• Key Characteristics of RNNs

• RNN Structure and How It Works

• Limitations of RNNs

• RNN vs LSTM vs GRU

• Training RNNs: BPTT and Its Challenges

• RNN Applications in NLP

• Hands-On: Classifying Name Origins with an RNN

• Understanding Checklist

• Quick Quiz

What is an RNN?

🤔 Guess First: what actually sets an RNN apart from a regular feedforward network?

The answer: **memory**. A feedforward network processes each input independently — there's no "recall" of what came before. An RNN, on the other hand, carries information forward through a *hidden state*, letting it pick up context across a sequence. A Recurrent Neural Network (RNN) is a neural architecture built specifically for sequential data — text, audio, or time series. Unlike a feedforward network, an RNN factors in information from previous steps while processing the current input, which makes it well suited for tasks like speech recognition, natural language processing, and time-series analysis.

Visually, an RNN is often drawn as a single unit with a self-loop, which gets "unfolded" into a chain of units across time steps t-1, t, t+1, and so on.

Key Characteristics of RNNs

1. Short-term memory — the hidden state carries information from previous steps to influence the current decision, capturing dependencies between elements in the sequence.

2. Recurrence — a feedback connection lets information from the neuron at time t-1 influence the neuron at time t.

3. Parameter sharing — the same weights are reused at every time step, which keeps the parameter count (and training cost) far lower than giving each step its own set of weights.

RNN Structure and How It Works

At each time step t, an RNN unit takes two inputs:

• the sequence input at time t (x_t), and

• the hidden state from the previous step (h_{t-1}). The hidden state is then updated with:

h_t = f(W_h · h_{t-1} + W_x · x_t + b)

where h_t is the hidden state at time t, W_h is the weight for the previous hidden state, W_x is the weight for the current input, b is the bias term, and f is an activation function — typically tanh. ReLU is rarely used directly in the recurrent connection, since it's more prone to causing exploding gradients when unrolled across many time steps.

The flow, step by step:

1. Input — the sequence is fed in one element at a time (e.g., word by word in a sentence).

2. Hidden state update — the current input is combined with the previous hidden state via the equation above.

3. Output — the updated hidden state produces an output, either as a prediction for that time step or as input for the next one.

Limitations of RNNs

Despite being well suited to sequential data, vanilla RNNs have two well-known limitations:

• Vanishing gradients — over long sequences, the gradients used to update weights can shrink drastically, making it hard for the network to learn from steps far back in the sequence.

• Difficulty capturing long-term dependencies — a direct consequence of vanishing gradients: vanilla RNNs tend to "forget" information from early steps in long sequences. These two problems are exactly what motivated more advanced RNN variants: LSTM and GRU.

RNN vs LSTM vs GRU

Note: the "training speed" and "long-term dependency handling" rows reflect general tendencies, not absolute rules — actual performance still depends on implementation, data size, and hardware.

• Vanilla RNN — the most basic form, and the foundation for more advanced models.

• LSTM (Long Short-Term Memory) — a modified RNN designed to tackle vanishing gradients using gates (forget, input, output) that regulate the flow of information.

• GRU (Gated Recurrent Unit) — a lighter version of LSTM with only two gates (reset, update), still effective at capturing long-term dependencies but computationally cheaper.

Training RNNs: BPTT and Its Challenges

Training an RNN means optimizing it to predict accurate outputs from given inputs. The overall process:

1. Data collection — relevant and sufficiently large.

2. Preprocessing — normalization, tokenization (for text), splitting into train/validation/test sets.

3. Model initialization — choosing the number of layers, neurons, and activation functions.

4. Loss function selection — matched to the task.

5. Optimizer selection — SGD, Adam, or RMSprop.

6. Backpropagation Through Time (BPTT) — a backpropagation variant that computes gradients across the full sequence, then updates weights based on every time step. Because BPTT is prone to vanishing/exploding gradients, a few complementary techniques are commonly used:

• Gradient clipping — caps gradient magnitude to prevent it from exploding.

• Learning rate tuning — too high destabilizes training; too low slows convergence.

• Regularization (dropout) — reduces overfitting risk in complex models.

• Transfer learning — reusing a model pretrained on a large dataset, then adapting it to a specific context.

• Cross-validation — checks that the model isn't overfitting to the training data.

RNN Applications in NLP

RNNs — especially LSTM and GRU — show up across a wide range of language and sequence tasks:

• Machine translation — processing a sentence in one language and producing an equivalent sentence in another, factoring in context from the whole sequence.

• Sentiment analysis — classifying a piece of text (a product review, a social media comment) as positive, negative, or neutral, based on word order.

• Text generation — learning language patterns from large text datasets to generate new, coherent sentences or paragraphs.

• Virtual assistants and chatbots — understanding conversational sequences to produce responses that fit the ongoing dialogue.

• Speech recognition — converting sequential audio signals into text, using context from preceding sound to improve accuracy.

Note: this session's material focuses on RNNs. Word embeddings (vector representations of words, like Word2Vec/GloVe) are covered separately in "Word Embedding & Sentiment Analysis with LSTM" — paired with a hands-on real-world example.

Modern context: for many NLP tasks today (translation, large-scale text classification, chatbots), Transformer-based architectures (like BERT, GPT) are generally the go-to choice, since they parallelize better and capture long-range dependencies more effectively. RNN/LSTM/GRU remain well worth learning as the foundation for understanding sequential data processing, and are still used for certain cases (lightweight time-series work, resource-constrained devices, and similar).

Hands-On: Classifying Name Origins with an RNN

To make the theory above less abstract, I implemented a vanilla RNN for a concrete task: guessing a name's language of origin, character by character, using exactly the equation covered above — h_t = tanh(W_h · h_{t-1} + W_x · x_t).

The setup:

• Input: 57 unique characters (letters plus a few symbols)

• Hidden size: 128

• Output: 18 language/origin categories (Japanese, Irish, Spanish, and so on)

• Training: 100,000 iterations, BPTT with manual SGD (learning rate 0.005) Verified training results (pulled directly from the run, not estimated):

The loss dropped steadily from ~2.87 to ~0.83 — a sign the model was actually learning, not just guessing.

On names that fall within the training distribution, predictions look reasonable:

But testing it on modern footballer names — clearly outside the original data distribution — the accuracy falls apart:

This isn't a bug — it's a textbook case of distribution shift: the training data is a set of historical surnames grouped by language, not a database of modern public figures' nationalities. It's a good reminder that understanding your training data matters as much as understanding your architecture.

The full notebook (training, confusion matrix evaluation, and inference) is available on GitHub: rnn-name-origin-classifier-arielshakaramiro.

Understanding Checklist

Before moving on to the next post (word embeddings and LSTM in practice), check your understanding:

• [ ] Can explain how an RNN differs from a feedforward neural network

• [ ] Understand the role of the hidden state and why parameters are shared across time steps

• [ ] Can explain why vanishing gradients happen in vanilla RNNs

• [ ] Know the core difference between LSTM and GRU

• [ ] Understand BPTT's role in training RNNs

Quick Quiz

1. What's the main role of the hidden state in an RNN? It carries information from previous time steps to influence how the current input is processed — this is what gives an RNN the ability to "remember" context across a sequence.

2. Why do vanilla RNNs struggle with long-term dependencies? Because of vanishing gradients — over long sequences, gradients propagated backward through BPTT shrink drastically, making it hard to update weights based on early steps in the sequence.

3. What's the key difference between LSTM and GRU? LSTM has three gates (forget, input, output) and is more complex; GRU has only two (reset, update), making it simpler and computationally lighter while still handling long-term dependencies effectively.

Part of the "AI Notes & Engineering" series — my personal AI engineering study notes.

TL;DR: RNN memproses data secara berurutan dengan "mengingat" langkah sebelumnya lewat hidden state. Kelemahan utamanya (vanishing gradient) diatasi oleh varian yang lebih canggih: LSTM dan GRU. Di bagian akhir, konsep ini dipraktikkan langsung untuk klasifikasi asal nama.

Daftar Isi

• Apa itu RNN?

• Karakteristik Utama RNN

• Struktur dan Cara Kerja RNN

• Kelemahan RNN

• RNN vs LSTM vs GRU

• Melatih RNN: BPTT dan Tantangannya

• Aplikasi RNN dalam NLP

• Praktik: Klasifikasi Asal Nama dengan RNN

• Checklist Pemahaman

• Kuis Singkat

Apa itu RNN?

🤔 Coba Tebak Dulu: menurutmu, apa yang membedakan RNN dari neural network biasa (feedforward)?

Jawabannya: **memori**. Neural network biasa memproses setiap input secara independen — tidak ada "ingatan" dari input sebelumnya. RNN, sebaliknya, menyimpan informasi dari langkah waktu sebelumnya lewat *hidden state*, sehingga bisa menangkap konteks dalam data yang berurutan. Recurrent Neural Network (RNN) adalah arsitektur jaringan saraf yang dirancang khusus untuk memproses data berurutan (sequential) seperti teks, audio, atau deret waktu. Bedanya dengan jaringan feedforward, RNN mempertimbangkan informasi dari langkah sebelumnya saat memproses input saat ini — sehingga cocok untuk tugas seperti pengenalan suara, pemrosesan bahasa alami, dan analisis deret waktu.

Secara visual, RNN sering digambarkan sebagai satu unit dengan koneksi loop ke dirinya sendiri, yang kemudian di-unfold menjadi rangkaian unit sepanjang langkah waktu t-1, t, t+1, dan seterusnya.

Karakteristik Utama RNN

1. Memori jangka pendek — RNN menyimpan informasi dari langkah sebelumnya dalam sebuah "memori" (hidden state) untuk mempengaruhi keputusan saat ini, sehingga bisa menangkap dependensi antar-elemen dalam urutan.

2. Pengulangan (recurrence) — ada koneksi umpan balik yang memungkinkan informasi dari neuron di waktu t-1 mempengaruhi neuron di waktu t.

3. Parameter sharing — RNN memakai parameter (bobot) yang sama di setiap langkah waktu, sehingga jumlah parameter yang perlu dilatih jauh lebih sedikit dibanding kalau tiap langkah waktu punya bobot sendiri.

Struktur dan Cara Kerja RNN

Setiap unit RNN menerima dua input pada waktu t:

• input data urutan pada waktu t (x_t), dan

• hidden state dari waktu sebelumnya (h_{t-1}). Hidden state ini kemudian diperbarui melalui persamaan:

h_t = f(W_h · h_{t-1} + W_x · x_t + b)

di mana h_t adalah hidden state pada waktu t, W_h adalah bobot untuk hidden state sebelumnya, W_x adalah bobot untuk input saat ini, b adalah bias, dan f adalah fungsi aktivasi — umumnya tanh. ReLU jarang dipakai langsung di koneksi rekuren karena lebih rawan menyebabkan exploding gradient saat di-unroll sepanjang banyak langkah waktu.

Alur kerjanya secara garis besar:

1. Input data — data urutan diterima satu per satu (misalnya kata demi kata dalam kalimat).

2. Update hidden state — input saat ini digabungkan dengan hidden state sebelumnya lewat persamaan di atas.

3. Output — dari hidden state yang sudah diperbarui, RNN menghasilkan output, baik untuk prediksi di langkah waktu itu maupun untuk diteruskan ke langkah berikutnya.

Kelemahan RNN

Meski cukup powerful untuk data sekuensial, RNN vanilla punya dua kelemahan utama:

• Vanishing gradient — pada urutan yang panjang, gradien yang dipakai untuk update bobot bisa mengecil drastis, sehingga jaringan kesulitan belajar dari langkah-langkah yang jauh di awal urutan.

• Kesulitan menangkap dependensi jangka panjang — akibat langsung dari vanishing gradient, RNN vanilla sering "lupa" informasi dari langkah-langkah awal pada sekuens yang panjang. Dua masalah inilah yang mendorong lahirnya varian RNN yang lebih canggih: LSTM dan GRU.

RNN vs LSTM vs GRU

Catatan: baris "kecepatan training" dan "menangani dependensi jangka panjang" adalah kecenderungan umum, bukan aturan mutlak — performa nyata tetap tergantung implementasi, ukuran data, dan hardware yang dipakai.

• Vanilla RNN — bentuk paling dasar, jadi fondasi model-model yang lebih kompleks.

• LSTM (Long Short-Term Memory) — versi modifikasi RNN yang dirancang khusus untuk mengatasi vanishing gradient dengan mekanisme gate (forget, input, output) yang mengontrol aliran informasi.

• GRU (Gated Recurrent Unit) — versi yang lebih sederhana dari LSTM (hanya reset dan update gate) tapi tetap efektif menangkap dependensi jangka panjang, dengan komputasi yang lebih ringan.

Melatih RNN: BPTT dan Tantangannya

Pelatihan RNN bertujuan mengoptimalkan jaringan agar bisa memprediksi output yang akurat dari input yang diberikan. Prosesnya secara garis besar:

1. Pengumpulan data yang relevan dan cukup besar.

2. Pra-pemrosesan — normalisasi, tokenisasi (untuk teks), pembagian data latih/validasi/uji.

3. Inisialisasi model — menentukan jumlah lapisan, neuron, dan fungsi aktivasi.

4. Pemilihan fungsi loss yang sesuai dengan tugas.

5. Pemilihan optimizer — SGD, Adam, atau RMSprop.

6. Backpropagation Through Time (BPTT) — varian backpropagation yang menghitung gradien dari keseluruhan urutan waktu, lalu memperbarui bobot berdasarkan seluruh langkah waktu tersebut. Karena BPTT rentan terhadap vanishing/exploding gradient, ada beberapa teknik tambahan yang biasa dipakai:

• Gradient clipping — membatasi magnitudo gradien supaya tidak meledak (exploding gradient).

• Pengaturan learning rate — terlalu besar bikin training tidak stabil, terlalu kecil bikin lambat konvergen.

• Regularisasi (dropout) — mengurangi risiko overfitting pada model yang kompleks.

• Transfer learning — memakai model yang sudah dilatih pada dataset besar, lalu disesuaikan ke konteks spesifik.

• Cross-validation — memastikan model tidak overfit terhadap data latih.

Aplikasi RNN dalam NLP

RNN (terutama LSTM dan GRU) dipakai luas dalam berbagai tugas pemrosesan bahasa dan urutan:

• Penerjemahan bahasa — memproses kalimat dalam satu bahasa dan menghasilkan kalimat setara di bahasa lain, dengan mempertimbangkan konteks seluruh urutan.

• Analisis sentimen — menentukan apakah sebuah teks (ulasan produk, komentar media sosial) bernada positif, negatif, atau netral, dengan memperhatikan urutan kata.

• Penghasilan teks (text generation) — mempelajari pola bahasa dari dataset besar untuk menghasilkan kalimat atau paragraf baru yang koheren.

• Asisten virtual dan chatbot — memahami urutan percakapan untuk memberi respons yang relevan dengan konteks dialog.

• Speech recognition — mengubah sinyal audio berurutan menjadi teks, dengan mempertimbangkan konteks suara sebelumnya.

Catatan: materi sesi ini fokus pada RNN. Pembahasan word embedding (representasi kata sebagai vektor, seperti Word2Vec/GloVe) saya bahas terpisah di tulisan "Word Embedding & Sentiment Analysis dengan LSTM" — sekalian dipraktikkan ke kasus nyata.

Konteks modern: untuk banyak tugas NLP saat ini (terjemahan, klasifikasi teks skala besar, chatbot), arsitektur berbasis Transformer (seperti BERT, GPT) umumnya jadi pilihan utama karena lebih mudah diparalelkan dan lebih baik menangkap dependensi jarak jauh. RNN/LSTM/GRU di tulisan ini tetap relevan dipelajari sebagai fondasi untuk memahami pemrosesan data sekuensial, dan masih dipakai untuk kasus-kasus tertentu (misalnya time-series ringan atau perangkat dengan resource terbatas).

Praktik: Klasifikasi Asal Nama dengan RNN

Biar teori di atas nggak cuma di kepala, saya coba langsung implementasi vanilla RNN untuk kasus nyata: menebak asal bahasa dari sebuah nama, huruf demi huruf, persis pakai persamaan h_t = tanh(W_h · h_{t-1} + W_x · x_t) yang sudah dibahas.

Setup singkatnya:

• Input: 57 karakter unik (huruf + beberapa simbol)

• Hidden size: 128

• Output: 18 kategori bahasa/asal (Japanese, Irish, Spanish, dst.)

• Training: 100.000 iterasi, BPTT + manual SGD (learning rate 0.005) Hasil training (diverifikasi langsung dari output, bukan estimasi):

Loss turun stabil dari ~2.87 ke ~0.83 — tanda modelnya memang belajar, bukan cuma nebak acak.

Pas dites ke nama-nama di dalam distribusi dataset, hasilnya masuk akal:

Tapi begitu saya coba nama-nama pesepakbola modern — yang jelas di luar distribusi dataset aslinya — hasilnya jadi jauh dari akurat:

Ini bukan bug — ini contoh nyata distribution shift: dataset training berisi nama-nama historis per kategori bahasa, bukan basis data kewarganegaraan tokoh publik modern. Justru dari sinilah kelihatan kenapa memahami data yang dipakai untuk melatih model itu sama pentingnya dengan memahami arsitekturnya.

Notebook lengkap (training, evaluasi confusion matrix, sampai inference) tersedia di GitHub: rnn-name-origin-classifier-arielshakaramiro.

Checklist Pemahaman

Sebelum lanjut ke topik berikutnya (praktik word embedding & LSTM di kasus nyata), coba cek pemahamanmu:

• [ ] Bisa menjelaskan bedanya RNN dengan feedforward neural network

• [ ] Paham fungsi hidden state dan kenapa parameter di-share antar langkah waktu

• [ ] Bisa menjelaskan kenapa vanishing gradient terjadi di RNN vanilla

• [ ] Tahu perbedaan mendasar antara LSTM dan GRU

• [ ] Paham peran BPTT dalam melatih RNN

Kuis Singkat

1. Apa fungsi utama hidden state dalam RNN? Menyimpan informasi dari langkah waktu sebelumnya agar bisa mempengaruhi pemrosesan input saat ini — inilah yang memberi RNN kemampuan "mengingat" konteks dalam urutan.

2. Kenapa RNN vanilla kesulitan menangkap dependensi jangka panjang? Karena masalah vanishing gradient — saat urutan panjang, gradien yang dipropagasi mundur (lewat BPTT) mengecil drastis, membuat bobot sulit diperbarui berdasarkan langkah-langkah awal urutan.

3. Apa perbedaan utama LSTM dan GRU? LSTM punya tiga gate (forget, input, output) dan lebih kompleks; GRU hanya punya dua gate (reset, update), lebih sederhana dan lebih ringan secara komputasi, namun tetap efektif menangkap dependensi jangka panjang.

Bagian dari seri "AI Notes & Engineering" — catatan belajar AI Engineering saya.

The model scored 100% accuracy. But that number turned out to be less straightforward than it looks.

The dataset used contains 94 Indonesian sentences with 3 labels: greeting, siapa_anda (who are you), sekarang_jam_berapa (what time is it) — bootcamp practice material, not production data.

How It Works

1. Preprocessing — lowercase, strip punctuation, stem with Sastrawi (same as the earlier FAQ search engine project).

2. Training — TfidfVectorizer + LogisticRegression combined in a single sklearn.Pipeline.

3. Evaluation — accuracy and classification report on the test set.

100% Accuracy — Coincidence or Real?

The first split (80% train, 20% test) produced 100% accuracy on 19 test samples. A number that good is easy to get excited about, but also easy to be misled by — 19 samples is small, and a result this clean could just be luck.

To check, I ran 5-fold stratified cross-validation — training and testing the model 5 times with different data splits:

Scores per fold: [1. 1. 1. 1. 1.]
Mean: 1.0000 (std: 0.0000)

Still 100% across every fold. So this isn't a fluke from one lucky split. But it's also not proof the technique itself is remarkably powerful — a more plausible explanation: the three intents in this dataset are lexically very distinct. Sentences about time ("jam", "sekarang", "pukul") almost never share words with greetings ("halo", "selamat") or identity questions ("siapa", "kamu"). TF-IDF + Logistic Regression will always look "perfect" when the classes are already this far apart in vocabulary — it's not a representative benchmark for intents that are more similar to each other.

Testing the Real Limits: Cases Outside the Training Data

To actually find the model's limits, I tried sentences that don't appear in the training data at all — slang, ambiguous phrasing, and completely off-topic sentences:

Two interesting findings here:

1. "kamu tau ga sekarang tanggal berapa" (asking about the date, not the time) still gets classified as sekarang_jam_berapa. The model doesn't actually distinguish the concepts of "date" and "time" — it just recognizes the word pattern "sekarang" + "berapa" (now + how much/what), which happens to resemble the time-intent pattern from training.

2. "random kalimat tidak jelas maksudnya apa" (a sentence with no clear topic at all) still gets forced into one of the 3 classes (greeting, with a confidence of just 0.439 — barely above a random guess among 3 classes). This model has no "unknown" class, so no matter the input, it will always return one of the 3 existing labels.

This mirrors the Euclidean distance bug from the earlier FAQ search engine note: a system built without considering out-of-scope input will always force an answer rather than admit it doesn't know. The mechanism is different — there it was a distance threshold, here it's the lack of a rejection class — but the lesson is the same.

Summary

Main takeaway: high accuracy on a small dataset with lexically distinct classes doesn't automatically mean the model is reliable for real-world cases that are more ambiguous or fall outside the training data's scope.

The full code and re-executed notebook are available in this GitHub repo.

Part of my AI Engineering study notes.

Modelnya dapat akurasi 100%. Tapi angka itu ternyata tidak sesederhana kelihatannya.

Dataset yang dipakai berisi 94 kalimat Bahasa Indonesia dengan 3 label: greeting, siapa_anda, sekarang_jam_berapa — dari materi latihan bootcamp, bukan data produksi.

Cara Kerja

1. Preprocessing — lowercase, hapus tanda baca, stemming pakai Sastrawi (sama seperti proyek FAQ search engine sebelumnya).

2. Training — TfidfVectorizer + LogisticRegression digabung dalam satu sklearn.Pipeline.

3. Evaluasi — akurasi dan classification report di test set.

Akurasi 100% — Kebetulan atau Nyata?

Split pertama (80% train, 20% test) menghasilkan akurasi 100% di 19 sampel test. Angka sebagus ini gampang bikin senang, tapi juga gampang menyesatkan — 19 sampel itu kecil, hasil sebagus ini bisa saja kebetulan.

Untuk memastikan, saya jalankan 5-fold stratified cross-validation — melatih dan menguji model 5 kali dengan pembagian data yang berbeda-beda:

Skor tiap fold: [1. 1. 1. 1. 1.]
Rata-rata: 1.0000 (std: 0.0000)

Tetap 100% di semua fold. Jadi ini bukan kebetulan hasil satu split yang beruntung. Tapi ini juga bukan bukti tekniknya luar biasa canggih — penjelasan yang lebih masuk akal: ketiga intent di dataset ini sangat berbeda secara leksikal. Kalimat soal waktu ("jam", "sekarang", "pukul") nyaris tidak pernah beririsan kata dengan kalimat sapaan ("halo", "selamat") atau kalimat identitas ("siapa", "kamu"). TF-IDF + Logistic Regression akan selalu terlihat "sempurna" kalau kelas-kelasnya sudah terpisah jauh secara kata — ini bukan ukuran yang representatif untuk kasus intent yang lebih mirip satu sama lain.

Menguji Batas Sebenarnya: Kasus di Luar Data Training

Untuk benar-benar tahu batas kemampuan model, saya coba kalimat yang tidak ada di data training sama sekali — termasuk slang, frasa ambigu, dan kalimat di luar topik:

Dua hal menarik dari sini:

1. "kamu tau ga sekarang tanggal berapa" — ini nanya soal tanggal, bukan jam — tapi model tetap mengklasifikasikannya sebagai sekarang_jam_berapa. Model ini tidak benar-benar membedakan konsep "tanggal" dan "jam"; dia cuma mengenali pola kata "sekarang" + "berapa" yang kebetulan mirip pola intent waktu di data training.

2. "random kalimat tidak jelas maksudnya apa" — kalimat ini di luar topik apa pun, tapi tetap dipaksa masuk ke salah satu dari 3 kelas (greeting, confidence cuma 0,439 — nyaris tebakan acak untuk 3 kelas). Model ini tidak punya kelas "tidak diketahui", jadi apa pun inputnya, dia akan selalu memberi salah satu dari 3 label yang ada.

Ini pola yang sama seperti bug Euclidean distance di catatan sebelumnya tentang FAQ search engine: sistem yang dibangun tanpa mempertimbangkan input di luar cakupan akan selalu "memaksakan" jawaban, bukan mengakui ketidaktahuannya. Bedanya cuma mekanismenya — di situ soal ambang batas jarak, di sini soal tidak adanya kelas penolakan.

Ringkasan

Pelajaran utama: akurasi tinggi di dataset kecil dengan kelas yang sangat berbeda secara leksikal tidak otomatis berarti model-nya andal untuk kasus dunia nyata yang lebih ambigu atau di luar cakupan data training.

Kode lengkap dan notebook yang sudah dijalankan ulang bisa dicek di repo GitHub ini.

Bagian dari catatan belajar AI Engineering saya.

Sounds simple. But once I started testing it with odd, off-topic questions, I found a bug that turned out to matter quite a bit for how "similarity" gets measured.

The FAQ dataset used here is entirely fictional — the store name and every Q&A pair are just examples for demonstrating the technique, not real business data.

How It Works

1. Preprocessing — every question and answer is lowercased, stripped of punctuation, and each word is stemmed down to its root form using Sastrawi (an Indonesian stemming library).

2. Vectorization — all the preprocessed text is converted into TF-IDF vectors.

3. Search — when a user types a question, the system computes a similarity score between the user's question and every question in the database, then returns the answer from whichever is most similar.

The Bug I Found: A Fragile Distance Threshold

The first version of this system measured "similarity" using Euclidean distance (the straight-line distance between two vector points) with a fixed threshold: if the distance was below 1.0, it counted as a match.

I threw four test questions at it — three relevant, and one deliberately unrelated:

Look at the last row. A question about a fried rice recipe — clearly unrelated to a furniture FAQ — produced a Euclidean distance of exactly 1.0, sitting precisely on the threshold line. Not "almost" — genuinely right at the boundary. If the comparison had come out even marginally different, the system would have confidently treated it as a "match" and returned an irrelevant answer.

Compare that with cosine similarity (which measures the angle between two vectors, not their absolute distance): for the same question, the result is 0.0000 — a far clearer signal that there's genuinely no match at all.

Why does this happen? Euclidean distance is sensitive to the length/density of text vectors, so two texts that are both "short and sparse" can end up looking close together in distance even when their content has nothing to do with each other. Cosine similarity doesn't have this problem, since it only measures the angle between vectors.

The Fix I Applied

• Switched the default metric from Euclidean distance to cosine similarity, using a minimum similarity threshold (0.15) instead of a distance threshold.

• Made sure the user's question goes through the exact same preprocessing function as the training data — in the earlier version, the user's question was vectorized directly with no preprocessing at all, even though every FAQ entry had already been lowercased and stemmed.

Honest note: that 0.15 threshold was picked manually based on these 4 test questions, not from systematic tuning across many phrasings. For "good fried rice recipe" specifically, the rejection was clean (similarity score of exactly zero) — but that's not a guarantee every off-topic question will be rejected this cleanly. A real production system would need testing against a much wider range of questions before relying on a single fixed threshold.

Try It Yourself

git clone https://github.com/arielshakaramiro/faq-search-engine-tfidf-sastrawi-arielshakaramiro.git
cd faq-search-engine-tfidf-sastrawi-arielshakaramiro
pip install -r requirements.txt
python train.py
python app.py

The full code, exploration notebook, and further audit details are available in this GitHub repo.

If you haven't read the TF-IDF fundamentals this project builds on, there's a previous note covering Bag of Words, TF-IDF, and Word Embedding from the ground up. Another way of putting TF-IDF to work — intent classification with Logistic Regression — is covered in a separate note.

https://shaka-ai.hashnode.dev/text-vectorizer-bow-tfidf-word-embedding-explained
https://shaka-ai.hashnode.dev/intent-classification-tfidf-logistic-regression

Part of my AI Engineering study notes.

Kedengarannya sederhana. Tapi begitu mulai diuji dengan pertanyaan aneh-aneh, saya menemukan satu bug yang cukup penting soal cara mengukur "kemiripan" antar teks.

Dataset FAQ yang dipakai di sini sepenuhnya fiktif — nama toko dan seluruh isi tanya-jawabnya cuma contoh untuk demonstrasi teknik, bukan data bisnis sungguhan.

Cara Kerjanya

1. Preprocessing — setiap pertanyaan dan jawaban di-lowercase, tanda bacanya dihapus, lalu tiap kata di-stem ke bentuk dasarnya pakai Sastrawi (library stemming Bahasa Indonesia).

2. Vektorisasi — seluruh teks yang sudah diproses diubah jadi vektor TF-IDF.

3. Pencarian — saat user mengetik pertanyaan, sistem menghitung skor kemiripan antara pertanyaan user dan setiap pertanyaan di database, lalu mengembalikan jawaban dari yang paling mirip.

Bug yang Ditemukan: Threshold Jarak yang Rapuh

Versi awal sistem ini mengukur "kemiripan" pakai Euclidean distance (jarak garis lurus antar dua titik vektor) dengan ambang batas tetap: kalau jaraknya di bawah 1.0, dianggap cocok.

Saya coba lempar empat pertanyaan uji — tiga yang relevan, satu yang sengaja sama sekali tidak nyambung:

Lihat baris terakhir. Pertanyaan soal resep nasi goreng — yang jelas tidak ada hubungannya dengan FAQ furnitur — menghasilkan Euclidean distance persis 1,0, tepat di angka ambang batasnya. Bukan "nyaris", tapi benar-benar pas di garis batas — kalau saja formula pembandingnya sedikit berbeda, sistem bisa saja menganggapnya "cocok" dan tetap memberi jawaban asal-asalan.

Bandingkan dengan cosine similarity (mengukur kemiripan arah dua vektor, bukan jarak absolutnya): untuk pertanyaan yang sama, hasilnya 0,0000 — sinyal yang jauh lebih tegas bahwa memang tidak ada kecocokan sama sekali.

Kenapa ini terjadi? Euclidean distance sensitif terhadap panjang/kepadatan vektor teks, sehingga dua teks yang sama-sama "pendek dan jarang" bisa kelihatan berdekatan secara jarak, padahal isinya sama sekali tidak berhubungan. Cosine similarity tidak punya masalah ini karena yang diukur cuma sudut antar vektor.

Perbaikan yang Saya Terapkan

• Mengganti metrik default dari Euclidean distance ke cosine similarity, dengan ambang batas kemiripan minimum (0,15) alih-alih ambang batas jarak.

• Memastikan pertanyaan dari user diproses dengan fungsi preprocessing yang sama persis seperti data training — di versi sebelumnya, pertanyaan user langsung divektorisasi tanpa preprocessing, padahal seluruh data FAQ sudah melalui proses lowercase + stemming.

Catatan jujur: angka ambang batas 0,15 di atas saya pilih manual berdasarkan 4 pertanyaan uji ini, bukan hasil tuning sistematis di banyak variasi kalimat. Untuk contoh "resep nasi goreng enak" di atas, hasilnya memang tegas (skor kemiripannya nol) — tapi ini bukan jaminan semua pertanyaan di luar topik akan selalu ditolak sebaik ini. Sistem produksi sungguhan sebaiknya diuji dengan variasi pertanyaan yang jauh lebih banyak sebelum mengandalkan satu angka ambang batas tetap.

Coba Sendiri

git clone https://github.com/arielshakaramiro/faq-search-engine-tfidf-sastrawi-arielshakaramiro.git
cd faq-search-engine-tfidf-sastrawi-arielshakaramiro
pip install -r requirements.txt
python train.py
python app.py

Kode lengkap, notebook eksplorasi, dan detail audit lainnya ada di repo GitHub ini.

Kalau kamu belum baca dasar-dasar TF-IDF yang dipakai di proyek ini, ada catatan sebelumnya yang membahas Bag of Words, TF-IDF, dan Word Embedding dari awal. Pendekatan lain untuk memanfaatkan TF-IDF — klasifikasi intent dengan Logistic Regression — dibahas di catatan terpisah.

https://shaka-ai.hashnode.dev/text-vectorizer-bow-tfidf-word-embedding-dijelaskan
https://shaka-ai.hashnode.dev/klasifikasi-intent-tfidf-logistic-regression

Bagian dari catatan belajar AI Engineering saya.