Semi-supervised and Self-supervised Learning
The agency wants a fake-listing detector. An expert has checked 200 listings by hand and marked each fake or genuine; that took a week. Meanwhile 20,000 unchecked listings sit in the database. Labels are expensive and unlabelled data is cheap, and two families of methods are built to exploit that gap.
The rule
Few labels? Borrow structure from unlabelled data, or make labels from the data itself.
Semi-supervised learning uses a small labelled set together with a large unlabelled one. It relies on one assumption: items that are close together, or in the same dense cluster, probably share a label. A common method is pseudo-labelling:
- Train a model on the 200 labelled listings.
- Predict on the 20,000 unlabelled ones.
- Keep only very confident predictions (say above 0.95) as new "pseudo-labels".
- Retrain on the labelled plus pseudo-labelled data, and repeat.
The risk: a confident mistake becomes a training label, and the model learns its own error more firmly. A high threshold and a check on the held-out labelled set keep this in check.
Self-supervised learning needs no human labels at all for its first stage. It invents a task whose answers come from the data itself:
- Hide a word in a listing description and predict it: "Sunny two-room flat near the ___ station."
- Predict the next word of a sentence.
- Blank out a patch of a flat photo and reconstruct it, or check that two crops of the same photo match.
To solve such tasks well, the model must learn a lot about language or images. That learned knowledge is then fine-tuned on the small labelled task, the 200 fake-or-genuine examples. Large language models are pretrained this way, which is why they could learn from enormous amounts of text nobody labelled.
The shortcut
Ask "Who made the labels?"
| Labels made by | Type |
|---|---|
| A person, for every example | Supervised |
| A person, for a few, with many unlabelled examples also used | Semi-supervised |
| The data itself, by hiding and predicting parts | Self-supervised |
| Nobody, and none are needed | Unsupervised |
Train on the few
Fit a detector on the 200 hand-labelled listings.
Drills
Answer fast — trust the rule.
A model learns by predicting a hidden word in a million listing descriptions, with no human labels. What kind of learning is this?
A detector is trained on 300 hand-labelled listings together with 30,000 unlabelled ones. What kind of learning is this?
With a 0.95 threshold, predictions are 0.99 fake, 0.60 fake, 0.97 genuine and 0.80 genuine. How many become pseudo-labels?
What is the main risk of pseudo-labelling?
Remember
- Few labels? Borrow structure from unlabelled data, or make labels from the data itself.
- Shortcut: ask who made the labels.
- Trap: trusting pseudo-labels without a strict threshold and a held-out check.