Web Scraping
A new café opened two streets away, and Nisha wants to compare their prices with hers every week. Their menu is on their website, but there is no download and no API. Reading data out of a web page with a program is called web scraping. It works, but it is the most fragile and most carefully governed way to collect data.
The rule
Scrape last, scrape politely, and expect it to break.
A web page is built from HTML, text wrapped in tags. A scraper downloads that HTML and picks out the parts it needs by their tags and classes:
<div class="menu-item">
<span class="name">Cold coffee</span>
<span class="price">₹130</span>
</div>A scraper here would find every menu-item, then read its name and price, giving a table like Cold coffee, 130.
Scrape last. Look for a better door first: an official API, a downloadable file, a public dataset, or simply asking the owner.
Scrape politely.
- Read the site's terms of use. Some forbid automated collection.
- Check robots.txt (for example
rival-cafe.example/robots.txt), where sites list paths they do not want bots to visit. - Go slowly: a few requests spaced apart, not hundreds per second.
- Take only what you need. Never collect personal data such as reviewers' names without a lawful reason.
Expect it to break. Pages are designed for people and change without warning. If the site renames price to cost, your scraper silently returns nothing. Pages that build their content with JavaScript may send almost empty HTML, which needs heavier tools that run a browser.
The shortcut
Before scraping anything, walk a four-step gate: API? Allowed? Gentle? Monitored? Only scrape if you can say yes to the last three after "no" to the first.
API?
No API and no download. Scraping is the remaining option.
The trap is a scraper that "works" until it quietly does not. Always check that each run returned a sensible number of rows.
Drills
Answer fast — trust the rule.
The rival café offers a downloadable price list. What should you do?
What does a site's robots.txt tell you?
Last week the scraper found 24 menu items; this week it found 0 and raised no error. What most likely happened?
Which scraping plan is the most responsible?
Remember
- Scrape last, scrape politely, and expect it to break.
- Shortcut: API? Allowed? Gentle? Monitored?
- Trap: a scraper that silently returns nothing; check the row count every run.