763: Web Scraping + Reverse Engineering APIs

Episode Summary

Executive Summary: The episode is a practical deep dive into web scraping and reverse engineering APIs, explaining when to scrape, how to choose between server-side parsing, private APIs, headless browsers, and client-side scripts, and how to handle auth, protected routes, and DOM selection. Wes and Scott also share real-world scraping projects, ethical/legal caveats, and tooling recommendations.

Main Topics: Why scrape instead of using APIs: The hosts explain that many websites lack public APIs, restrict access, or charge for API usage, making scraping a necessary alternative for obtaining data at scale. Types of scraping approaches: They break scraping into client-side scripts, server-side fetch + DOM parsing, private API replication, server-rendered HTML extraction, and headless browser automation. Reverse engineering private and protected APIs: The episode covers using DevTools or Proxyman to inspect network traffic, copy requests, identify needed headers/cookies, and reconstruct calls to hidden endpoints. Selecting data from messy markup: They discuss practical selector strategies including ARIA labels, test IDs, XPath, and initial-state JSON blobs, especially for sites like Twitter, Instagram, and Facebook. Tools and runtimes for scraping: Recommended tools include Node, Deno, Bun, Proxyman, Linked DOM, Playwright/Puppeteer/Cypress, and fetch-cookie, along with notes about Bun’s convenient file APIs. Ethics, legality, and anti-bot defenses: The hosts acknowledge scraping as a legal gray area and discuss CAPTCHAs, bot detection, API restrictions, and why some sites aggressively limit access due to AI training and abuse. Personal scraping use cases: They share examples like stock alerts, vaccine notifications, marketplace monitoring, social stats tracking, CloudApp export, bank transaction logging, and exercise-machine data extraction.

Key Arguments: Scraping is often the only practical way to obtain data when no public API exists or when an API is restricted, expensive, or unstable. The best scraping path is usually server-side JavaScript with direct network requests; browser automation should be a fallback when the page is client-rendered or JS-dependent. Inspecting a site’s own network requests is the fastest way to discover private APIs and the minimum headers/cookies required to authenticate. Many sites embed useful data in server-rendered HTML or initial-state JSON, which can be extracted without full browser automation. Accessibility attributes, test IDs, and XPath are valuable selector strategies when class names are obfuscated or unstable. Scrapers frequently break because websites change markup or internal APIs, so maintainability and adaptability are essential. AI can help parse HTML into structured arrays quickly, but human review is still needed to ensure correctness and robustness. Scraping teaches how modern web apps work, including authentication flows, client/server rendering, anti-bot tactics, and data access patterns.

Data Points: Covid vaccine notifier polling interval: Every half hour - Wes describes a scraper that pinged about 100 pharmacies across six stores to find vaccine availability. Pharmacy sources checked: ~100 pharmacies - Used for the COVID vaccine notifier project. Stores queried: 6 stores - Part of the COVID vaccine notifier example. Marketplace scraping cadence: Every 5 minutes - Used in a PlayStation 5 stock notifier scraper. CloudApp account history: 13 years - Wes said he had 13 years of data on the screenshot/file hosting service. CloudApp file count: 12,000 files - The amount of content Wes exported via reverse-engineered API calls. Export size: 300-gig folder - Approximate total download of all CloudApp drops/files. Screenshot directory size: 1,000 items - Scott noted his screenshot folder had accumulated many files. Screenshot directory size on disk: 100 megabytes - Scott’s screenshot folder storage footprint. Clearance threshold: Over 90% off - Canadian Tire scraper logged items meeting this discount threshold. Cookie lifetime example: About an hour - Wes described a bank login cookie that expired quickly, requiring reauthentication. Social/video ranking context: First 24 hours - They discussed tracking podcast performance against other episodes in the early hours after release.

Pivotal Quotes: "we've got a show for you on web scraping and reverse engineering APIs" — Wes: Opening framing of the episode’s topic. "scraping is here to give you access to all of the data on the The World Wide Web for the most part" — Wes: Core justification for why people scrape websites. "you learn a lot about how websites try to stop mischievous actors from running" — Wes: Explaining the educational value of building scrapers.

Implications: For developers, scraping remains a vital skill for data access, automation, and debugging. The episode suggests the best practice is to prefer direct network calls, preserve auth carefully, and expect brittle selectors and changing markup.