Case study · open source
magpie: an open-source scraper that survives the open web
Our own scraping engine, built in the open. Proxy pools, real-browser escalation and 600+ tests, so client missions don't break on week two.
Websites are built to keep robots out
Modern sites hide data behind bot walls, rotate their markup and throttle anyone who asks too fast. A scraper that works on Monday is dead by Friday, and one blocked IP can burn a whole project. DIY scripts, freelancer prototypes and bought datasets all fail the same way: quietly, a week after delivery.
So before selling a single mission, we built the tool we needed ourselves, in the open, with the boring discipline most scrapers skip.
Assume the site will fight back
magpie rotates through a proxy pool with automatic failover, so one blocked exit never stops a run. When a challenge shows up, it escalates to real browser profiles, six of them, each with its own fingerprint, instead of hammering harder. Request pacing stays under thresholds, runs are batched and diffed so you see exactly what changed, and every capability is also exposed as MCP tools, so AI agents can drive it.
Everything is covered by 600+ automated tests and ships as a single static Go binary. No runtime dependencies, nothing to install on your side.
The engine behind every mission
magpie runs every Motherlode client job. When a target fights back, the fight is already handled, which is how we can promise 72-hour delivery with sample rows the same day.
And because it's open source, you can read every line before you trust it with your data. Want the same engine pointed at your target? That's literally our job.
Why it exists
"We didn't set out to build a product. We set out to stop rewriting the same scraper every three weeks. magpie is what that refusal turned into."
Ready to see your data?
Send the site and the fields. You'll have a fixed quote within 24 hours. No commitment, no pitch deck.