Is Web Scraping Legal? A Practitioner's Guide (2026)
Cover Image
Every scraping project starts with the same question, usually asked right after the code works: is this legal? The honest answer is unsatisfying — it depends on what you scrape, how you access it, and what you do with it. But "it depends" is not a plan, so this guide turns the legal landscape into something you can actually use: what courts decided, the five risk factors you control, and a pre-scrape checklist.
Not legal advice. I am a software engineer, not a lawyer. Everything below is public information organized into a decision framework. For anything with real money or real risk behind it, talk to counsel in your jurisdiction. What follows keeps you out of the obvious ditches; counsel keeps you out of the subtle ones.
The short version for the impatient: scraping publicly accessible data without bypassing authentication survived its biggest court tests in the US. Everything around that sentence — terms of service, personal data, burden on targets, cease-and-desist letters — is where risk actually lives. Our anti-bot playbook promised this guide as its sequel; here it is.
What Courts Actually Said (hiQ and Van Buren)
Two cases shape every current discussion. Know them well enough to reason, not well enough to cite in court.
hiQ Labs v. LinkedIn is the landmark. hiQ scraped publicly available LinkedIn member profiles; LinkedIn sent cease-and-desist and blocked them technically. The Ninth Circuit ruled hiQ could continue scraping public profiles, and after the Supreme Court remanded the case in light of Van Buren (below), the Ninth Circuit affirmed its stance again. But the story did not end in triumph: the district court later found hiQ had breached LinkedIn's User Agreement, and the parties settled. Takeaway: beating the hacking charge is not beating every charge. Public access defeats the unauthorized-access argument; it does not erase the contract you clicked through.
Van Buren v. United States narrowed the Computer Fraud and Abuse Act. The Supreme Court held that "exceeds authorized access" means entering off-limits areas of a system — not using properly accessed data for improper purposes. For scrapers, this matters enormously: viewing public pages the server freely serves is not "exceeding access" in the CFAA sense. Using valid credentials to pull data you are allowed to see, then repurposing it, is (under this reading) not a hacking crime either — though it may still breach contracts or other laws.
Together: the CFAA-as-anti-scraping-hammer is much weaker than it looked a decade ago. The remaining teeth are contracts (ToS), trespass-adjacent theories (burdening servers), and data-protection law (personal data). Those are the five factors below.
Five Risk Factors You Actually Control
Forget "legal vs illegal" as a binary. Think risk surface — five dials, all in your hands:
Authentication bypass. Scraping public pages is one universe; circumventing logins, paywalls, or access controls is another. The moment you defeat a technical barrier to reach non-public data, Van Buren's protection thins fast. Rule: scrape what an anonymous visitor can see, or have explicit permission for the rest.
Terms-of-service breach. hiQ won the CFAA fight and still lost on the User Agreement. "No scraping" clauses are enforceable contracts in many jurisdictions once you have notice (account creation, continued use after reading). Factor ToS into risk; do not pretend unread terms do not exist.
Personal data. Names, emails, phone numbers, photos — scraping personal data triggers data-protection law (GDPR in Europe, CCPA/CPRA in California, equivalents spreading). Our Maps lead guide shows the compliant pattern: collect only what you need, enrich from allowed sources, honor deletion, minimize storage. Bulk personal-data scraping without a lawful basis is the fastest route from "grey area" to "enforcement action."
Burden and harm. Aggressive crawling that degrades a target's service invites trespass-to-chattels claims — the oldest scraping cause of action and still alive. Rate-limit yourself, crawl off-peak, cache aggressively. Being a polite guest is legal strategy, not just manners.
Ignoring cease-and-desist. A C&D letter changes the game: continued scraping after explicit revocation of permission strengthens every claim against you (hiQ kept going under a court order protecting them — you will not have one). Take C&Ds to a lawyer immediately; do not "wait and see."
⚠️ Warning: These factors compound. Public data + polite rate + no personal data = low risk. Authenticated data + ToS breach + personal data + ignored C&D = the fact pattern lawsuits are made of. Audit all five together, never one in isolation.
robots.txt, ToS, and Rate Respect (What Each Practically Means)
Three instruments practitioners constantly misunderstand. Plain versions:
robots.txt is a voluntary standard, not a statute. Courts have treated ignoring it as evidence of bad faith rather than a standalone violation — it colors intent. Respect it by default; document your reason the rare times you deviate (e.g., stale file blocking clearly public content, with polite rates).
Terms of Service are contracts with uneven enforceability. Browsewrap (a link buried in a footer) is weaker than clickwrap (an "I agree" you clicked). Having an account strengthens enforceability — anonymous scraping of ToS-forbidding sites sits in a greyer zone than logged-in scraping of the same. Read the ToS of every serious target; it takes five minutes and informs the whole risk call.
Rate respect is the cheapest insurance available. Crawl-delay directives, exponential backoff on 429s, off-peak scheduling, request caching — all standard practice from our scraping guides, all doubling as evidence of good faith if anything is ever questioned.
Personal Data Rule (GDPR in One Paragraph)
If you scrape EU residents' personal data, GDPR applies regardless of where you sit. You need a lawful basis (legitimate interest is the usual claim — documented, balanced, and deletable on request), data minimization (take fields you use, not everything renderable), and honest retention (delete when the purpose ends). California's regime rhymes. For lead-gen scraping specifically, follow the Maps-post pattern: narrow targeting, enrichment from allowed sources, dedupe, opt-out handling, and outreach that identifies you. Bulk-emailing scraped addresses with no basis or opt-out is where regulators actually bite.
Pre-Scrape Checklist (Copy-Paste)
Run this before every new target:
[ ] Pages publicly accessible without login or paywall bypass?
[ ] ToS read — scraping clause found and risk accepted or cleared?
[ ] robots.txt checked — disallows noted, deviation (if any) documented?
[ ] Personal data minimized — only fields with a use, deletion plan set?
[ ] Rate plan — delays, backoff, caching, off-peak where possible?
[ ] No prior C&D or access revocation from this target?
[ ] Purpose documented — what, why, how long, who sees it?
[ ] Lawyer consulted if any box above is red?
💡 Tip: Save the completed checklist per target with a date. A paper trail of good-faith diligence is the single most useful artifact if a dispute ever arises — it converts "reckless scraper" into "careful practitioner who made a judgment call."
When to Call a Lawyer (Not a Blog Post)
Call counsel when: the target sent any legal communication; you need authenticated or paywalled data at scale; personal data is central (not incidental) to the project; you operate across EU/US/UK regimes simultaneously; or revenue depends on one scraping source that could vanish with a ruling. A one-hour consultation before building beats a litigation hold after launching — price it into every serious scraping project the way you price proxies.
Scraping law rewards the careful and punishes the brazen. Public data, polite rates, clean contracts reading, minimal personal data, documented decisions — that combination has survived every major test so far. Build on it, and keep counsel's number next to your proxy dashboard.
