2  Ethics, Law, and Responsible Data Collection

TipLearning Objectives
  • Explain the legal frameworks governing web data access, including the CFAA, GDPR, and platform Terms of Service
  • Retrieve and interpret a website’s robots.txt file to determine what is permitted for automated access
  • Configure responsible HTTP request headers including custom User-Agent strings with contact information
  • Apply a structured ethical decision framework before collecting web data
  • Implement rate limiting with time.sleep() to avoid overloading servers
TipCompanion Notebook

Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).

2.1 The Ethics of Retrieval

The phrase “web scraping” is colloquial and popular but has pejorative connotations. Data is valuable: other people invested time, money, and effort to create and host the content you are about to retrieve programmatically. Before writing a single line of scraping code, you need to understand the legal, normative, and ethical landscape that governs your work.

This chapter establishes a decision framework that you should revisit before every data collection project in this book and beyond. The question is never simply “can I scrape this?” but rather “should I, and if so, how do I do it responsibly?”

2.3 Technical Norms: robots.txt

Beyond legal requirements, the web has developed a set of community norms for responsible automated access. The most important of these is the robots.txt protocol.

A robots.txt file lives at the root of a website (e.g., https://example.com/robots.txt) and specifies which parts of the site automated agents are permitted or forbidden from accessing. It uses a simple syntax:

import requests

# Retrieve robots.txt for Wikipedia
response = requests.get("https://en.wikipedia.org/robots.txt")
print(response.text[:500])
# The generic rules (as of mid-2026) include directives like:
# User-agent: *
# Allow: /w/api.php?action=mobileview&
# Allow: /w/load.php?
# Disallow: /w/
# Disallow: /api/
# Disallow: /wiki/Special:

The key directives are User-agent (which crawler the rule applies to), Disallow (paths that should not be accessed), Allow (exceptions to disallow rules), and Crawl-delay (requested time between requests). You can parse these programmatically:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://en.wikipedia.org/robots.txt")
rp.read()

# Can a generic crawler fetch an encyclopedia article?
print(rp.can_fetch("*", "https://en.wikipedia.org/wiki/Python_(programming_language)"))
# True — article pages under /wiki/ are open to crawlers

# Can it fetch the dynamic pages under /w/, including the API?
print(rp.can_fetch("*", "https://en.wikipedia.org/w/api.php"))
# False — /w/ is disallowed for crawlers, with only narrow exceptions

# Can it fetch special/internal pages?
print(rp.can_fetch("*", "https://en.wikipedia.org/wiki/Special:Search"))
# False — Special: pages are disallowed

The second result deserves a pause, because it teaches an important nuance. Wikipedia wants programmatic users on its API — Chapter 10 is built on it — yet robots.txt disallows /w/ paths. The two facts are not in conflict: robots.txt addresses crawlers that discover and fetch pages by following links, and Wikipedia would rather crawlers index its stable article URLs than its infinite space of dynamic, server-intensive /w/ URLs. Deliberate API clients are governed by a different set of norms — the Wikimedia User-Agent policy and API etiquette guidelines — that you will follow in Chapter 10. The general lesson: robots.txt tells you how a site wants automated page-fetching to behave, and it is one input among several (Terms of Service, API documentation, rate-limit guidance) rather than the complete rulebook.

If a site declares a Crawl-delay, you can read it programmatically and respect it:

delay = rp.crawl_delay("*")
print(delay)
# None if the site declares no crawl delay; otherwise seconds between requests
wait = delay if delay is not None else 1.0  # Fall back to a polite default

Two important caveats. First, robots.txt is a norm, not a law. Violating it is not illegal (in most jurisdictions), but it is disrespectful of the site operator’s wishes and may result in your IP being blocked. Second, if a site does not have a robots.txt file (the request returns a 404), that means there are no crawling restrictions — not that you should crawl aggressively.

2.3.1 Comparing robots.txt Across Platforms

Different organizations have dramatically different approaches to automated access, and their robots.txt files reveal these philosophies. Compare three major sites:

import requests

sites = {
    "Wikipedia": "https://en.wikipedia.org/robots.txt",
    "Reddit": "https://www.reddit.com/robots.txt",
    "NY Times": "https://www.nytimes.com/robots.txt",
}

for name, url in sites.items():
    response = requests.get(url)
    lines = response.text.strip().split("\n")
    disallow_count = sum(1 for line in lines if line.startswith("Disallow"))
    allow_count = sum(1 for line in lines if line.startswith("Allow"))
    print(f"{name}: {len(lines)} lines, {disallow_count} Disallow rules, {allow_count} Allow rules")

Wikipedia’s robots.txt is permissive where it matters most — the encyclopedia articles under /wiki/ are open to crawlers — while restricting dynamic pages, editing interfaces, and internal administration paths under /w/ and Special:. Programmatic access to its data happens instead through the API and public database dumps, governed by their own etiquette policies. This division of labor reflects Wikipedia’s organizational commitment to openness: the data exists to be shared, through channels designed for sharing it. Reddit’s file is more complex, restricting many URL patterns (user profiles, search results, sorted listings) while permitting basic page access. The New York Times is more restrictive still, blocking large sections of the site from automated access — reflecting the fact that its content is a commercial product.

These differences are not arbitrary. They reflect each organization’s relationship with automated access: Wikipedia encourages it as part of its mission, Reddit tolerates it within limits, and the Times treats its content as proprietary. Reading robots.txt before you scrape tells you not just what is permitted but what kind of organization you are dealing with.

2.4 Responsible Request Headers

When your code makes requests to a web server, the server sees metadata about the request including the User-Agent header, which identifies who is making the request. By default, the requests library sends a generic User-Agent string. You should replace it with one that identifies you and provides contact information:

import requests

headers = {
    "User-Agent": "WebDataScience/1.0 (INFO 4617; brian.keegan@colorado.edu)"
}

response = requests.get("https://example.com", headers=headers)

This practice serves two purposes: it lets site operators contact you if your scraping causes problems, and it signals that you are a responsible actor rather than a malicious bot. Some APIs (including the Wikimedia API you used in Chapter 1) require a meaningful User-Agent and will return errors or impose stricter rate limits if you do not provide one.

TipMissing Manual Reference

For a deeper introduction to HTTP requests, headers, and status codes — the machinery beneath every example in this chapter — see Missing Manual Chapter 24: HTTP and Web APIs.

2.5 Rate Limiting

Making hundreds of requests per second to a web server can degrade its performance for other users — effectively mounting a denial-of-service attack on a site you are trying to study. Always space out your requests:

import time
import requests

urls = ["https://example.com/page1", "https://example.com/page2"]

for url in urls:
    response = requests.get(url, headers=headers)
    # Process the response...
    time.sleep(1)  # Wait 1 second between requests

A good default is one request per second. If the site’s robots.txt specifies a Crawl-delay, respect it. If you are hitting an API with documented rate limits, stay well within them.

2.5.1 Building a Responsible Scraper Function

The principles above — custom User-Agent, rate limiting, error handling — appear in every scraping project. Rather than implementing them ad hoc each time, you can wrap them into a reusable function:

import requests
import time
import logging

def responsible_get(url, headers=None, delay=1.0):
    """Make an HTTP GET request with responsible defaults.

    Parameters
    ----------
    url : str
        The URL to request
    headers : dict, optional
        Custom headers; a default User-Agent is added if none provided
    delay : float
        Seconds to wait after the request (default: 1.0)

    Returns
    -------
    requests.Response or None
        The response object, or None if the request failed
    """
    default_headers = {
        "User-Agent": "WebDataScience/1.0 (student-project; your-email@colorado.edu)"
    }
    if headers:
        default_headers.update(headers)

    try:
        response = requests.get(url, headers=default_headers, timeout=30)
        response.raise_for_status()  # Treat 4xx/5xx statuses as failures
        logging.info(f"{response.status_code} {url}")
        time.sleep(delay)
        return response
    except requests.exceptions.HTTPError as e:
        logging.error(f"HTTP error: {e}")
        time.sleep(delay)  # A failed request still counts against the server
        return None
    except requests.exceptions.RequestException as e:
        logging.error(f"Request failed ({type(e).__name__}): {url}")
        return None

Every design choice in this function reflects a principle from this chapter. The default User-Agent identifies you and provides contact information, so site operators can reach you if your scraping causes problems. The configurable delay enforces rate limiting by default — you have to explicitly override it, which makes aggressive scraping a conscious decision rather than an oversight. The raise_for_status() call ensures that a 404 Not Found or 403 Forbidden is treated as the failure it is, rather than silently passed downstream to your parser. The except blocks handle both HTTP errors and the whole family of network failures (ConnectionError, Timeout, SSLError, and other RequestException subtypes) gracefully, logging the error instead of crashing your entire scraping run. The timeout parameter prevents your script from hanging indefinitely on an unresponsive server. And if a site declares a Crawl-delay in its robots.txt, pass that value as delay so your tool honors the site’s own stated preference.

You will use variations of this pattern in Chapter 6 when scraping HTML tables, in Chapter 7 when retrieving historical snapshots, and throughout the API chapters. Building responsible defaults into your tools is more reliable than remembering to add them each time.

2.6 An Ethical Decision Framework

Before any data collection project, work through these five questions:

  1. Legality. Is this data collection permitted under the CFAA, GDPR, and any applicable local laws? Is it consistent with the site’s Terms of Service?

  2. Consent. Did the people whose data you are collecting consent to this use? Is the data truly public, or did users share it within a platform with reasonable expectations of limited visibility?

  3. Proportionality. Are you collecting only the data you need for your research question? Could you answer your question with less data or with less sensitive data?

  4. Privacy. Does your dataset contain personally identifiable information? Could individuals be re-identified from ostensibly anonymized data? What are the risks if the data is breached or misused?

  5. Server impact. Will your scraping impose a meaningful load on the server? Are you rate-limiting appropriately? Are you scraping during off-peak hours for high-traffic sites?

No framework can substitute for judgment. But working through these questions systematically will help you identify risks and make defensible decisions.

2.6.1 IRBs and Web Data

If you are conducting research at a university, there is an additional layer of oversight: the Institutional Review Board (IRB). IRBs review research involving human subjects to ensure it meets ethical standards derived from the Belmont Report’s three principles — respect for persons, beneficence, and justice.

When does web scraping become human subjects research? The general rule is that it qualifies when you collect data about identifiable people to produce generalizable knowledge. Scraping public government records — legislative votes, budget documents, meeting minutes — is usually not human subjects research because the subjects are institutions and public officials acting in their official capacities. But scraping social media profiles to study health behaviors, political attitudes, or personal relationships almost certainly is, because you are studying identifiable individuals and their behavior.

The line is not always clear. Scraping aggregated, anonymous data (like Wikipedia edit counts without usernames) falls into a gray area. When in doubt, consult your institution’s IRB. Many universities have expedited review processes for minimal-risk research involving publicly available data, and the process of writing an IRB protocol forces you to think carefully about privacy, consent, and data security — exactly the issues this chapter’s framework addresses. The Belmont Report’s principles complement the five-question framework above: respect for persons maps to consent, beneficence to proportionality and privacy, and justice to the question of who bears the risks and who receives the benefits of your research.

2.7 Exercises

  1. Compare robots.txt files. Retrieve and compare robots.txt for five websites in the same category (e.g., five news organizations, five universities, or five social media platforms). What patterns emerge in what they allow and restrict? Write a brief analysis in a Markdown cell.

  2. Terms of Service audit. Read the Terms of Service for a platform you use regularly. Identify three clauses that restrict automated data collection. For each, assess whether academic research would be covered by any exceptions or fair use arguments.

  3. Ethical decision framework. Apply the five-question framework to a hypothetical scenario: a researcher wants to scrape user profiles from a dating app to study patterns in self-presentation across demographic groups. Work through each question and write a recommendation.

  4. Rate limiting experiment. Using time.time() to measure elapsed time, write a loop that makes five requests to a public website with a one-second delay between each. Then modify the delay to two seconds. Confirm that your actual request timing matches your intended rate.

  5. Personal robots.txt audit. Retrieve robots.txt for a website you use daily but have never scraped. Read the Disallow rules carefully. Then find and read the site’s Terms of Service (look for links in the footer). Write a 300-word analysis: if you wanted to collect data from this site for a class project, what would robots.txt, the Terms of Service, and the five-question ethical framework each permit and restrict? Where do these three sources of guidance agree, and where do they conflict?

  6. Graduate extension (INFO 5617). Read Fiesler et al. (2020) alongside the EFF’s CFAA explainer and a summary of Van Buren v. United States. Then write a two-page memo advising a university research team that wants to collect public data from three platforms of your choosing. For each platform, make an explicit determination on four dimensions: CFAA exposure after Van Buren, Terms of Service and contract risk after hiQ, robots.txt guidance, and whether the project likely requires IRB review. Conclude with a collection protocol the team should follow.

NotePublic Interest Connection

The ethical framework introduced here connects directly to the values of oversight and openness discussed in Chapter 3. Responsible data collection practices are themselves a form of oversight — they make your methods transparent and accountable. And the commitment to documenting your scraping practices (User-Agent strings, rate limiting, data handling) embodies the openness that the web data science community depends on.

2.8 Social History and Public Interest

The tension between data access and data protection is not new. Journalists have long navigated similar terrain through freedom-of-information laws, source protection, and editorial ethics. The CFAA was originally designed to combat hacking, but its vague language about “unauthorized access” has been used to threaten researchers, journalists, and security auditors. The Aaron Swartz case — in which a young activist faced federal charges for downloading academic articles from JSTOR through MIT’s network — remains a sobering reminder of how aggressively the CFAA can be applied (Fiesler et al. 2020).

The Cambridge Analytica scandal of 2018 was a turning point. A researcher collected Facebook data through the platform’s API, ostensibly for academic purposes, then shared it with a political consulting firm. The fallout led Facebook, and then other platforms, to drastically restrict API access — a closure that affected all researchers, not just those who had misused data. This history is central to understanding the post-API landscape described in Chapter 3.

The aftermath of Cambridge Analytica created a chilling effect that extended far beyond Facebook. IRBs at universities became more cautious about approving any research involving social media data, even when the data was public and the research served clear public interests. Platforms shut down APIs that legitimate researchers had depended on for years, citing privacy concerns that conveniently also prevented the kind of external scrutiny that might reveal algorithmic bias or content moderation failures. The academic research community is still rebuilding trust and access through alternative frameworks: the European Union’s Digital Services Act (DSA) now mandates researcher access to platform data for public interest research, data donation programs ask users to voluntarily share their data with researchers, and negotiated research agreements between universities and platforms provide structured access with privacy safeguards. These developments represent an ongoing effort to balance legitimate privacy concerns with the public’s interest in understanding how platforms shape information, attention, and behavior. The tools and ethical frameworks in this chapter position you to participate in that effort responsibly.

2.9 Common Issues to Debug

  • robots.txt not found (404): This means there are no crawling restrictions, not that the site is blocking you.
  • Requests blocked despite robots.txt allowing access: Some sites use additional bot detection (CAPTCHAs, JavaScript challenges, behavioral analysis) beyond robots.txt.
  • 403 Forbidden with a responsible User-Agent: The site may be blocking all automated access regardless of your headers. Respect this.
  • Forgetting time.sleep(): If your IP gets temporarily banned, wait at least an hour before trying again, and add appropriate delays to your code.

2.10 Key Takeaways

Every capability introduced in subsequent chapters raises the questions you have worked through here. The legal landscape is unsettled; technical norms like robots.txt provide guidance but not protection; and ethical principles require case-by-case judgment. The framework — legality, consent, proportionality, privacy, server impact — should become habitual. When in doubt, err on the side of caution: use an API instead of scraping, collect less data rather than more, and always identify yourself in your request headers.

2.11 Further Reading

Fiesler, Casey, Nathan Beard, and Brian C. Keegan. 2020. “No Robots, Spiders, or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social Media Terms of Service.” Proceedings of the International AAAI Conference on Web and Social Media 14: 187–96.
Freelon, Deen. 2018. “Computational Research in the Post-API Age.” Political Communication 35 (4): 665–68. https://doi.org/10.1080/10584609.2018.1477506.