3 The Post-API Age
- Describe the historical arc of web data access from open APIs to restricted, monetized, and disappearing endpoints
- Analyze specific cases of access restriction using the enclosure, exemption, and erosion framework
- Identify alternative data access strategies including archives, FOIA requests, data donations, and federated platforms
- Connect contemporary data access challenges to longer traditions of public interest inquiry
- Articulate the values of openness, oversight, and ownership as guiding principles for web data practice
This chapter’s companion notebook contains the discussion and exercises for note-taking — this is a conceptual chapter with no code (see Appendix A for all notebooks).
3.1 The Golden Age of Web Data Access
For roughly a decade — from about 2008 to 2018 — the web behaved as if it wanted to be studied. Twitter opened its API in 2006 and for years offered researchers a generous view of public conversation. Facebook launched the Graph API in 2010, and academic teams used it to map friendship networks at unprecedented scale. Reddit’s API was free and essentially unrestricted, Google exposed programmatic access to search trends, maps, and books, and a generation of startups assumed that open data access was simply how platforms worked.
Platforms were not being charitable. Open APIs served their strategic interests: third-party developers built apps that made platforms more useful, academic partnerships lent credibility and surfaced insights, and every integration deepened the platform’s position as infrastructure that other people’s software depended on. The unspoken bargain was that platforms got ecosystems and researchers got data.
What that access enabled was real. Computational social science emerged as a field in this window (Lazer et al. 2009), built substantially on platform data: studies of information diffusion, social movements, public health signals, and political communication that would have been unimaginable with surveys alone. Data journalists used APIs to hold institutions accountable. Civic technologists built tools on open government and platform data. When researchers today describe the “post-API age” (Freelon 2018), this is the era they measure against.
My own research career took shape inside this bargain. The Wikipedia studies that formed my dissertation — how volunteer editors self-organize to cover breaking news events — were possible because the MediaWiki API and public database dumps exposed every edit, by every editor, with timestamps, for free (Chapter 10 teaches you the same interfaces). Later work on Reddit communities depended on Pushshift, a volunteer-run archive that made the site’s full comment history queryable for researchers (Baumgartner et al. 2020). Neither project required permission, partnership, or payment. That is what has changed.
The landscape of web data access has undergone a structural transformation since approximately 2018. Three pressures — enclosure, exemption, and erosion — explain not just that access is shrinking but why, and what forces are driving the change.
Technical systems embed political choices — who gets access, on what terms, and who decides. For a broader introduction to this idea, see Missing Manual Chapter 8: Artifacts Have Politics.
3.2 Three Structural Pressures
3.2.1 Enclosure
Enclosure is the privatization of previously accessible data: platforms converting a shared resource into a controlled asset. The name deliberately echoes the enclosure of common land in England between the sixteenth and nineteenth centuries, when fields that villagers had collectively farmed and grazed for generations were fenced off, by force of law, into private holdings. The land did not disappear. What disappeared was the commons — the shared arrangement under which ordinary people could use it. Data enclosure works the same way: the posts, edits, and traces are still there, but the terms of access have been fenced.
The mechanisms are familiar by now. APIs are shut down or repriced beyond reach. Third-party archives are cut off. Bulk exports are restricted. And, increasingly, access is bundled into exclusive licensing deals.
Two cases anchor the pattern:
- Twitter/X. In January 2021, Twitter launched an Academic Research track offering qualified researchers free access to the full historical archive — arguably the high-water mark of platform openness. Two years later, following the company’s 2022 acquisition, free access was eliminated, the academic track was discontinued, and enterprise access was priced at $42,000 per month and up. Hundreds of research projects and public-interest tools — bot detectors, misinformation trackers, disaster-response monitors — went dark within months.
- Reddit. Reddit’s API had been free since 2008. In 2023 the company announced pricing of $0.24 per 1,000 API calls, effective that July. The developer of Apollo, a popular third-party client, calculated the change would cost roughly $20 million a year and shut the app down; more than seven thousand subreddits went dark in protest. The same enforcement wave cut off Pushshift (Baumgartner et al. 2020), the archive that had underpinned hundreds of published Reddit studies. The research community lost not just future access but its accumulated observational infrastructure.
Facebook’s trajectory is the template both followed: after the Cambridge Analytica scandal broke in 2018, Meta locked down the Graph API, and in August 2024 it shut down CrowdTangle — the tool journalists and researchers used to monitor viral content — replacing it with a Meta Content Library available only through gated application.
Why did enclosure accelerate when it did? Generative AI is a large part of the answer. Once large language models made user-generated text valuable as training data, the calculus flipped: an open API stopped looking like ecosystem-building and started looking like giving away inventory. Reddit’s reported licensing deal with Google — roughly $60 million per year for training access — was announced within months of the API repricing that had priced out researchers and hobbyists. Data that communities produced collectively is now sold privately, which is the enclosure of the commons in nearly literal form.
3.2.2 Exemption
Exemption is the second pressure: platforms operating beyond meaningful outside scrutiny, invoking privacy, intellectual property, or security to resist independent investigation while conducting the same investigations internally at will.
The asymmetry is the tell. Platforms run experiments on their users continuously — every A/B test is a behavioral study — and employ large internal research teams with total data access. But when outside researchers attempt to study the platforms, the tools of resistance come out: cease-and-desist letters citing the Computer Fraud and Abuse Act, Terms of Service that prohibit the data collection that public-interest research requires (Fiesler et al. 2020), and account bans for researchers who persist. In 2021, Meta disabled the accounts of NYU’s Ad Observatory team, who were studying political ad targeting with data volunteered by consenting users — an action Meta justified as protecting user privacy. Courts have pushed back at the margins (Chapter 2 covers Sandvig v. Barr, Van Buren, and the hiQ litigation), but legal ambiguity itself does deterrence work: a graduate student weighing a scraping study against a potential federal claim usually just picks a different topic.
The privacy justification deserves particular scrutiny. Platforms routinely cite user protection when closing researcher access — while continuing to sell fine-grained targeting on the same users to advertisers. Privacy is genuinely important, and Chapter 2 takes it seriously. But when “privacy” restricts only accountability research and never revenue, it is functioning as exemption, not protection.
Regulation is beginning to respond. The European Union’s Digital Services Act includes a provision — Article 40 — that requires very large platforms to provide vetted researchers with data access for studying systemic risks, with implementing rules still maturing as of mid-2026. Whether such mandates produce usable access, or a new bureaucratic bottleneck, is one of the defining open questions of this decade of internet research.
3.2.3 Erosion
Erosion is the third pressure, and the quietest: the decay of the infrastructure for sustained observation. Enclosure and exemption are things platforms do; erosion is what happens to the web itself.
Links rot. Roughly half the hyperlinks cited in U.S. Supreme Court opinions no longer point to the content the justices cited, and a 2024 Pew study found that about a third of webpages that existed in 2013 were gone a decade later. Platforms churn: GeoCities vanished with its era of amateur web culture; Parler went offline overnight when its hosting was pulled; Twitter became X and broke a decade of embedded links and stored identifiers. Sites redesign their HTML and silently break every scraper built against the old structure — a fragility you will experience personally in Chapter 6 and Chapter 8.
For researchers, erosion produces a reproducibility crisis. A study built on the Twitter Academic API cannot be replicated at any price today. Papers that cite Pushshift point at an archive that no longer accepts queries. When the data underlying published findings disappears, the scientific record quietly loses its evidentiary base (Davidson et al. 2023). And migration to federated platforms — Mastodon, Bluesky — solves some access problems (Chapter 12) while complicating basic measurement: there is no single API endpoint for “all of Mastodon,” because there is no single Mastodon.
Erosion is also why two later chapters exist. The Wayback Machine (Chapter 7) is counter-erosion infrastructure — a deliberate institutional bet that the web is worth preserving. And automation (Chapter 14) is counter-erosion practice: if data disappears, systematic collection now is the only insurance for research later.
3.3 Three Counter-Values
In response to these pressures, three values can guide how researchers, technologists, and institutions build and sustain access to web data for the public interest. They are not a solution to the post-API age; they are a stance for working inside it.
3.3.1 Openness
Openness means conducting your own work so that it resists enclosure. Transparent methods and pipelines let others evaluate and reproduce what you did. Documenting not just what data you collected but how — which endpoints, which parameters, which dates, what the platform’s constraints were at collection time — is what allows a future researcher to understand your dataset after the source has changed or vanished (datasheets are the tool for this (Gebru et al. 2021)). Redundancy is openness in practice: depositing data in durable public repositories like Zenodo or institutional archives, relying on community resources like the Internet Archive and Common Crawl, and preferring platforms whose data carries open licenses. Wikipedia, which anchors Chapter 10, is the standing counter-example to enclosure: open API, open license, full public dumps.
3.3.2 Oversight
Oversight means insisting that consequential systems be inspectable by parties who do not own them. In practice this includes independent audits of platform algorithms and content moderation; documentation standards that make datasets and collection methods legible to review; impact assessment when access regimes change; and regulatory frameworks — like DSA Article 40 — that convert researcher access from a privilege platforms grant into an obligation they carry. The professional skills this book teaches are oversight skills: someone who can retrieve, parse, and analyze web data independently is someone a platform cannot simply refuse.
3.3.3 Ownership
Ownership means preferring arrangements where communities govern the infrastructure their data lives in. Federated systems built on open protocols — ActivityPub, which underlies Mastodon, and the AT Protocol, which underlies Bluesky — distribute control so that no single acquisition or pricing decision can enclose the whole network (Chapter 12 works with both). Community-maintained archives, data cooperatives, and volunteer preservation efforts like ArchiveTeam embody the same value: the traces of collective life are held collectively. Ownership is the long-horizon value — hardest to practice, and the only durable answer to enclosure.
3.4 Public Interest Lineages
The tensions described in this chapter are not new to data science. Other professions have confronted analogous challenges — powerful actors controlling access to information that the public needs — and developed institutional responses worth learning from.
Journalism built the most direct precedent. Freedom of information laws exist because reporters and reformers spent decades establishing that government records belong to the public; the muckraking tradition established that private power, too, is a legitimate target of sustained scrutiny. When you file a public records request to obtain data an agency will not publish (a strategy revisited below), you are working in this lineage.
Law institutionalized compelled disclosure. Discovery rules force parties to produce documents they would prefer to hide, on the theory that adjudication without evidence is theater. Due process embodies a proceduralist answer to arbitrary power — decisions must be explainable and reviewable — that algorithmic accountability research inherits almost verbatim.
Accounting shows how disclosure becomes infrastructure. After the 1929 crash, mandatory audits and standardized reporting turned corporate finances from private mystery into public record. The lesson for data science: trustworthy disclosure at scale requires standards, professional norms, and independent examiners — not goodwill.
Urban planning developed participation as method. Environmental impact statements and public comment periods encode the principle that decisions affecting shared resources require advance disclosure and community input — a template now echoed in calls for algorithmic impact assessments.
Engineering professionalized public safety. Licensure, safety codes, and ethics canons — many written in the aftermath of disasters — established that technical practitioners owe duties to the public that override duties to employers. Data science is still writing its equivalent.
The argument of this section is not that data science should copy any one of these professions, but that it is developing its own public interest mission — and that mission has precedents, institutions, and hard-won lessons to draw on.
3.5 What This Means for Practice
The rest of this book equips you with a diverse technical toolkit precisely because no single method of access is reliable. When APIs close, you may need to scrape HTML (Chapter 6, Chapter 8). When web pages change, archives provide continuity (Chapter 7). When documents are locked in inaccessible formats, PDF extraction opens them up (Chapter 9). When platforms restrict automated access, federated alternatives offer open protocols (Chapter 12). And when you need sustained observation over time, automation turns one-time data collection into durable infrastructure (Chapter 14).
Beyond this book’s core toolkit, four alternative access strategies belong in your repertoire:
- Official research access programs. Some platforms now offer application-gated researcher access — the Meta Content Library, TikTok’s Research API, Reddit’s researcher program. These are real but rationed: eligibility is narrow, terms restrict publication and sharing, and access can be revoked. Use them, and document their constraints in your methods.
- Data donation. Users have rights to their own data — in many jurisdictions, legally enforceable export rights — and can volunteer it to research. Donation studies recruit participants to contribute their download packages or install instrumented browser extensions, converting individual data rights into collective observation, with consent built in.
- FOIA and public records requests. Government-held data is subject to disclosure law. A well-scoped records request can produce datasets no API offers — inspection records, communications, contracts — with Chapter 11 covering the data agencies publish proactively.
- Crowdsourcing and citizen science. When no one holds the data, communities can create it: distributed observation, structured reporting, and collaborative annotation have produced public-interest datasets from air quality to political advertising.
The strategic lesson is redundancy and flexibility. Build your skills across the full range of techniques so that the closure of any single data source does not shut down your research program.
3.6 Common Misconceptions
Students often arrive at this material with intuitions that the post-API age has quietly invalidated. Five are worth correcting explicitly:
- “The post-API age means APIs are gone.” There are more APIs than ever. What changed is the terms: free and open access for outside observers is what is disappearing, replaced by paid tiers, gated programs, and licensing deals.
- “If the API closes, I’ll just scrape.” Sometimes — this book teaches you how. But scraping carries its own legal ambiguity (Chapter 2), breaks when sites redesign, and scales poorly. Treat it as one tool in a portfolio, not an escape hatch that makes enclosure irrelevant.
- “If data is publicly visible, access must be permitted and ethical.” Visibility, permission, and ethics are three different questions (Fiesler et al. 2020). A post being public does not mean its author expected it in a dataset, and a page being viewable does not mean its ToS allows automated collection.
- “This only matters for social media research.” Enclosure logic is spreading wherever data gained AI-training value: news archives, code repositories, forums, image libraries. Government data (Chapter 11) is the notable counter-current, held open by legal mandate.
- “Nothing can be done about it.” The counter-values are not wishful thinking: regulation is creating access obligations, archives are institutionalizing preservation, and federated platforms are growing. The constraint environment is contested, and practitioners’ choices are part of the contest.
3.7 Exercises
Timeline exercise. Create a timeline of major API access changes from 2008 to the present. Include at least five events across at least three platforms. For each event, identify which of the three pressures (enclosure, exemption, erosion) best characterizes it.
Framework application. Choose a platform API change not discussed in this chapter. Write a 500-word analysis applying the enclosure/exemption/erosion framework.
Data contingency plan. Design a research project that depends on a specific web data source. Then write a contingency plan: if your primary source disappears mid-study, what are your backup strategies? Consider at least three alternatives.
Public interest audit. Identify a web data research project (published paper, journalism investigation, or civic tech tool) that was enabled by API access. Assess how the project would be affected by current access restrictions. Which of the three values — openness, oversight, or ownership — does the project embody?
Graduate extension (INFO 5617). Read Davidson et al. (2023) and Perriam et al. (2020). Choose one platform and write a two-to-three page “state of research access” brief: what data researchers could obtain five years ago, what is available today and on what terms (cost, eligibility, restrictions), what official research-access programs exist, and which of this chapter’s three pressures best explains the trajectory. Close with a concrete recommendation for a researcher starting a study of that platform this year.
3.8 Key Takeaways
The core argument of this chapter is that understanding why access is shrinking — not just that it is — equips you to navigate the constraints strategically rather than being surprised by them.
- The open-API era (roughly 2008–2018) was a strategic bargain, not a natural state: platforms traded data access for ecosystems, and computational social science was built on the trade.
- Enclosure converts collectively produced data into privately controlled assets — API shutdowns, repricing, archive removals, and AI-training licensing deals are its mechanisms.
- Exemption is the asymmetry of scrutiny: platforms study their users freely while deploying legal and technical tools against independent researchers who study the platforms.
- Erosion is infrastructural decay — link rot, platform churn, and disappearing datasets — and it makes reproducibility a design problem you must solve at collection time.
- Openness, oversight, and ownership are the counter-values: document and deposit so your work resists enclosure, insist on inspectability, and prefer community-governed infrastructure.
- The practical response is redundancy: master multiple access techniques (documents, archives, APIs, automation) so no single closure ends your research program.
3.9 Further Reading
- Freelon (2018) — the original articulation of the “post-API age”
- Bruns (2019) — platforms’ fight against critical scholarly research
- Davidson et al. (2023) — empirical study of how API restrictions threaten open science
- Perriam et al. (2020) — digital methods in a post-API environment
- Lazer et al. (2020) — obstacles and opportunities in computational social science