import requests
url = "https://www.cnn.com"
timestamp = "20000815" # August 15, 2000
# Pass query arguments as a params dictionary — requests handles the
# URL encoding, which matters here because one of the values is
# itself a URL
api_url = "https://archive.org/wayback/available"
response = requests.get(api_url, params={"url": url, "timestamp": timestamp})
data = response.json()
# Check if a snapshot is available
if data["archived_snapshots"]:
snapshot = data["archived_snapshots"]["closest"]
print(f"Snapshot available: {snapshot['available']}")
print(f"URL: {snapshot['url']}")
print(f"Timestamp: {snapshot['timestamp']}")
else:
print("No snapshot available near this date")7 Archived Web Pages and the Wayback Machine
TipLearning Objectives
- Explain the Internet Archive’s role as a public infrastructure for web preservation
- Query the Wayback Machine’s Availability API and CDX Server API to find historical snapshots
- Retrieve and parse historical page content to track changes over time
- Apply basic NLP techniques (tokenization, stopword removal, bag-of-words) to measure document similarity
- Serialize scraped data to JSON files for persistence and reproducibility
TipCompanion Notebook
Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).
7.1 The Web Is Not Permanent
Web pages change, disappear, and get redesigned constantly. Studies have found that a significant fraction of URLs break within just a few years — a phenomenon called link rot. For researchers, this impermanence creates a fundamental challenge: how do you study something that might not be there tomorrow? And how do you study how something has changed over time when only the current version is visible?
The Internet Archive, founded by Brewster Kahle in 1996, addresses this challenge by archiving the web. Its Wayback Machine has captured over 800 billion web pages, creating a historical record of the web that is freely accessible to anyone. For web data scientists, the Wayback Machine is both a data source and a form of the counter-erosion infrastructure described in Chapter 3 — a community-governed resource that preserves access to web content even as platforms and governments take pages down.
7.2 The Availability API
The simplest way to check whether the Wayback Machine has a snapshot of a URL is the Availability API:
The API returns the closest snapshot to your requested timestamp. Before you retrieve it, one crucial detail. A snapshot URL like https://web.archive.org/web/20000815052826/http://www.cnn.com/ does not return the page exactly as it was captured: the Wayback Machine injects its own toolbar, banner, and JavaScript into the page, and rewrites every link, script, and image reference to point back into the archive. That is convenient for browsing but poisonous for measurement — counts of links, scripts, and images, and even text length, will include the archive’s scaffolding along with the original page. Appending id_ to the timestamp (.../web/20000815052826id_/http://www.cnn.com/) asks for the original capture, unmodified. The tradeoff is that id_ returns the rawest form of the capture: relative links are left unresolved and the page may not render nicely in a browser. For measurement, raw is exactly what you want.
from bs4 import BeautifulSoup
# Append id_ to the timestamp to request the original, unmodified
# capture — no Wayback toolbar, no rewritten links
ts = snapshot["timestamp"]
archived_url = snapshot["url"].replace(ts, ts + "id_")
archived_response = requests.get(archived_url)
soup = BeautifulSoup(archived_response.text, "html.parser")
# Extract links from the year-2000 version of CNN
links = soup.find_all("a")
print(f"Number of links on CNN in August 2000: {len(links)}")7.2.1 Comparing Multiple Snapshots
A single historical snapshot is interesting; comparing snapshots across time is where the real research value lies. Let us retrieve CNN’s homepage at three different points in history and see how the web has evolved:
import time
import requests
from bs4 import BeautifulSoup
def get_snapshot_stats(url, timestamp, headers=None):
"""Retrieve a Wayback snapshot and compute basic statistics."""
try:
api_response = requests.get(
"https://archive.org/wayback/available",
params={"url": url, "timestamp": timestamp},
headers=headers,
)
api_response.raise_for_status()
api_data = api_response.json()
if not api_data.get("archived_snapshots"):
return None
# Fetch the original capture (id_), not the toolbar-wrapped
# version, so the statistics measure the page and only the page
closest = api_data["archived_snapshots"]["closest"]
snapshot_url = closest["url"].replace(
closest["timestamp"], closest["timestamp"] + "id_"
)
page_response = requests.get(snapshot_url, headers=headers)
page_response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Request failed for {url} at {timestamp}: {e}")
return None
soup = BeautifulSoup(page_response.text, "html.parser")
return {
"year": timestamp[:4],
"num_links": len(soup.find_all("a")),
"text_length": len(soup.get_text()),
"num_images": len(soup.find_all("img")),
"num_scripts": len(soup.find_all("script")),
}
# Compare CNN across three eras
timestamps = ["20000815", "20120815", "20240815"]
stats = []
for ts in timestamps:
result = get_snapshot_stats("www.cnn.com", ts)
if result:
stats.append(result)
time.sleep(2) # Be respectful of the ArchiveThe numbers tell a story about the web’s evolution: pages have grown dramatically in complexity, with more links, more text, more images, and — most strikingly — far more JavaScript. The 2000-era web was largely static HTML; the 2024 web is a JavaScript application that happens to render as a web page. This increasing complexity is part of why the techniques in Chapter 8 become necessary for modern scraping.
You can extend this approach to any pair of sites or time periods. Comparing how two competing news organizations’ homepages evolved reveals different editorial strategies. Tracking a government agency’s page across presidential administrations can reveal shifts in priorities and transparency. The Wayback Machine turns the web from a snapshot medium into a longitudinal data source — one where you can measure change rather than merely observing it.
Note the importance of the time.sleep(2) call in the loop above. The Internet Archive is a nonprofit that provides this service for free. Aggressive scraping degrades the experience for everyone. Treat the Archive with the same courtesy you would extend to any community resource — use it responsibly, and it will be there when you need it.
7.3 The CDX Server API
The Availability API finds one snapshot near a date. The CDX Server API lets you query the entire history of snapshots for a URL, which is essential for longitudinal research:
import requests
import pandas as pd
# Query all snapshots of Facebook's Terms of Service
cdx_url = "https://web.archive.org/cdx/search/cdx"
params = {
"url": "facebook.com/terms",
"output": "json",
"fl": "timestamp,original,statuscode,length",
"filter": "statuscode:200", # Server-side: only successful captures
"collapse": "digest", # Server-side: skip consecutive identical captures
}
response = requests.get(cdx_url, params=params)
data = response.json()
# First row is column headers, rest is data
df = pd.DataFrame(data[1:], columns=data[0])
df["timestamp"] = pd.to_datetime(df["timestamp"], format="%Y%m%d%H%M%S")
df["length"] = pd.to_numeric(df["length"], errors="coerce")
print(f"Total snapshots: {len(df)}")
print(f"Date range: {df['timestamp'].min()} to {df['timestamp'].max()}")Two of these parameters do server-side filtering that saves you cleanup work later. The filter=statuscode:200 parameter returns only successful captures, so failed requests and redirects never reach your DataFrame. The collapse=digest parameter drops consecutive captures whose content hash is identical — the Archive often captures a page daily even when nothing changed, and collapsing on digest keeps one capture per distinct version. While developing, it is also worth adding "limit": 100 to cap the response at a manageable size until your code works.
You can visualize how frequently the page was captured and how its size changed over time:
import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True)
# Snapshot frequency over time
df.set_index("timestamp").resample("ME").size().plot(ax=axes[0])
axes[0].set_ylabel("Snapshots per month")
axes[0].set_title("Wayback Machine Capture Frequency")
# Page size over time
df.set_index("timestamp")["length"].plot(ax=axes[1])
axes[1].set_ylabel("Page size (bytes)")
axes[1].set_title("Page Size Over Time")
plt.tight_layout()
plt.show()7.3.1 Filtering and Analyzing CDX Data
The raw CDX feed includes every capture attempt — failed requests, redirects, and duplicate captures that can skew your analysis. Because our query already filtered to status 200 and collapsed identical captures on the server side, the DataFrame arrives analysis-ready; had you omitted those parameters, the equivalent pandas filter would be df[df["statuscode"] == "200"]. Prefer the server-side version: it transfers less data and leaves less cleanup code to maintain.
# Every row is already a successful, distinct capture (filtered
# server-side), so we can go straight to trend analysis
df_ok = df.set_index("timestamp")
# Resample to quarterly frequency
quarterly = df_ok.resample("QS").agg(
captures=("length", "size"),
mean_size=("length", "mean"),
)
# Compute year-over-year page size growth
yearly = df_ok.resample("YS").agg(mean_size=("length", "mean"))
print(yearly.tail(10))fig, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True)
quarterly["captures"].plot(ax=axes[0])
axes[0].set_ylabel("Captures per quarter")
axes[0].set_title("Capture Frequency (Status 200 Only)")
quarterly["mean_size"].plot(ax=axes[1])
axes[1].set_ylabel("Mean page size (bytes)")
axes[1].set_title("Average Page Size Over Time")
plt.tight_layout()
plt.show()Status codes matter because CDX data records every capture attempt, not just successful ones. If you want to study the failures, drop the filter parameter and look at what comes back: a spike in 301 (redirect) codes might indicate a site restructuring; a cluster of 503 (Service Unavailable) codes might indicate a period of downtime. These metadata are themselves interesting — they tell you about the infrastructure behind the page, not just its content.
Page size trends are particularly revealing. A steady increase in page size often correlates with the addition of tracking scripts, advertising frameworks, and JavaScript-heavy interfaces. A sudden jump might indicate a redesign; a sudden drop might mean the site moved content behind a login wall. When combined with the content analysis techniques later in this chapter, CDX metadata provides the quantitative backbone for your longitudinal research — telling you how much changed while bag-of-words analysis tells you what changed.
7.4 Tracking Policy Changes Over Time
A current research project explores how social media platforms’ terms of service have evolved. Let us track changes in Facebook’s Terms of Service by retrieving content at regular intervals.
7.4.1 Generating Date Ranges
7.4.2 Retrieving Historical Content
import time
from datetime import datetime
fb_tos_url = "facebook.com/terms"
def get_wayback_content(url, date, headers=None):
"""Retrieve a page from the Wayback Machine closest to a given date.
Parameters
----------
url : str
The original URL (without http://)
date : datetime
The target date for the snapshot
headers : dict, optional
HTTP headers for the request
Returns
-------
dict or None
Dictionary with 'timestamp', 'url', 'content', and 'word_count',
or None if no snapshot exists or the request failed
"""
timestamp = date.strftime("%Y%m%d")
try:
api_response = requests.get(
"https://archive.org/wayback/available",
params={"url": url, "timestamp": timestamp},
headers=headers,
)
api_response.raise_for_status()
api_data = api_response.json()
if not api_data.get("archived_snapshots"):
return None
# Fetch the original capture (id_) so the extracted text doesn't
# include the Wayback toolbar and banner
closest = api_data["archived_snapshots"]["closest"]
snapshot_url = closest["url"].replace(
closest["timestamp"], closest["timestamp"] + "id_"
)
page_response = requests.get(snapshot_url, headers=headers)
page_response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Request failed for {url} near {timestamp}: {e}")
return None
soup = BeautifulSoup(page_response.text, "html.parser")
# Extract text content
text = soup.get_text(separator=" ", strip=True)
return {
"timestamp": closest["timestamp"],
"url": snapshot_url,
"word_count": len(text.split()),
"content": text[:5000] # First 5000 characters to avoid memory issues
}7.4.3 Collecting the History
The function retrieves one snapshot; a loop over the quarterly dates assembles the history. Note where the time.sleep() lives — in the loop, between requests, just as in Chapter 6 — and that failed retrievals (returned as None) are simply skipped rather than crashing the run:
results = []
for date in dates:
print(f"Fetching snapshot near {date.date()}...")
result = get_wayback_content(fb_tos_url, date)
if result is not None:
results.append(result)
time.sleep(1) # Rest between iterations — the Archive is a nonprofit
print(f"Retrieved {len(results)} snapshots from {len(dates)} time points")
# Retrieved 68 snapshots from 77 time points7.4.4 Serializing Data
Once you have collected historical content, save it so you do not need to re-scrape:
7.4.5 Caching Raw Captures
Serializing your final results is one half of a reproducible workflow. The other half is caching each raw capture as you fetch it, so that re-running the collection loop never re-downloads something you already have:
import os
os.makedirs("wayback_cache", exist_ok=True)
def get_wayback_content_cached(url, date, headers=None):
"""Fetch a snapshot, reading from and writing to a local cache."""
# Key the cache file by the snapshot's target date
cache_path = f"wayback_cache/{date.strftime('%Y%m%d')}.json"
if os.path.exists(cache_path): # Already fetched — skip the network
with open(cache_path, "r") as f:
return json.load(f)
result = get_wayback_content(url, date, headers=headers)
if result is not None: # Save for next time
with open(cache_path, "w") as f:
json.dump(result, f)
return resultSwap this wrapper into the collection loop and your pipeline gains two properties worth having. It is polite: no matter how many times you restart the loop while debugging, each snapshot is requested from the Archive exactly once. And it is reproducible: your analysis is anchored to the exact content you retrieved, even if a capture later becomes unavailable or the service is down when you rerun your notebook.
7.5 Document Similarity with Bag-of-Words
To measure how much a document changed between versions, you can use a bag-of-words approach. This technique represents each document as a vector of word frequencies, ignoring word order, and then measures the similarity between vectors.
from gensim.utils import simple_preprocess
from nltk.corpus import stopwords
import nltk
# Download stopwords (first time only)
# nltk.download('stopwords')
stop_words = set(stopwords.words('english'))
def preprocess_document(text):
"""Tokenize, lowercase, and remove stopwords from a document."""
tokens = simple_preprocess(text) # Tokenize and lowercase
return [token for token in tokens if token not in stop_words]
# Preprocess two versions of the ToS
doc_v1 = preprocess_document(results[0]["content"])
doc_v2 = preprocess_document(results[-1]["content"])
print(f"Version 1 tokens: {len(doc_v1)}")
print(f"Version 2 tokens: {len(doc_v2)}")Using gensim’s dictionary and bag-of-words tools, you can build a corpus and compute similarity:
from gensim import corpora, similarities
# Build a dictionary mapping words to IDs
documents = [doc_v1, doc_v2]
dictionary = corpora.Dictionary(documents)
# Convert documents to bag-of-words vectors
corpus = [dictionary.doc2bow(doc) for doc in documents]
# Compute similarity
index = similarities.SparseMatrixSimilarity(corpus, num_features=len(dictionary))
sims = index[corpus[0]]
print(f"Similarity between versions: {sims[1]:.3f}")
# 1.0 = identical, 0.0 = completely different7.5.1 Visualizing Document Change Over Time
The bag-of-words similarity measure tells you how similar two documents are, but plotting similarity between consecutive versions over time creates a “change timeline” — a visual record of when and how much a document evolved:
from gensim import corpora, similarities
from gensim.utils import simple_preprocess
from nltk.corpus import stopwords
stop_words = set(stopwords.words("english"))
# Preprocess all versions
processed_docs = []
for r in results:
if r and r.get("content"):
tokens = [t for t in simple_preprocess(r["content"]) if t not in stop_words]
processed_docs.append(tokens)
# Build dictionary and corpus
dictionary = corpora.Dictionary(processed_docs)
corpus = [dictionary.doc2bow(doc) for doc in processed_docs]
# Compute similarity between consecutive versions
change_scores = []
index = similarities.SparseMatrixSimilarity(corpus, num_features=len(dictionary))
for i in range(1, len(corpus)):
sim = index[corpus[i-1]][i]
change_scores.append(1 - sim) # Convert similarity to change score
# Plot the change timeline
plt.figure(figsize=(12, 4))
plt.plot(range(len(change_scores)), change_scores, marker="o")
plt.xlabel("Version Transition")
plt.ylabel("Change Score (0 = identical, 1 = completely different)")
plt.title("Document Change Over Time")
plt.tight_layout()
plt.show()Spikes in this timeline correspond to major revisions — a new Terms of Service policy, a site redesign, or a significant content update. By cross-referencing these spikes with real-world events (a data breach, a regulatory action, a public controversy), you can begin to tell a story about why the document changed, not just that it changed. This kind of temporal analysis is at the heart of longitudinal web research.
The change score visualization complements the CDX analysis from earlier in this chapter. CDX data tells you how often the page was captured and how its size changed; the change timeline tells you how different consecutive versions are from each other. Together, they create a multi-dimensional picture of a document’s evolution. A page might be captured frequently but change very little (a stable, well-maintained site), or captured infrequently but show dramatic changes between captures (a site that undergoes periodic major overhauls).
You can also extend this approach beyond consecutive comparisons. Computing pairwise similarity between all versions produces a similarity matrix that you can visualize as a heatmap. Clusters of similar versions reveal stable periods; off-diagonal dissimilarities reveal turning points. For research on platform governance, these turning points often align with public controversies, regulatory actions, or leadership changes — patterns that would be invisible without longitudinal data.
The combination of the Wayback Machine’s historical data with computational text analysis techniques creates a powerful research methodology. You are not just reading old web pages — you are measuring institutional behavior over time, producing evidence that can inform policy debates, academic research, and public accountability. This is precisely the kind of public interest data science that Chapter 3 describes.
TipMissing Manual Reference
For more on data serialization formats including JSON, CSV, and pickle, see Missing Manual Chapter 20: Data File Formats.
7.6 Exercises
Policy tracking. Track the evolution of a policy document of your choice (a university’s academic integrity policy, a company’s privacy policy, or a government agency’s FAQ page) over at least five time points using the Wayback Machine. Visualize how document length changes and identify periods of major revision.
CDX analysis. Use the CDX Server API to compare the snapshot frequency and page size trends for two competing organizations (e.g., two news outlets, two universities, or two tech companies). What does the capture frequency tell you about the organizations’ web presence?
Document similarity. Compute pairwise document similarity across all versions of your tracked document from Exercise 1. Create a heatmap or matrix visualization. Which version transitions involved the largest changes?
Government web archive. Use the CDX Server API to analyze the archival history of a government website (e.g., a state legislature, a federal agency, or a city government). How frequently was the page captured? Are there gaps in coverage? How has the page size changed over time? Write a brief analysis of what the archival patterns reveal about the site’s evolution.
Then-and-now similarity. For a website of your choice, retrieve the earliest available Wayback Machine snapshot and the most recent snapshot. Preprocess both with tokenization and stopword removal, compute bag-of-words similarity, and write a 200-word analysis of what changed. What kinds of content were added, removed, or restructured?
Graduate extension (INFO 5617). Either read the chapter on website history in Rogers (2019) and write a critical assessment connecting its archival methods to the techniques in this chapter, or use the CDX Server API to reconstruct the editorial evolution of a news organization’s homepage across at least ten years, sampling snapshots at regular intervals. Whichever path you choose, produce a methods write-up that squarely addresses the limitations of Wayback data as a research source: the toolbar injection and link rewriting that the
id_suffix avoids, capture-frequency bias (heavily trafficked pages are archived far more often, so observed “change” is confounded with attention), and the gaps where no capture exists at all.
NotePublic Interest Connection
The Wayback Machine embodies the value of ownership discussed in Chapter 3 — it is collectively governed, freely accessible, and exists to preserve the web as a public resource. When platforms delete content, change their policies, or redesign their sites, the Wayback Machine often provides the only independent record of what existed before.
7.8 Common Issues to Debug
- Wayback Machine rate limiting: Space your requests with
time.sleep(). The service is free and community-maintained; do not abuse it. - Incomplete archived pages: Many snapshots are missing CSS, images, or JavaScript-rendered content. The text content is usually preserved, but visual rendering may be broken.
- CDX data inconsistencies: Some entries may have missing fields or redirect status codes instead of actual content. Filter your data carefully.
- NLTK download errors: Run
nltk.download('stopwords')once in a fresh Python session to download the stopwords corpus.
7.9 Key Takeaways
The web is not permanent, and the Wayback Machine is your most important tool for longitudinal web research. The Availability API finds specific snapshots; the CDX Server enumerates the full history; and BeautifulSoup parses archived pages just like current ones. Document similarity techniques give you quantitative tools for measuring change over time. Always serialize your scraped data — you may not be able to re-scrape it later, and the page you studied today may be gone tomorrow.
7.10 Further Reading
- Internet Archive Wayback Machine APIs: https://archive.org/help/wayback_api.php
- CDX Server API: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server
- gensim documentation: https://radimrehurek.com/gensim/
- Rogers (2019) — Chapter on web historiography
Rogers, Richard. 2019. Doing Digital Methods. SAGE Publications.
7.7 Social History and Public Interest
Brewster Kahle founded the Internet Archive with the vision of creating a “library of everything” — a comprehensive, freely accessible archive of human knowledge. The Wayback Machine, launched in 2001, has become indispensable for journalists investigating corporate claims, lawyers establishing timelines of events, and researchers conducting longitudinal studies of online content.
The Internet Archive has faced legal challenges from publishers, platforms, and governments that would prefer certain content to remain forgotten. Its continued operation depends on the support of its users and donors — a reminder that the infrastructure of openness requires ongoing stewardship.
Journalists have been among the most creative users of the Wayback Machine. When politicians delete social media posts or companies quietly change their terms of service, archived versions provide accountability. Legal teams use archived web pages as evidence in intellectual property disputes, contract disagreements, and regulatory investigations — establishing what a website said on a specific date can be the difference between winning and losing a case. Researchers in fields from political science to public health use longitudinal snapshots to study how organizations communicate about crises, how policy language evolves, and how online communities form and fragment over time.
The Internet Archive’s work also raises fundamental questions about the right to be forgotten versus the public’s right to know. The European Union’s GDPR includes provisions allowing individuals to request deletion of personal data, which can conflict with the archival mission. Some organizations have requested that the Wayback Machine remove their content. These tensions mirror the broader conflicts between openness, oversight, and ownership that Chapter 3 explores — the same data that enables accountability can also enable surveillance, and the infrastructure that preserves public knowledge can also preserve content that individuals wish to retract.