import requests
url = "https://en.wikipedia.org/w/api.php"
params = {
"action": "query", # We want to query for information
"titles": "Elizabeth_II", # About this article
"prop": "info", # Specifically, basic info about the page
"format": "json" # Return the response as JSON
}
# Define this once near the top of your script and reuse it everywhere
HEADERS = {"User-Agent": "WebDataScience/1.0 (INFO 4617; brian.keegan@colorado.edu)"}
response = requests.get(url, params=params, headers=HEADERS)
data = response.json()
print(data)10 Introduction to APIs: Wikipedia
TipLearning Objectives
- Explain what an API is and how it differs from web scraping as a data access method
- Read API documentation to identify endpoints, parameters, and response formats
- Construct parameterized API requests and handle pagination with continuation tokens
- Use multiple MediaWiki API endpoints to answer research questions
- Write raw API requests from documentation rather than relying on wrapper functions
TipCompanion Notebook
Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).
10.1 What Is an API?
An API (Application Programming Interface) is a contract between a client and a server that specifies what data you can request, what parameters you can pass, and in what format you will receive the response. Unlike web scraping, where you reverse-engineer the HTML structure of a page designed for human readers, APIs provide structured, documented, and intentional access to data. They return clean JSON rather than cluttered HTML. They offer stable endpoints rather than fragile CSS selectors. And they come with documentation that tells you exactly what is available and how to ask for it.
The tradeoff, of course, is that APIs only expose what their operators choose to expose. You cannot query a field that the API does not provide. You cannot exceed the rate limits the server enforces. And as Chapter 3 describes, APIs can be restricted, repriced, or shut down entirely at the operator’s discretion. But when they are available, APIs are almost always preferable to scraping.
This chapter marks the transition from document scraping (Part II) to API-based data access (Part III). The skills you learn here — reading documentation, constructing parameterized URLs, handling pagination, parsing nested JSON responses — transfer directly to every API you will work with in subsequent chapters.
10.2 Why Wikipedia?
Wikipedia’s MediaWiki API is pedagogically ideal for learning API access. It is free, requires no authentication beyond a User-Agent header, has comprehensive documentation, and provides data rich enough to support diverse research designs — from content analysis to network mapping to event detection. It is also an example of what Chapter 3 describes as openness: a mission-driven organization that maintains free, well-documented data access by design.
NotePublic Interest Connection
Wikipedia is the counter-example to the enclosure story of Chapter 3. While commercial platforms restricted, repriced, or shut down their APIs, Wikipedia kept its API open, licensed its content for free reuse, and published complete database dumps for anyone to download. That combination — open API, open license, public dumps — is why this book uses Wikipedia as its teaching platform for API fundamentals: nothing you learn here sits at the mercy of a pricing change or a corporate pivot.
We will use the English Wikipedia article for Elizabeth II as our primary case study. The article has a rich editing history with dramatic spikes around major events, making it ideal for demonstrating how to extract and analyze temporal patterns from API data.
10.3 The Anatomy of an API Call
Every MediaWiki API request follows the same structure: a base URL, an action parameter specifying what you want to do, additional parameters narrowing your request, and a format parameter specifying the response format. Let us trace through a simple example:
The HEADERS constant deserves a comment before we go further. The Wikimedia User-Agent policy asks every client to identify itself with a descriptive User-Agent string that includes contact information — anonymous or generic user agents may be throttled or blocked outright. Define it once as a module-level constant, and every function you write in this chapter can use it as a default.
The response is a nested JSON structure. You navigate it one level at a time:
# The data lives inside data > query > pages > {page_id}
pages = data["query"]["pages"]
# Page IDs are keys — get the first (and usually only) one
page_id = list(pages.keys())[0]
page_info = pages[page_id]
print(f"Title: {page_info['title']}")
print(f"Page ID: {page_info['pageid']}")
print(f"Last edited: {page_info.get('touched', 'unknown')}")This nested navigation pattern — data["query"]["pages"][page_id] — is consistent across the MediaWiki API. Once you internalize it, every endpoint feels familiar.
TipMissing Manual Reference
For strategies on reading and navigating unfamiliar API documentation, see Missing Manual Chapter 5: Reading Official Documentation.
10.4 Getting Revisions
Every edit ever made to a Wikipedia article is recorded and accessible through the API. The revisions endpoint returns metadata about each edit: who made it, when, how much the article size changed, and what edit summary they left.
import requests
import pandas as pd
def get_revisions(title, headers=HEADERS):
"""Get all revisions for a Wikipedia article.
Parameters
----------
title : str
The article title (e.g., 'Elizabeth_II')
headers : dict, optional
HTTP headers; defaults to the module-level HEADERS
Returns
-------
pd.DataFrame
DataFrame with revision metadata
"""
url = "https://en.wikipedia.org/w/api.php"
params = {
"action": "query",
"titles": title,
"prop": "revisions",
"rvprop": "ids|timestamp|user|size|comment",
"rvlimit": "max", # Get as many as possible per request
"format": "json"
}
all_revisions = []
while True:
response = requests.get(url, params=params, headers=headers)
response.raise_for_status() # Stop on HTTP errors
data = response.json()
# Navigate the nested response
pages = data["query"]["pages"]
page_id = list(pages.keys())[0]
revisions = pages[page_id].get("revisions", [])
all_revisions.extend(revisions)
# Handle pagination
if "continue" in data:
params.update(data["continue"])
print(f" Fetched {len(all_revisions)} revisions so far...")
else:
break
return pd.DataFrame(all_revisions)
df_revisions = get_revisions("Elizabeth_II")
print(f"Total revisions: {len(df_revisions)}")10.4.1 Understanding Pagination
The MediaWiki API limits the number of results returned per request. When more results are available, the response includes a "continue" key with parameters you must add to your next request. This loop — request, extract, check for continuation, update parameters, repeat — is the standard pagination pattern for most APIs. You will see it again in Chapter 11 and Chapter 12.
The critical mistake newcomers make is forgetting to check for continuation. Without the while True loop and "continue" check, you only get the first batch of results — potentially a small fraction of the total dataset.
Two smaller details in get_revisions() are worth adopting as habits. The response.raise_for_status() call makes HTTP failures loud: if the server returns an error status, your code stops with an informative exception instead of failing confusingly when you try to parse the JSON. And for bulk jobs, API etiquette suggests adding "maxlag": 5 to your parameters, which tells overloaded Wikimedia servers to refuse your request with a retryable error so that automated traffic yields to human readers.
10.4.2 Analyzing Revision History
# Convert timestamps to datetime
df_revisions["timestamp"] = pd.to_datetime(df_revisions["timestamp"])
# Count monthly edits
monthly_edits = df_revisions.set_index("timestamp").resample("ME").size()
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(14, 5))
monthly_edits.plot(ax=ax)
ax.set_title("Monthly Edits to 'Elizabeth II' Wikipedia Article")
ax.set_ylabel("Number of edits")
ax.set_xlabel("")
plt.tight_layout()
plt.show()
# You'll see a massive spike around September 2022 (her death)You can also analyze who edits the article:
10.5 Getting Pageviews
The Wikimedia REST API (separate from the MediaWiki Action API) provides article pageview data — how many times an article was accessed across web and mobile platforms:
def get_pageviews(title, start="20230101", end="20231231", headers=HEADERS):
"""Get daily pageviews for a Wikipedia article.
Parameters
----------
title : str
Article title with underscores for spaces
start, end : str
Date range in YYYYMMDD format
headers : dict, optional
HTTP headers; defaults to the module-level HEADERS
Returns
-------
pd.DataFrame
DataFrame with daily pageview counts
"""
url = (
f"https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article"
f"/en.wikipedia/all-access/all-agents/{title}/daily/{start}/{end}"
)
response = requests.get(url, headers=headers)
response.raise_for_status() # Stop on HTTP errors
data = response.json()
df = pd.DataFrame(data["items"])
df["timestamp"] = pd.to_datetime(df["timestamp"], format="%Y%m%d00")
return df
views = get_pageviews("Elizabeth_II", start="20220101", end="20221231")
plt.figure(figsize=(14, 5))
plt.plot(views["timestamp"], views["views"])
plt.title("Daily Pageviews: Elizabeth II (2022)")
plt.ylabel("Views")
plt.axvline(pd.Timestamp("2022-09-08"), color="red", linestyle="--", label="Death (Sep 8)")
plt.legend()
plt.tight_layout()
plt.show()10.6 Getting Links and Categories
The API exposes the internal structure of Wikipedia — the web of connections between articles:
def get_outlinks(title, headers=HEADERS):
"""Get all internal wikilinks from an article."""
url = "https://en.wikipedia.org/w/api.php"
params = {
"action": "query",
"titles": title,
"prop": "links",
"pllimit": "max",
"format": "json"
}
all_links = []
while True:
response = requests.get(url, params=params, headers=headers)
response.raise_for_status() # Stop on HTTP errors
data = response.json()
pages = data["query"]["pages"]
page_id = list(pages.keys())[0]
links = pages[page_id].get("links", [])
all_links.extend([link["title"] for link in links])
if "continue" in data:
params.update(data["continue"])
else:
break
return all_links
outlinks = get_outlinks("Elizabeth_II")
print(f"Outgoing wikilinks: {len(outlinks)}")
print(f"First 10: {outlinks[:10]}")You can do the same for external links (URLs that point outside Wikipedia), interlanguage links (corresponding articles in other language editions), and category memberships:
def get_categories(title, headers=HEADERS):
"""Get all categories an article belongs to."""
url = "https://en.wikipedia.org/w/api.php"
params = {
"action": "query",
"titles": title,
"prop": "categories",
"cllimit": "max",
"format": "json"
}
all_categories = []
while True:
response = requests.get(url, params=params, headers=headers)
response.raise_for_status() # Stop on HTTP errors
data = response.json()
pages = data["query"]["pages"]
page_id = list(pages.keys())[0]
cats = pages[page_id].get("categories", [])
all_categories.extend([c["title"] for c in cats])
if "continue" in data:
params.update(data["continue"])
else:
break
return all_categories
categories = get_categories("Elizabeth_II")
print(f"Categories: {len(categories)}")
for cat in categories[:10]:
print(f" {cat}")10.6.1 Interlanguage Links
One of the most revealing structural features of Wikipedia is which articles exist in which languages. The langlinks property tells you how many of Wikipedia’s 300+ language editions have a corresponding article for a given topic — and which languages those are.
def get_langlinks(title, headers=HEADERS):
"""Get all interlanguage links for an article.
Parameters
----------
title : str
Article title
headers : dict, optional
HTTP headers; defaults to the module-level HEADERS
Returns
-------
pd.DataFrame
DataFrame with language codes and article titles
"""
url = "https://en.wikipedia.org/w/api.php"
params = {
"action": "query",
"titles": title,
"prop": "langlinks",
"lllimit": "max",
"format": "json"
}
all_links = []
while True:
response = requests.get(url, params=params, headers=headers)
response.raise_for_status() # Stop on HTTP errors
data = response.json()
pages = data["query"]["pages"]
page_id = list(pages.keys())[0]
links = pages[page_id].get("langlinks", [])
all_links.extend(links)
if "continue" in data:
params.update(data["continue"])
else:
break
return pd.DataFrame(all_links)
langs = get_langlinks("Elizabeth_II")
print(f"Article exists in {len(langs)} language editions")
print(langs["lang"].head(20))The number of language editions an article appears in is itself a measure of global significance — or at least of global interest as perceived by Wikipedia’s volunteer editors. Articles about Western political leaders, major cities, and scientific concepts tend to appear in 100 or more language editions. Articles about topics primarily relevant to the Global South — local politicians, regional cultural practices, indigenous languages — may exist in only a handful. This asymmetry reflects the demographic composition of Wikipedia’s editor community, which skews heavily toward the Global North and toward speakers of European languages. When you use Wikipedia data for cross-cultural analysis, interlanguage link counts are a useful but imperfect proxy for a topic’s global visibility.
10.7 Getting User Contributions
You can also retrieve the editing history of individual Wikipedia users, which is useful for studying editor behavior and community dynamics:
def get_user_contributions(username, limit=50, headers=HEADERS):
"""Get recent contributions by a Wikipedia user."""
url = "https://en.wikipedia.org/w/api.php"
params = {
"action": "query",
"list": "usercontribs",
"ucuser": username,
"uclimit": limit,
"ucprop": "title|timestamp|sizediff|comment",
"format": "json"
}
response = requests.get(url, params=params, headers=headers)
response.raise_for_status() # Stop on HTTP errors
data = response.json()
return pd.DataFrame(data["query"]["usercontribs"])
# Get contributions from a prolific editor
contribs = get_user_contributions("Drmies", limit=100)
print(f"Recent edits: {len(contribs)}")
print(contribs[["title", "timestamp", "sizediff"]].head())10.8 Writing Raw API Requests from Documentation
The functions above packaged API requests for you. (Production code often goes a step further and uses wrapper libraries like mwclient or pywikibot, which handle continuation, throttling, and errors automatically — this chapter teaches raw requests deliberately so that you understand exactly what those wrappers do on your behalf.) But the transferable skill is writing requests from scratch using only the documentation. Here is the process:
- Go to the MediaWiki API documentation
- Find an endpoint you have not used (e.g.,
action=query&list=search) - Read its parameters: what is required, what is optional, what values are accepted
- Construct a request, make it, and inspect the response structure
# Example: search Wikipedia for articles about "web scraping"
url = "https://en.wikipedia.org/w/api.php"
params = {
"action": "query",
"list": "search",
"srsearch": "web scraping",
"srlimit": 10,
"srprop": "size|wordcount|timestamp|snippet",
"format": "json"
}
response = requests.get(url, params=params, headers=HEADERS)
data = response.json()
results = data["query"]["search"]
for r in results:
print(f"{r['title']} — {r['wordcount']} words, last edited {r['timestamp'][:10]}")The ability to go from documentation to working code is the skill that transfers to every API in Chapter 11, Chapter 12, and Chapter 13. The specifics change — different base URLs, different authentication methods, different response structures — but the pattern remains: read the docs, construct the request, parse the response.
10.9 A Second Case Study: World War II
To reinforce these patterns, try applying the same functions to a different article. The World War II article is one of the most-edited on Wikipedia:
ww2_revisions = get_revisions("World_War_II")
print(f"World War II revisions: {len(ww2_revisions)}")
ww2_views = get_pageviews("World_War_II", start="20230101", end="20231231")
ww2_links = get_outlinks("World_War_II")
print(f"World War II outlinks: {len(ww2_links)}")
# Compare with Elizabeth II
print(f"\nElizabeth II revisions: {len(df_revisions)}")
print(f"Elizabeth II outlinks: {len(outlinks)}")The comparison reveals how article structure and editing dynamics vary: a biographical article about a contemporary figure versus a historical event article that has been built over nearly two decades of collaborative editing.
10.9.1 Combining Endpoints
The real analytical power of the MediaWiki API emerges when you combine data from multiple endpoints to answer questions that no single endpoint can address. A natural question is whether editing activity and viewing activity are related: do articles get edited more during periods when they are being read more, or do edits happen independently of audience attention?
# Get monthly revision counts for Elizabeth II in 2022.
# Revision timestamps parse as timezone-aware UTC (their ISO strings end in "Z");
# drop the timezone so they align with the tz-naive pageview timestamps
df_revisions["timestamp"] = pd.to_datetime(df_revisions["timestamp"]).dt.tz_localize(None)
monthly_revs = (df_revisions
.set_index("timestamp")
.loc["2022"]
.resample("ME")
.size()
.reset_index(name="revisions"))
# Get monthly pageviews for the same period
views_2022 = get_pageviews("Elizabeth_II", start="20220101", end="20221231")
monthly_views = (views_2022
.set_index("timestamp")
.resample("ME")["views"]
.sum()
.reset_index())
# Dual-axis plot: revisions on left, pageviews on right
fig, ax1 = plt.subplots(figsize=(12, 5))
ax1.bar(monthly_revs["timestamp"], monthly_revs["revisions"],
width=20, alpha=0.7, color="steelblue", label="Revisions")
ax1.set_ylabel("Monthly Revisions", color="steelblue")
ax2 = ax1.twinx()
ax2.plot(monthly_views["timestamp"], monthly_views["views"],
color="darkred", linewidth=2, label="Pageviews")
ax2.set_ylabel("Monthly Pageviews", color="darkred")
plt.title("Elizabeth II: Editing Activity vs. Reading Activity (2022)")
fig.tight_layout()
plt.show()The September 2022 spike — corresponding to the death of Queen Elizabeth II — should appear in both time series, but their relationship across quieter months is less obvious. In general, major events drive both pageviews and revisions simultaneously, but routine editing often happens independently of audience attention. Many of Wikipedia’s most active editors work on articles regardless of whether anyone is reading them. This decoupling of production and consumption is a distinctive feature of Wikipedia’s volunteer model.
10.10 Exercises
Comparative revision analysis. Retrieve the revision histories for two related Wikipedia articles (e.g., “Democratic_Party_(United_States)” and “Republican_Party_(United_States)”). Compare the number of unique editors, the distribution of edits per editor (is editing concentrated or distributed?), and the pattern of monthly editing activity. Are there periods where editing spikes simultaneously on both articles?
Event detection with pageviews. Use the pageviews API to retrieve daily data for five related articles over a full year. Identify spikes in each article and map them to real-world events. Did all articles spike together (suggesting a common cause), or were there article-specific events? Create a multi-line plot with labeled annotations for the key events.
Link network. Retrieve the outgoing wikilinks from three articles on related topics. How much do their link sets overlap? What articles appear in all three link sets? This overlap analysis is a simple form of the network analysis described in Appendix C.
Raw API practice. Using only the MediaWiki API documentation at https://www.mediawiki.org/wiki/API:Main_page, construct a working request for an endpoint not covered in this chapter. Good candidates include
prop=extlinks(what external websites does an article link to?),list=recentchanges(what are the most recent edits across all of Wikipedia?),prop=redirects(what alternative titles redirect to an article?), orprop=pageviews(the Action API’s own view counts for the last 60 days — compare them against the REST endpoint you used earlier). Document your process: what parameters did you choose and why?Editor network. For a single article, identify the top 20 editors by revision count. Then use
get_user_contributions()to find what other articles these editors work on. Do they share other editing interests, or is this article their only point of overlap?Graduate extension (INFO 5617). Read Mesgari et al. (2015), a systematic review of the first decade of scholarly research on Wikipedia. Choose one research stream from the review — collaboration inequality, for instance — and reproduce a miniature version of its core measurement using this chapter’s helper functions, such as computing a Gini coefficient of edits per editor across three articles of your choosing. Then write a short reflection on how your measurement choices (which articles, what time window, whether anonymous editors count) would change your conclusions, and how the original studies in that stream handled the same choices.
10.12 Common Issues to Debug
- Pagination: The single most common mistake. Always check for
"continue"in the response. Without it, you get only the first batch (often 500 or fewer results out of thousands). - Nested JSON navigation: The MediaWiki API nests data several levels deep. Print
data.keys(), thendata["query"].keys(), thendata["query"]["pages"].keys()to navigate step by step. - URL encoding: Article titles with spaces use underscores in the API. For titles with special characters (parentheses, accented letters), use
urllib.parse.quote(). - Rate limiting: The MediaWiki Action API does not publish a fixed requests-per-second quota; instead, its etiquette guidelines ask clients to make requests serially (wait for each response before sending the next) and to use the
maxlagparameter for bulk work. The REST pageviews endpoints ask clients to stay under roughly 100 requests per second, but course code should stay far below that. The practical rule: serial requests, followcontinuetokens, and always send an informative User-Agent per the Wikimedia User-Agent policy — without one, you may be throttled or blocked. - Missing pages: If you request an article that does not exist, the page ID will be negative (e.g.,
-1). Always check for this before trying to access revision data.
10.13 Key Takeaways
APIs provide structured, documented, intentional access to web data. Learning to work with one API well gives you a transferable skill for any API: read the documentation, construct parameterized requests, handle pagination with continuation tokens, and parse nested JSON responses one level at a time. Wikipedia’s MediaWiki API demonstrates these patterns clearly and generously — it is a model of what open, mission-driven data access looks like. The key habits: always handle pagination, always use a meaningful User-Agent, always read the documentation before writing code, and always check the response structure before assuming its contents.
10.14 Further Reading
- MediaWiki API documentation: https://www.mediawiki.org/wiki/API:Main_page
- Wikimedia REST API (pageviews): https://wikimedia.org/api/rest_v1/
- Ford (2022) — Wikipedia’s role in the survival of facts in the digital age
- Mesgari et al. (2015) — systematic review of scholarly research on Wikipedia content
- Salganik (2018) — Chapter 2: Observing Behavior, on using digital trace data for research
Ford, Heather. 2022. Writing the Revolution: Wikipedia and the Survival of Facts in the Digital Age. MIT Press.
Mesgari, Mostafa, Chitu Okoli, Mohamad Mehdi, Finn Årup Nielsen, and Arto Lanamäki. 2015. “‘The Sum of All Human Knowledge’: A Systematic Review of Scholarly Research on the Content of Wikipedia.” Journal of the Association for Information Science and Technology 66 (2): 219–45. https://doi.org/10.1002/asi.23172.
Salganik, Matthew J. 2018. Bit by Bit: Social Research in the Digital Age. Princeton University Press. https://www.bitbybitbook.com/.
10.11 Social History and Public Interest
Wikipedia is the largest collaboratively-authored reference work in human history, maintained by tens of thousands of volunteer editors working in over 300 languages. Its commitment to open APIs reflects its broader mission: anyone can read, edit, and computationally access Wikipedia’s content. The Wikimedia Foundation explicitly positions API access as part of its commitment to free knowledge (Ford 2022).
But openness does not mean representativeness. Research has documented significant demographic skew in who edits Wikipedia — editors are disproportionately male, from the Global North, and technically fluent. This translates into content gaps: articles about women, people of color, and the Global South are systematically shorter, less detailed, and less well-referenced than articles about men and Western topics (Mesgari et al. 2015). The data you retrieve from Wikipedia’s API reflects these patterns. Ease of access does not guarantee completeness or neutrality — a principle that applies equally to every data source in this book.
Wikipedia’s gender gap illustrates how community demographics shape content. Research has consistently found that Wikipedia’s editor community is predominantly male, and this demographic skew is reflected in coverage — biographies of women are fewer, shorter, and more likely to mention marital status or family relationships than biographies of men. WikiProject Women in Red has worked systematically to address this gap, organizing edit-a-thons and maintaining lists of missing biographies. Data retrieved from the API reflects these structural patterns: any Wikipedia-based analysis inherits the biases of its volunteer editor community. Recognizing this is not a reason to avoid Wikipedia data but a reason to interpret it carefully.
Wikipedia also demonstrates what sustainable, mission-driven data access looks like. While commercial platforms have restricted and monetized their APIs (see Chapter 3), Wikipedia’s API has remained free and has expanded in functionality over time. This is possible because the Wikimedia Foundation is a nonprofit funded by donations, not advertising revenue. The organizational structure shapes the data access policies.