13 AI and Language Model APIs
- Set up authenticated access to LLM APIs (OpenAI, Anthropic)
- Construct chat completion requests with system, user, and assistant roles
- Use few-shot prompting and structured output to extract data from unstructured text
- Compute text embeddings for document similarity and clustering
- Assess the costs, reproducibility challenges, and appropriate uses of LLMs in research
Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).
13.1 LLMs as Web Data Science Tools
Large language model APIs represent a fundamentally new category of tool for web data science. They can transform the unstructured text outputs of Chapters 6–9 (scraped web pages, PDF extractions, archived documents) into structured, analyzable data — performing tasks like sentiment analysis, entity extraction, content coding, and summarization at scale.
This chapter treats LLMs as another API to learn: the same patterns of authentication, request construction, and response parsing apply. But LLMs have distinctive properties — non-determinism, cost, and limited reproducibility — that require careful handling.
13.2 The OpenAI API
13.2.1 Basic Chat Completion
Provider model lineups turn over every few months. The names used in this chapter — gpt-4o-mini, gpt-4o, and claude-sonnet-4-20250514 — were current when this book was written; check the providers’ model pages for what is current when you read it. For research code, pin the exact dated version and record it in your documentation — the reproducibility section later in this chapter explains why this matters.
13.2.2 Customizing Roles
The system message sets the assistant’s behavior. The user message is your input. The assistant message lets you simulate prior conversation turns:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a legislative analyst specializing in Colorado state policy."},
{"role": "user", "content": "Summarize the key provisions of this bill in three bullet points."},
{"role": "assistant", "content": "I'd be happy to analyze that bill. Could you share the text?"},
{"role": "user", "content": bill_text} # Variable containing bill text
]
)13.2.3 Structured Output and Few-Shot Prompting
For research applications, you typically want structured data, not free text. Use few-shot prompting — providing examples of the desired input-output format:
prompt = """Classify the following bill summary into one of these categories:
education, healthcare, environment, taxation, criminal_justice, other.
Examples:
Summary: "Increases funding for K-12 teacher salaries across the state."
Category: education
Summary: "Establishes new emission standards for power plants."
Category: environment
Now classify:
Summary: "{bill_summary}"
Category:"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Respond with only the category label, nothing else."},
{"role": "user", "content": prompt.format(bill_summary=bill_summary)}
],
temperature=0 # Reduce randomness for classification tasks
)
category = response.choices[0].message.content.strip()
print(f"Category: {category}")13.3 Text Embeddings
Embeddings are vector representations that encode the semantic meaning of text. You can use them to measure document similarity:
def get_embedding(text, model="text-embedding-3-small"):
"""Get the embedding vector for a text string."""
response = client.embeddings.create(input=text, model=model)
return response.data[0].embedding
# Compute embeddings for bill summaries
import numpy as np
from scipy.spatial.distance import cosine
bill_summaries = ["Summary of bill 1...", "Summary of bill 2...", "Summary of bill 3..."]
embeddings = [get_embedding(s) for s in bill_summaries]
# Compute pairwise similarity
embedding_matrix = np.array(embeddings)
n = len(bill_summaries)
similarity_matrix = np.zeros((n, n))
for i in range(n):
for j in range(n):
similarity_matrix[i][j] = 1 - cosine(embedding_matrix[i], embedding_matrix[j])
# Visualize as a heatmap
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize=(6, 5))
sns.heatmap(similarity_matrix, annot=True, fmt=".2f", cmap="YlOrRd",
xticklabels=range(1, n+1), yticklabels=range(1, n+1))
plt.title("Bill Summary Similarity (Cosine)")
plt.tight_layout()
plt.show()13.4 Applied Example: Legislative Bill Analysis
Let us apply these tools to a concrete web data science task. Suppose you have scraped bill summaries from the Colorado General Assembly website (building on the scraping techniques from Chapter 6). You want to classify each bill by policy area, extract the key stakeholders mentioned, and measure how similar bills are to each other.
13.4.1 Step 1: Classifying Bills
def classify_bill(summary, client):
"""Classify a bill summary into a policy category.
Parameters
----------
summary : str
The text of the bill summary
client : OpenAI
An authenticated OpenAI client
Returns
-------
str
The policy category
"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": (
"You are a legislative analyst. Classify bill summaries into "
"exactly one category: education, healthcare, environment, "
"taxation, criminal_justice, transportation, housing, labor, "
"technology, or other. Respond with only the category label."
)
},
{"role": "user", "content": summary}
],
temperature=0
)
return response.choices[0].message.content.strip().lower()
# Apply across a list of bills
import time
bills = [
{"id": "HB24-1001", "summary": "Increases funding for K-12 teacher salaries..."},
{"id": "HB24-1042", "summary": "Establishes new emission standards for power plants..."},
{"id": "SB24-087", "summary": "Requires health insurers to cover annual mental health..."},
]
for bill in bills:
bill["category"] = classify_bill(bill["summary"], client)
print(f"{bill['id']}: {bill['category']}")
time.sleep(1) # Rate limit between API calls13.4.2 Step 2: Extracting Structured Information
You can use few-shot prompting to extract multiple fields at once:
import json
extraction_prompt = """Extract the following fields from this bill summary as JSON:
- primary_topic: one phrase describing the main topic
- stakeholders: list of groups affected
- fiscal_impact: "yes", "no", or "unclear"
- geographic_scope: "statewide", "local", or "specific_region"
Example input: "Requires all school districts to provide free breakfast and lunch to students."
Example output: {{"primary_topic": "school meals", "stakeholders": ["students", "school districts", "food service workers"], "fiscal_impact": "yes", "geographic_scope": "statewide"}}
Now extract from: "{summary}"
"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Respond with valid JSON only. No markdown formatting."},
{"role": "user", "content": extraction_prompt.format(summary=bills[0]["summary"])}
],
temperature=0
)
try:
extracted = json.loads(response.choices[0].message.content)
print(json.dumps(extracted, indent=2))
except json.JSONDecodeError:
print("Failed to parse response as JSON")
print(response.choices[0].message.content)13.4.3 Structured Outputs: Letting the API Enforce the Format
The try/except pattern above treats malformed JSON as a fact of life. It remains a good fallback — and it teaches you exactly what can go wrong — but for production work, current best practice is to have the API enforce the format for you. OpenAI’s Structured Outputs feature accepts a JSON Schema and guarantees that the response conforms to it:
# Define the schema the response must follow
bill_schema = {
"name": "bill_classification",
"strict": True,
"schema": {
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": ["education", "healthcare", "environment", "taxation",
"criminal_justice", "transportation", "housing",
"labor", "technology", "other"]
},
"fiscal_impact": {"type": "string", "enum": ["yes", "no", "unclear"]}
},
"required": ["category", "fiscal_impact"],
"additionalProperties": False
}
}
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Classify this Colorado bill summary."},
{"role": "user", "content": bills[0]["summary"]}
],
response_format={"type": "json_schema", "json_schema": bill_schema},
temperature=0
)
# The response is guaranteed to parse and to match the schema
result = json.loads(response.choices[0].message.content)
print(result)
# {'category': 'education', 'fiscal_impact': 'yes'}A lighter-weight option, response_format={"type": "json_object"} (JSON mode), guarantees syntactically valid JSON without enforcing a particular schema. Anthropic’s API achieves the same guarantee through tool use: you define a “tool” whose input schema is your desired output structure, and the model’s tool call is constrained to match it — see the Anthropic tool-use documentation for details. Either way, schema enforcement eliminates a whole class of parsing failures. Keep the try/except fallback in your repertoire for providers and models that offer no such guarantee.
13.4.4 Step 3: Measuring Similarity with Embeddings
Embeddings let you move beyond keyword matching to semantic similarity. Two bills about “school funding” and “educational resource allocation” would have low keyword overlap but high embedding similarity:
import numpy as np
from scipy.spatial.distance import cosine
def get_embeddings(texts, client, model="text-embedding-3-small"):
"""Get embedding vectors for a list of texts."""
embeddings = []
for text in texts:
response = client.embeddings.create(input=text, model=model)
embeddings.append(response.data[0].embedding)
time.sleep(0.5) # Rate limit
return np.array(embeddings)
# Get embeddings for all bill summaries
summaries = [b["summary"] for b in bills]
embedding_matrix = get_embeddings(summaries, client)
# Compute pairwise cosine similarity
n = len(summaries)
similarity = np.zeros((n, n))
for i in range(n):
for j in range(n):
similarity[i][j] = 1 - cosine(embedding_matrix[i], embedding_matrix[j])
# Visualize
import matplotlib.pyplot as plt
import seaborn as sns
labels = [b["id"] for b in bills]
plt.figure(figsize=(8, 6))
sns.heatmap(similarity, annot=True, fmt=".2f", cmap="YlOrRd",
xticklabels=labels, yticklabels=labels)
plt.title("Bill Summary Similarity (Cosine Distance of Embeddings)")
plt.tight_layout()
plt.show()13.5 Fine-Tuning for Domain-Specific Tasks
Sometimes a general model does not produce consistent results for your specific task. Fine-tuning lets you train the model on your own examples:
# Prepare training data in JSONL format
# Each line is a complete conversation demonstrating the desired behavior
training_data = [
{
"messages": [
{"role": "system", "content": "Classify Colorado bill summaries by policy area."},
{"role": "user", "content": "Increases the minimum wage to $15/hour by 2026."},
{"role": "assistant", "content": "labor"}
]
},
{
"messages": [
{"role": "system", "content": "Classify Colorado bill summaries by policy area."},
{"role": "user", "content": "Requires electric vehicle charging at new buildings."},
{"role": "assistant", "content": "environment"}
]
},
# ... at least 10 examples, ideally 50+
]
# Save to JSONL file
import json
with open("training_data.jsonl", "w") as f:
for example in training_data:
f.write(json.dumps(example) + "\n")
# Upload the training file
training_file = client.files.create(
file=open("training_data.jsonl", "rb"),
purpose="fine-tune"
)
# Start a fine-tuning job
job = client.fine_tuning.jobs.create(
training_file=training_file.id,
model="gpt-4o-mini-2024-07-18"
)
print(f"Fine-tuning job started: {job.id}")
print(f"Status: {job.status}")Fine-tuning takes time (minutes to hours depending on dataset size). You can check the status periodically and then use the fine-tuned model by specifying its ID in place of the base model name.
13.6 Cost Management
LLM API calls cost money. Before running a large batch, estimate the cost:
# Rough cost estimation for GPT-4o-mini
# Input: ~$0.15 per 1M tokens; Output: ~$0.60 per 1M tokens
# A typical bill summary is ~200 tokens; a classification response is ~5 tokens
num_bills = 500
input_tokens = num_bills * 200
output_tokens = num_bills * 5
estimated_cost = (input_tokens * 0.15 / 1_000_000) + (output_tokens * 0.60 / 1_000_000)
print(f"Estimated cost for classifying {num_bills} bills: ${estimated_cost:.4f}")
# Typically a few cents for small jobs, but can add up with larger modelsUse gpt-4o-mini for development and testing. Switch to gpt-4o only when you need higher quality and have validated your prompts on a smaller sample.
13.7 Reproducibility and Documentation
LLM-based research faces reproducibility challenges that are fundamentally different from those of traditional computational methods. When you write a Python script to parse HTML, anyone with the same input file will get the same output. LLMs break this guarantee in multiple ways. Even with temperature=0, model outputs can vary slightly between calls due to floating-point non-determinism in the inference process. More significantly, model versions change over time: a prompt that works reliably with GPT-4 in early 2024 may produce different results with the same model name in late 2024, because the underlying weights have been updated. And when a model is retired entirely — as happened to the original GPT-3 models in early 2024 — exact replication becomes impossible. These properties do not disqualify LLMs as research tools, but they demand a level of documentation that most researchers are not yet accustomed to providing.
At a minimum, every research project that uses LLM APIs should record the following for each API call: the exact model name and version (e.g., gpt-4o-mini-2024-07-18, not just gpt-4o-mini), the complete prompt text including the system message, all generation parameters (temperature, max_tokens, top_p, frequency_penalty), the date and time of the API call, and any post-processing or validation steps applied to the output. Store this metadata alongside your results, not in a methods section written months later from memory. A simple practice is to log each API call to a JSONL file that includes both the request parameters and the full response object — this creates an audit trail that supports both reproducibility and debugging.
This documentation burden connects to the broader norms of scientific reproducibility discussed in Appendix B. The AI disclosure framework presented there provides a structured template for reporting LLM use in research. The core principle is transparency: another researcher should be able to read your methods section and understand exactly what model, prompt, and parameters produced your results — even if they cannot reproduce them identically, they should be able to reproduce them approximately and assess whether the differences are meaningful. Treating LLM documentation as a first-class methodological concern — not an afterthought — is what separates rigorous research from ad hoc analysis.
13.8 The Anthropic API
Anthropic’s Claude provides an alternative LLM API with a similar interface:
# Install: pip install anthropic
from anthropic import Anthropic
client_anthropic = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY"))
response = client_anthropic.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=1024,
messages=[
{"role": "user", "content": "What is the AT Protocol in one sentence?"}
]
)
print(response.content[0].text)13.8.1 Comparing Model Outputs
When you use LLMs for research tasks like content classification, a critical question is whether your results depend on which model you chose. The answer, almost always, is yes — and the degree of that dependence matters for the credibility of your findings. You can test this directly by sending the same prompt to both APIs and comparing their responses.
# Send the same classification prompt to both models
bill_text = "Establishes new emission standards for power plants operating in Colorado."
system_prompt = (
"You are a legislative analyst. Classify this bill summary into exactly one "
"category: education, healthcare, environment, taxation, criminal_justice, "
"transportation, housing, labor, technology, or other. "
"Respond with only the category label."
)
# OpenAI classification
openai_response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": bill_text}
],
temperature=0
)
openai_label = openai_response.choices[0].message.content.strip()
# Anthropic classification
anthropic_response = client_anthropic.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=50,
system=system_prompt,
messages=[
{"role": "user", "content": bill_text}
]
)
anthropic_label = anthropic_response.content[0].text.strip()
print(f"OpenAI: {openai_label}") # environment
print(f"Anthropic: {anthropic_label}") # environment
print(f"Agreement: {openai_label == anthropic_label}")For straightforward cases like the example above, both models will likely agree. The disagreements emerge at the boundaries — a bill about “electric vehicle incentives” might be classified as environment by one model and transportation or taxation by the other. These boundary cases are not failures; they reflect genuine ambiguity in the classification scheme. But they mean that your results are partially an artifact of your model choice, not purely a reflection of the underlying data.
This makes validation against human judgment essential. If two state-of-the-art models disagree on 15 percent of your cases, you need to know whether one of them aligns better with how a domain expert would classify the same text. Running a cross-model comparison on a human-labeled sample is the minimum standard for any research that uses LLM-based classification. Treat the model as a measurement instrument, and like any instrument, it needs calibration.
For a deeper understanding of how language models work internally — tokens, embeddings, attention, context windows — see Missing Manual Chapter 36: How Language Models Work.
13.9 Exercises
Structured extraction. Use an LLM API to extract structured metadata (topic, stakeholders, geographic scope) from a set of at least 10 web-scraped documents. Design your prompt, test it, and assess the quality of the results.
Embedding analysis. Compute embeddings for a corpus of at least 20 related documents and use cosine similarity to identify clusters. Visualize the results. Do the clusters match your intuitive groupings?
Human-AI agreement. Design a classification task for a set of documents. Manually classify 20 documents, then have an LLM classify the same documents. Compute the agreement rate. Where does the LLM disagree with your judgment?
Cross-model comparison. Code 20 text samples manually according to a classification scheme of your choice. Have both GPT-4o-mini and Claude classify the same 20 samples. Compute the agreement rate between each model and your human labels, and between the two models. Where do the models disagree with each other, and what does that tell you about the task?
Hierarchical clustering. Generate embeddings for 30 Wikipedia article titles spanning three different topic areas (10 per topic). Use
scipy.cluster.hierarchyto create a dendrogram. Do the embeddings cluster by topic? How does the clustering compare to your intuitive groupings?Graduate extension (INFO 5617). Read Ziems et al. (2024), which evaluates whether LLMs can serve as annotators for computational social science. Design and run a small validation study of your own: choose a classification task relevant to your research interests, classify 50 items with an LLM, and independently hand-code 25 of them before looking at the model’s labels. Report your agreement rate (ideally alongside a chance-corrected measure such as Cohen’s kappa), characterize any systematic patterns in the disagreements, and assess whether the paper’s claims about zero-shot annotation hold for your task.
13.11 Common Issues to Debug
- API key exposure: Use environment variables. Never commit keys to version control.
- Rate limiting and costs: Estimate costs before large batch runs. Use
gpt-4o-minifor development,gpt-4ofor production. - JSON parsing failures: When requesting structured output, the model may not follow your format exactly. Use
try/exceptaround JSON parsing. - Model version changes: Pin your model version in code. Document which model you used in your methods section.
13.12 Key Takeaways
LLM APIs are powerful tools for processing web data, but they require careful handling: non-determinism, cost, and reproducibility challenges are real. Use them for tasks where their strengths — scale, multilingual capability, structured extraction — outweigh their weaknesses. Always validate outputs against manual inspection. And treat the API like any other: read the documentation, manage your keys, and parse your responses carefully.
13.13 Further Reading
- OpenAI API documentation: https://platform.openai.com/docs/
- Anthropic API documentation: https://docs.anthropic.com/
- Ziems et al. (2024) — LLMs and computational social science
13.10 Social History and Public Interest
LLMs have accelerated the enclosure dynamics described in Chapter 3 — the enormous value of user-generated content as training data has given platforms new incentives to restrict access. At the same time, LLMs provide researchers with powerful analytical tools that can partially offset restricted data access: if you can retrieve even a small sample of text, LLMs can help you code, classify, and extract information from it at scales that would be impossible manually (Ziems et al. 2024).
The key challenge is reproducibility. LLM outputs are non-deterministic (even with
temperature=0, there is some variation), model versions change, and the cost structure means that replication requires financial investment. These properties require new norms for documentation and disclosure — see Appendix B.There is also a cost-based access inequality shaping who can use these tools for research. Flagship models cost more than an order of magnitude more per token than the same providers’ small models — GPT-4o versus GPT-4o-mini, for example — creating a tier system where well-funded labs can afford the most capable models while under-resourced researchers are limited to cheaper, less capable alternatives. Open-weight models like LLaMA, Mistral, and Qwen represent a partial counterbalance — their weights are freely available for download and local inference. But running these models requires significant GPU infrastructure, creating a different kind of barrier rooted in compute rather than API fees. The result is a landscape where access to the most capable AI tools is stratified by institutional resources, raising questions about whose research questions get answered and whose analytical methods are considered state-of-the-art.
The cost structure of LLM APIs creates a form of oversight asymmetry (Chapter 3). Well-resourced organizations can afford to run large-scale analyses that smaller teams cannot replicate or verify. Open-weight models and transparent pricing help, but the fundamental tension between capability and accessibility remains unresolved.