1 Introduction to Web Data Science

Learning Objectives

Articulate what distinguishes web data science from general data science
Set up a reproducible computing environment with Anaconda, Python, and Jupyter Notebooks
Make a first HTTP request with the requests library and interpret the response
Locate, read, and apply library documentation as a core professional skill
Adopt the dispositions — growth mindset, computational thinking, hacker ethic, and scientific norms — that distinguish effective practitioners

Companion Notebook

Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).

1.1 Why Web Data Science?

The web is the largest, most dynamic, and most contested data source ever created. Every day, government agencies publish legislative records, researchers share datasets, journalists file stories, and billions of people generate behavioral traces on social platforms. This data can answer questions that matter: How does misinformation spread during elections? How have housing costs changed in your community? How do online communities govern themselves? How are platforms shaping public discourse?

But the web is not a database. It is a chaotic, evolving ecosystem of documents, APIs, protocols, and norms. Data on the web does not come in tidy CSV files ready for analysis. It comes as HTML tables embedded in pages full of tracking scripts, as JSON responses from APIs that require authentication and rate limiting, as scanned PDFs of city council meeting minutes, and as social media posts that might disappear tomorrow. Transforming this raw material into analyzable data requires a specific set of skills that sits at the intersection of programming, research design, and ethical judgment.

That is what this book teaches. Web data science is the practice of retrieving data from the web, parsing it into structured formats, and analyzing it to produce knowledge. It differs from general data science in that your first challenge is not modeling or visualization — it is getting the data in the first place.

1.2 The Data Science Mindset

Working with web data requires not just technical fluency but a professional disposition. Four components of this disposition will serve you throughout this book and your career.

Growth mindset means understanding that ability develops through effort. You will encounter libraries you have never used, error messages you do not understand, and web pages that resist your parsing attempts. This is normal. The goal is continual improvement, not instant mastery. Treat each frustration as a learning signal, not evidence of inadequacy.

Computational thinking is the disciplined approach that computer scientists bring to problems: decomposing large tasks into smaller pieces, recognizing patterns across examples, abstracting away inessential details, and specifying unambiguous sequences of steps (Wing 2006). When you face a complex scraping task, you will practice breaking it into retrieve, parse, clean, and analyze stages — each with clear inputs and outputs.

The hacker ethic emphasizes sharing, openness, creativity, and a bias toward action. The open-source libraries you will use throughout this book exist because communities of developers chose to share their work. You should approach problems with curiosity, experiment freely, and share what you learn.

Scientific norms ground your work in communalism, skepticism, and reproducibility. Document your methods so others can replicate them. Question your assumptions about the data. Communicate your findings honestly, including their limitations.

Missing Manual Reference

For a deeper discussion of computational thinking and its four components — decomposition, pattern recognition, abstraction, and algorithmic thinking — see Missing Manual Chapter 1: Introduction.

1.3 Setting Up Your Environment

You will need three things to work through this book: a Python installation, a way to run Jupyter Notebooks, and a few core libraries.

1.3.1 Installing Anaconda

The Anaconda distribution of Python provides a curated collection of scientific computing libraries along with the conda package manager. Download the latest version from anaconda.com and follow the installation instructions for your operating system.

After installation, open a terminal (macOS/Linux) or Anaconda Prompt (Windows) and update your installation:

# Run these commands in a terminal, not in Python
# conda update conda
# conda install anaconda=2024.06

Anaconda’s default base environment is fine for your first session, but as you accumulate projects, their library requirements will eventually conflict. The conda package manager solves this with environments: isolated Python installations, each with its own set of packages. Create one for this book and activate it whenever you sit down to work:

# Run these commands in a terminal, not in Python
# conda create -n webdata python=3.11
# conda activate webdata

Giving each project its own environment is a reproducibility practice, not just tidiness: it records exactly which Python and package versions your analysis ran against, so you — or someone replicating your work — can recreate the same setup on another machine months later.

Missing Manual Reference

If you run into installation issues, see Missing Manual Chapter 14: Package Management for troubleshooting conda and pip.

1.3.2 Jupyter Notebooks

Jupyter Notebooks are interactive computing environments that keep your code, documentation, and results in a single file. Launch one from the terminal with jupyter notebook, which will open a browser window where you can create and edit notebooks.

A notebook consists of cells. Code cells contain Python that you can execute with Shift+Enter. Markdown cells contain formatted text for documentation. The ability to interleave code with narrative explanation makes notebooks ideal for the kind of exploratory, iterative work that characterizes web data science.

# This is a code cell. When you run it, Python executes the code
# and displays any output below the cell.
print("Hello, web data science!")
# Output: Hello, web data science!

Missing Manual Reference

For a thorough introduction to Jupyter Notebooks — cell types, execution order, keyboard shortcuts, kernel management, and common pitfalls — see Missing Manual Chapter 16: Jupyter.

1.3.3 Core Libraries

This book uses several Python libraries extensively. Install them now if they are not already available in your Anaconda distribution:

# Run in a terminal:
# pip install requests beautifulsoup4 pandas matplotlib seaborn

Throughout this book, you will install additional libraries as needed for specific chapters. For now, confirm that the core libraries import without error:

import requests
import pandas as pd
import matplotlib.pyplot as plt
from bs4 import BeautifulSoup

print("All core libraries loaded successfully.")

1.4 Your First Request

Let us preview the loop that structures every chapter in this book: retrieve data from the web, parse it into a usable structure, and analyze it to produce insight.

The requests library lets you make HTTP requests — the same kind of request your browser makes when you type a URL. Here you will retrieve the Wikipedia article for the University of Colorado Boulder:

import requests

# Retrieve the page
url = "https://en.wikipedia.org/wiki/University_of_Colorado_Boulder"
response = requests.get(url)

# Check the response
print(response.status_code)  # 200 means success
print(type(response))        # <class 'requests.models.Response'>
print(len(response.text))    # Number of characters in the HTML

The response object contains everything the server sent back: a status code indicating success or failure, headers with metadata about the response, and the body of the document as a text string. You can inspect the first 500 characters of the HTML:

print(response.text[:500])
# You'll see raw HTML: <!DOCTYPE html>, <head>, <title>, etc.

This is raw HTML — not yet useful for analysis. In Chapter 4 and Chapter 6, you will learn to parse this HTML into structured data using BeautifulSoup. For now, the point is that retrieving a web page programmatically is as simple as requests.get(url).

You can also retrieve data in JSON format from an API. Here is a request to the Wikimedia pageviews API, which returns the number of times a Wikipedia article was viewed on a given day:

import requests

api_url = "https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article"
article = "University_of_Colorado_Boulder"
url = f"{api_url}/en.wikipedia/all-access/all-agents/{article}/daily/20240101/20240131"

headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)

data = response.json()  # Parse the JSON response into a Python dictionary
print(type(data))        # <class 'dict'>
print(data.keys())       # What keys are in the response?

Notice two things. First, you included a custom User-Agent header identifying yourself and providing contact information. This is a norm of responsible web data collection that you will learn more about in Chapter 2. Second, the .json() method converts the response body from a JSON string into a Python dictionary — a data structure you already know how to navigate.

1.4.1 From Response to DataFrame

The data["items"] key contains a list of dictionaries, one per day, each with fields like timestamp, views, article, and project. This is exactly the kind of structure that pandas can convert into a DataFrame with a single call:

import pandas as pd

# Convert the list of dictionaries into a DataFrame
df = pd.DataFrame(data["items"])
print(df.head())
# You'll see columns: project, article, granularity, timestamp, access, agent, views

# The timestamp column looks like "2024010100" — convert it to a proper date
df["date"] = pd.to_datetime(df["timestamp"], format="%Y%m%d00")

# Plot the time series
import matplotlib.pyplot as plt

plt.figure(figsize=(10, 4))
plt.plot(df["date"], df["views"])
plt.xlabel("Date")
plt.ylabel("Daily Pageviews")
plt.title(f"Wikipedia Pageviews: {article}")
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

This retrieve-parse-DataFrame-visualize pipeline is the fundamental pattern of this book. In every subsequent chapter, you will repeat these steps with different data sources: retrieve HTML pages, XML feeds, JSON API responses, or PDF documents; parse them into structured records; convert those records into a pandas DataFrame; and analyze or visualize the result. The specifics change — BeautifulSoup for HTML, json.loads() for JSON, regular expressions for messy text — but the arc remains the same. Recognizing this pattern early will help you approach each new chapter with a clear mental model of what you are trying to accomplish.

The DataFrame also gives you immediate analytical power. You can compute summary statistics with df["views"].describe(), identify the highest-traffic day with df.loc[df["views"].idxmax()], or resample the data to weekly totals. These operations are not the focus of this chapter, but they illustrate why getting data into a DataFrame is such a valuable intermediate step.

1.4.2 Saving Your Work

Once you have data in a DataFrame, save it to disk. This is not optional — it is a core practice of responsible web data science:

# Save to CSV
df.to_csv("cu_boulder_pageviews.csv", index=False)

# Later, load it back without making another API request
df = pd.read_csv("cu_boulder_pageviews.csv")
print(f"Loaded {len(df)} rows from disk.")

Serialization — saving your data to a file — matters for three reasons. First, courtesy: every API call consumes server resources, and re-requesting data you already have is wasteful. Second, reproducibility: if you save the data you retrieved on a specific date, your analysis can be reproduced even if the API changes or goes offline. Third, the data itself may change — Wikipedia pageview counts are finalized after a short delay, and an article’s content can be edited at any moment. The version you retrieved is a snapshot, and saving it preserves that snapshot. You will explore JSON serialization for more complex data structures in Chapter 4.

1.5 Documentation as Professional Practice

Your previous classes may have discouraged using online resources. The training wheels are off now. Finding, reading, interpreting, and applying documentation is an essential professional skill — not a shortcut, not cheating, but the primary way that working programmers, data scientists, and researchers learn to use new tools.

Bookmark the documentation for the libraries you will use throughout this book:

requests: requests.readthedocs.io
BeautifulSoup: crummy.com/software/BeautifulSoup/bs4/doc
pandas: pandas.pydata.org/docs
matplotlib: matplotlib.org/stable

When you encounter an error or do not know how to do something, start with the official documentation. If that does not help, search for the specific error message. When asking for help — from a classmate, an instructor, or an AI tool — “it’s not working” is not an actionable request. Describe what you tried, what you expected to happen, what actually happened, and what the error message says.

Missing Manual Reference

For strategies on asking effective technical questions and reading official documentation, see Missing Manual Chapter 2: Asking Technical Questions and Chapter 5: Reading Official Documentation.

1.6 Exercises

Environment check. Create a new Jupyter Notebook. In the first cell, import requests, pandas, matplotlib.pyplot, and BeautifulSoup. In the second cell, use requests.get() to retrieve any web page of your choice and print the status code. In the third cell, write a Markdown cell explaining what the status code means.
Exploring the Response object. Use the dir() function on a requests.Response object to list all its attributes and methods. Identify three attributes not discussed in this chapter and use the help() function or online documentation to figure out what they do. Write a brief explanation of each in a Markdown cell.
Your first API call. Using the Wikimedia pageviews API URL pattern from this chapter, retrieve the daily pageview data for a Wikipedia article of your choice over a month-long period. Convert the data['items'] list into a pandas DataFrame and create a simple line plot of views over time using matplotlib.
POST vs. GET. Read the requests library documentation and find the method for making a POST request. In a Markdown cell, explain how POST differs from GET and describe a scenario where you would use each. What kind of data does a POST request typically send, and why would an API require it instead of GET?
Multi-article comparison. Retrieve daily pageview data for three Wikipedia articles of your choice over the same month. Save each to a separate CSV file using df.to_csv(). Load all three back with pd.read_csv() and create a single plot with all three time series on the same axes, with a legend identifying each article. Which article had the most variable traffic, and can you hypothesize why?
Graduate extension (INFO 5617). Read the computational social science manifesto by Lazer et al. (2009) — a short, field-defining argument about what digital trace data makes possible. Identify three specific claims the authors make about the promise or the perils of studying society through digital traces, such as claims about data access, privacy, or the capacity of academic institutions to compete with industry. For each claim, write a paragraph connecting it to a concrete technique or data source covered in this book’s table of contents, and assess whether the claim has held up in the years since publication. Submit your response as a Jupyter Notebook that uses Markdown cells for the written analysis.

1.7 Social History and Public Interest

Web data science emerged at the intersection of several traditions. Computational social scientists in the 2000s and 2010s recognized that the web was generating behavioral data at unprecedented scale — traces of how people communicate, collaborate, and organize (Lazer et al. 2009). Data journalists discovered that web scraping could hold institutions accountable by making nominally public data actually analyzable. Civic technologists built tools to make government data accessible to citizens.

These traditions share a commitment to the idea that data fluency is a form of power, and that this power should serve the public interest. As you develop your skills throughout this book, you will encounter recurring questions about who benefits from web data collection, who is harmed by it, and what obligations come with the ability to automate access to information at scale.

Some of the most consequential data science work of the past decade has come from this tradition. ProPublica’s “Machine Bias” investigation in 2016 used scraped court records to reveal racial bias in criminal risk assessment algorithms — a finding that shaped national policy debates about algorithmic fairness. The Markup’s “Citizen Browser” project recruited a panel of real users to study how Facebook’s news feed algorithm distributed political content, producing evidence that would have been impossible to obtain through the platform’s official research tools. These projects combined technical scraping skills with domain expertise and ethical frameworks to produce knowledge that served the public interest.

The intellectual roots of web data science run deeper than any single investigation. The computational social science manifesto by Lazer et al. (2009) articulated a vision of research that leverages the “digital traces” left by online behavior — search queries, social media posts, hyperlink structures, editing histories — to answer questions at population scale. These traces are not designed for research; they are byproducts of ordinary activity. The researcher’s task is to transform them into meaningful evidence while respecting the privacy and dignity of the people who generated them. This tension between the analytical power of digital trace data and the ethical obligations it creates is a thread that runs through every chapter of this book.

Public Interest Connection

The tools you learn in this book are not neutral. Web data collection can serve openness — making public information practically accessible — or it can enable surveillance and exploitation. The ethical frameworks in Chapter 2 and the public interest framing in Chapter 3 will help you navigate this tension throughout the course.

1.8 Common Issues to Debug

ModuleNotFoundError: A library is not installed in your current Python environment. Run pip install library-name from a terminal, then restart your Jupyter kernel.
ConnectionError: Your computer cannot reach the server. Check your internet connection; if you are on a university network, try a different network.
Status code 403 (Forbidden): The server rejected your request, often because you did not include a User-Agent header. See Chapter 2 for how to set custom headers.
Cells running out of order: Jupyter cells can be executed in any order, which leads to stale variables. When in doubt, restart the kernel and run all cells from the top.

1.9 Key Takeaways

The web is a data source unlike any other — massive, dynamic, messy, and politically contested. Working with it requires not just code but a professional disposition: the habits of documentation, skepticism, and ethical reflection that distinguish a data scientist from someone who can run a script. Every technique in this book follows the same basic loop: retrieve data from the web, parse it into a usable structure, and analyze it to produce knowledge. You now have your environment set up and have made your first request. The rest of the book builds from here.

1.10 Further Reading

requests library documentation: https://requests.readthedocs.io/
Jupyter Notebook documentation: https://jupyter-notebook.readthedocs.io/
Wilson et al. (2017) — practical guidance on scientific computing workflows
Salganik (2018) — Chapter 1: Introduction, for an overview of social research in the digital age

Lazer, David, Alex Pentland, Lada Adamic, et al. 2009. “Computational Social Science.” Science 323 (5915): 721–23. https://doi.org/10.1126/science.1167742.

Salganik, Matthew J. 2018. Bit by Bit: Social Research in the Digital Age. Princeton University Press. https://www.bitbybitbook.com/.

Wilson, Greg, Jennifer Bryan, Karen Cranston, Justin Kitzes, Lex Nederbragt, and Tracy K. Teal. 2017. “Good Enough Practices in Scientific Computing.” PLOS Computational Biology 13 (6): e1005510. https://doi.org/10.1371/journal.pcbi.1005510.

Wing, Jeannette M. 2006. “Computational Thinking.” Communications of the ACM 49 (3): 33–35. https://doi.org/10.1145/1118178.1118215.

# Introduction to Web Data Science {#sec-introduction} ::: {.callout-tip} ## Learning Objectives - Articulate what distinguishes web data science from general data science - Set up a reproducible computing environment with Anaconda, Python, and Jupyter Notebooks - Make a first HTTP request with the `requests` library and interpret the response - Locate, read, and apply library documentation as a core professional skill - Adopt the dispositions — growth mindset, computational thinking, hacker ethic, and scientific norms — that distinguish effective practitioners ::: ::: {.callout-tip} ## Companion Notebook Run this chapter's code as you read: download the [companion notebook](notebooks/ch-01-introduction.ipynb) (see @sec-appendix-notebooks for all of them). ::: ## Why Web Data Science? The web is the largest, most dynamic, and most contested data source ever created. Every day, government agencies publish legislative records, researchers share datasets, journalists file stories, and billions of people generate behavioral traces on social platforms. This data can answer questions that matter: How does misinformation spread during elections? How have housing costs changed in your community? How do online communities govern themselves? How are platforms shaping public discourse? But the web is not a database. It is a chaotic, evolving ecosystem of documents, APIs, protocols, and norms. Data on the web does not come in tidy CSV files ready for analysis. It comes as HTML tables embedded in pages full of tracking scripts, as JSON responses from APIs that require authentication and rate limiting, as scanned PDFs of city council meeting minutes, and as social media posts that might disappear tomorrow. Transforming this raw material into analyzable data requires a specific set of skills that sits at the intersection of programming, research design, and ethical judgment. That is what this book teaches. Web data science is the practice of retrieving data from the web, parsing it into structured formats, and analyzing it to produce knowledge. It differs from general data science in that your first challenge is not modeling or visualization — it is *getting the data in the first place*. ## The Data Science Mindset Working with web data requires not just technical fluency but a professional disposition. Four components of this disposition will serve you throughout this book and your career. **Growth mindset** means understanding that ability develops through effort. You will encounter libraries you have never used, error messages you do not understand, and web pages that resist your parsing attempts. This is normal. The goal is continual improvement, not instant mastery. Treat each frustration as a learning signal, not evidence of inadequacy. **Computational thinking** is the disciplined approach that computer scientists bring to problems: decomposing large tasks into smaller pieces, recognizing patterns across examples, abstracting away inessential details, and specifying unambiguous sequences of steps [@wing2006computational]. When you face a complex scraping task, you will practice breaking it into retrieve, parse, clean, and analyze stages — each with clear inputs and outputs. **The hacker ethic** emphasizes sharing, openness, creativity, and a bias toward action. The open-source libraries you will use throughout this book exist because communities of developers chose to share their work. You should approach problems with curiosity, experiment freely, and share what you learn. **Scientific norms** ground your work in communalism, skepticism, and reproducibility. Document your methods so others can replicate them. Question your assumptions about the data. Communicate your findings honestly, including their limitations. ::: {.callout-tip} ## Missing Manual Reference For a deeper discussion of computational thinking and its four components — decomposition, pattern recognition, abstraction, and algorithmic thinking — see *Missing Manual* Chapter 1: Introduction. ::: ## Setting Up Your Environment You will need three things to work through this book: a Python installation, a way to run Jupyter Notebooks, and a few core libraries. ### Installing Anaconda The Anaconda distribution of Python provides a curated collection of scientific computing libraries along with the `conda` package manager. Download the latest version from [anaconda.com](https://www.anaconda.com/download) and follow the installation instructions for your operating system. After installation, open a terminal (macOS/Linux) or Anaconda Prompt (Windows) and update your installation: ```{python} # Run these commands in a terminal, not in Python # conda update conda # conda install anaconda=2024.06 ``` Anaconda's default `base` environment is fine for your first session, but as you accumulate projects, their library requirements will eventually conflict. The `conda` package manager solves this with *environments*: isolated Python installations, each with its own set of packages. Create one for this book and activate it whenever you sit down to work: ```{python} # Run these commands in a terminal, not in Python # conda create -n webdata python=3.11 # conda activate webdata ``` Giving each project its own environment is a reproducibility practice, not just tidiness: it records exactly which Python and package versions your analysis ran against, so you — or someone replicating your work — can recreate the same setup on another machine months later. ::: {.callout-tip} ## Missing Manual Reference If you run into installation issues, see *Missing Manual* Chapter 14: Package Management for troubleshooting `conda` and `pip`. ::: ### Jupyter Notebooks Jupyter Notebooks are interactive computing environments that keep your code, documentation, and results in a single file. Launch one from the terminal with `jupyter notebook`, which will open a browser window where you can create and edit notebooks. A notebook consists of cells. **Code cells** contain Python that you can execute with Shift+Enter. **Markdown cells** contain formatted text for documentation. The ability to interleave code with narrative explanation makes notebooks ideal for the kind of exploratory, iterative work that characterizes web data science. ```{python} # This is a code cell. When you run it, Python executes the code # and displays any output below the cell. print("Hello, web data science!") # Output: Hello, web data science! ``` ::: {.callout-tip} ## Missing Manual Reference For a thorough introduction to Jupyter Notebooks — cell types, execution order, keyboard shortcuts, kernel management, and common pitfalls — see *Missing Manual* Chapter 16: Jupyter. ::: ### Core Libraries This book uses several Python libraries extensively. Install them now if they are not already available in your Anaconda distribution: ```{python} # Run in a terminal: # pip install requests beautifulsoup4 pandas matplotlib seaborn ``` Throughout this book, you will install additional libraries as needed for specific chapters. For now, confirm that the core libraries import without error: ```{python} import requests import pandas as pd import matplotlib.pyplot as plt from bs4 import BeautifulSoup print("All core libraries loaded successfully.") ``` ## Your First Request Let us preview the loop that structures every chapter in this book: **retrieve** data from the web, **parse** it into a usable structure, and **analyze** it to produce insight. The `requests` library lets you make HTTP requests — the same kind of request your browser makes when you type a URL. Here you will retrieve the Wikipedia article for the University of Colorado Boulder: ```{python} import requests # Retrieve the page url = "https://en.wikipedia.org/wiki/University_of_Colorado_Boulder" response = requests.get(url) # Check the response print(response.status_code) # 200 means success print(type(response)) # <class 'requests.models.Response'> print(len(response.text)) # Number of characters in the HTML ``` The `response` object contains everything the server sent back: a status code indicating success or failure, headers with metadata about the response, and the body of the document as a text string. You can inspect the first 500 characters of the HTML: ```{python} print(response.text[:500]) # You'll see raw HTML: <!DOCTYPE html>, <head>, <title>, etc. ``` This is raw HTML — not yet useful for analysis. In @sec-data-formats and @sec-static-pages, you will learn to parse this HTML into structured data using BeautifulSoup. For now, the point is that retrieving a web page programmatically is as simple as `requests.get(url)`. You can also retrieve data in JSON format from an API. Here is a request to the Wikimedia pageviews API, which returns the number of times a Wikipedia article was viewed on a given day: ```{python} import requests api_url = "https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article" article = "University_of_Colorado_Boulder" url = f"{api_url}/en.wikipedia/all-access/all-agents/{article}/daily/20240101/20240131" headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"} response = requests.get(url, headers=headers) data = response.json() # Parse the JSON response into a Python dictionary print(type(data)) # <class 'dict'> print(data.keys()) # What keys are in the response? ``` Notice two things. First, you included a custom `User-Agent` header identifying yourself and providing contact information. This is a norm of responsible web data collection that you will learn more about in @sec-ethics. Second, the `.json()` method converts the response body from a JSON string into a Python dictionary — a data structure you already know how to navigate. ### From Response to DataFrame The `data["items"]` key contains a list of dictionaries, one per day, each with fields like `timestamp`, `views`, `article`, and `project`. This is exactly the kind of structure that pandas can convert into a DataFrame with a single call: ```{python} import pandas as pd # Convert the list of dictionaries into a DataFrame df = pd.DataFrame(data["items"]) print(df.head()) # You'll see columns: project, article, granularity, timestamp, access, agent, views # The timestamp column looks like "2024010100" — convert it to a proper date df["date"] = pd.to_datetime(df["timestamp"], format="%Y%m%d00") # Plot the time series import matplotlib.pyplot as plt plt.figure(figsize=(10, 4)) plt.plot(df["date"], df["views"]) plt.xlabel("Date") plt.ylabel("Daily Pageviews") plt.title(f"Wikipedia Pageviews: {article}") plt.xticks(rotation=45) plt.tight_layout() plt.show() ``` This retrieve-parse-DataFrame-visualize pipeline is the fundamental pattern of this book. In every subsequent chapter, you will repeat these steps with different data sources: retrieve HTML pages, XML feeds, JSON API responses, or PDF documents; parse them into structured records; convert those records into a pandas DataFrame; and analyze or visualize the result. The specifics change — BeautifulSoup for HTML, `json.loads()` for JSON, regular expressions for messy text — but the arc remains the same. Recognizing this pattern early will help you approach each new chapter with a clear mental model of what you are trying to accomplish. The DataFrame also gives you immediate analytical power. You can compute summary statistics with `df["views"].describe()`, identify the highest-traffic day with `df.loc[df["views"].idxmax()]`, or resample the data to weekly totals. These operations are not the focus of this chapter, but they illustrate why getting data into a DataFrame is such a valuable intermediate step. ### Saving Your Work Once you have data in a DataFrame, save it to disk. This is not optional — it is a core practice of responsible web data science: ```{python} # Save to CSV df.to_csv("cu_boulder_pageviews.csv", index=False) # Later, load it back without making another API request df = pd.read_csv("cu_boulder_pageviews.csv") print(f"Loaded {len(df)} rows from disk.") ``` Serialization — saving your data to a file — matters for three reasons. First, courtesy: every API call consumes server resources, and re-requesting data you already have is wasteful. Second, reproducibility: if you save the data you retrieved on a specific date, your analysis can be reproduced even if the API changes or goes offline. Third, the data itself may change — Wikipedia pageview counts are finalized after a short delay, and an article's content can be edited at any moment. The version you retrieved is a snapshot, and saving it preserves that snapshot. You will explore JSON serialization for more complex data structures in @sec-data-formats. ## Documentation as Professional Practice Your previous classes may have discouraged using online resources. The training wheels are off now. Finding, reading, interpreting, and applying documentation is an essential professional skill — not a shortcut, not cheating, but the primary way that working programmers, data scientists, and researchers learn to use new tools. Bookmark the documentation for the libraries you will use throughout this book: - **requests**: [requests.readthedocs.io](https://requests.readthedocs.io/) - **BeautifulSoup**: [crummy.com/software/BeautifulSoup/bs4/doc](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - **pandas**: [pandas.pydata.org/docs](https://pandas.pydata.org/docs/) - **matplotlib**: [matplotlib.org/stable](https://matplotlib.org/stable/) When you encounter an error or do not know how to do something, start with the official documentation. If that does not help, search for the specific error message. When asking for help — from a classmate, an instructor, or an AI tool — "it's not working" is not an actionable request. Describe what you tried, what you expected to happen, what actually happened, and what the error message says. ::: {.callout-tip} ## Missing Manual Reference For strategies on asking effective technical questions and reading official documentation, see *Missing Manual* Chapter 2: Asking Technical Questions and Chapter 5: Reading Official Documentation. ::: ## Exercises 1. **Environment check.** Create a new Jupyter Notebook. In the first cell, import `requests`, `pandas`, `matplotlib.pyplot`, and `BeautifulSoup`. In the second cell, use `requests.get()` to retrieve any web page of your choice and print the status code. In the third cell, write a Markdown cell explaining what the status code means. 2. **Exploring the Response object.** Use the `dir()` function on a `requests.Response` object to list all its attributes and methods. Identify three attributes not discussed in this chapter and use the `help()` function or online documentation to figure out what they do. Write a brief explanation of each in a Markdown cell. 3. **Your first API call.** Using the Wikimedia pageviews API URL pattern from this chapter, retrieve the daily pageview data for a Wikipedia article of your choice over a month-long period. Convert the `data['items']` list into a pandas DataFrame and create a simple line plot of views over time using `matplotlib`. 4. **POST vs. GET.** Read the `requests` library documentation and find the method for making a POST request. In a Markdown cell, explain how POST differs from GET and describe a scenario where you would use each. What kind of data does a POST request typically send, and why would an API require it instead of GET? 5. **Multi-article comparison.** Retrieve daily pageview data for three Wikipedia articles of your choice over the same month. Save each to a separate CSV file using `df.to_csv()`. Load all three back with `pd.read_csv()` and create a single plot with all three time series on the same axes, with a legend identifying each article. Which article had the most variable traffic, and can you hypothesize why? 6. **Graduate extension (INFO 5617).** Read the computational social science manifesto by @lazer2009computational — a short, field-defining argument about what digital trace data makes possible. Identify three specific claims the authors make about the promise or the perils of studying society through digital traces, such as claims about data access, privacy, or the capacity of academic institutions to compete with industry. For each claim, write a paragraph connecting it to a concrete technique or data source covered in this book's table of contents, and assess whether the claim has held up in the years since publication. Submit your response as a Jupyter Notebook that uses Markdown cells for the written analysis. ## Social History and Public Interest Web data science emerged at the intersection of several traditions. Computational social scientists in the 2000s and 2010s recognized that the web was generating behavioral data at unprecedented scale — traces of how people communicate, collaborate, and organize [@lazer2009computational]. Data journalists discovered that web scraping could hold institutions accountable by making nominally public data actually analyzable. Civic technologists built tools to make government data accessible to citizens. These traditions share a commitment to the idea that data fluency is a form of power, and that this power should serve the public interest. As you develop your skills throughout this book, you will encounter recurring questions about who benefits from web data collection, who is harmed by it, and what obligations come with the ability to automate access to information at scale. Some of the most consequential data science work of the past decade has come from this tradition. ProPublica's "Machine Bias" investigation in 2016 used scraped court records to reveal racial bias in criminal risk assessment algorithms — a finding that shaped national policy debates about algorithmic fairness. The Markup's "Citizen Browser" project recruited a panel of real users to study how Facebook's news feed algorithm distributed political content, producing evidence that would have been impossible to obtain through the platform's official research tools. These projects combined technical scraping skills with domain expertise and ethical frameworks to produce knowledge that served the public interest. The intellectual roots of web data science run deeper than any single investigation. The computational social science manifesto by @lazer2009computational articulated a vision of research that leverages the "digital traces" left by online behavior — search queries, social media posts, hyperlink structures, editing histories — to answer questions at population scale. These traces are not designed for research; they are byproducts of ordinary activity. The researcher's task is to transform them into meaningful evidence while respecting the privacy and dignity of the people who generated them. This tension between the analytical power of digital trace data and the ethical obligations it creates is a thread that runs through every chapter of this book. ::: {.callout-note} ## Public Interest Connection The tools you learn in this book are not neutral. Web data collection can serve **openness** — making public information practically accessible — or it can enable surveillance and exploitation. The ethical frameworks in @sec-ethics and the public interest framing in @sec-post-api will help you navigate this tension throughout the course. ::: ## Common Issues to Debug - **`ModuleNotFoundError`**: A library is not installed in your current Python environment. Run `pip install library-name` from a terminal, then restart your Jupyter kernel. - **`ConnectionError`**: Your computer cannot reach the server. Check your internet connection; if you are on a university network, try a different network. - **Status code 403 (Forbidden)**: The server rejected your request, often because you did not include a `User-Agent` header. See @sec-ethics for how to set custom headers. - **Cells running out of order**: Jupyter cells can be executed in any order, which leads to stale variables. When in doubt, restart the kernel and run all cells from the top. ## Key Takeaways The web is a data source unlike any other — massive, dynamic, messy, and politically contested. Working with it requires not just code but a professional disposition: the habits of documentation, skepticism, and ethical reflection that distinguish a data scientist from someone who can run a script. Every technique in this book follows the same basic loop: retrieve data from the web, parse it into a usable structure, and analyze it to produce knowledge. You now have your environment set up and have made your first request. The rest of the book builds from here. ## Further Reading - `requests` library documentation: <https://requests.readthedocs.io/> - Jupyter Notebook documentation: <https://jupyter-notebook.readthedocs.io/> - @wilson2017good — practical guidance on scientific computing workflows - @salganik2018bit — Chapter 1: Introduction, for an overview of social research in the digital age