6  Parsing Static Web Pages

TipLearning Objectives

• Use browser developer tools to identify HTML elements containing target data
• Extract data from HTML tables manually using BeautifulSoup’s tag navigation
• Use pandas.read_html() as a shortcut for well-structured tables
• Write a custom HTML parser for non-tabular data using DOM tree navigation
• Abstract repeated scraping logic into reusable functions with error handling

TipCompanion Notebook

Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).

6.1 From HTML to Data

The overall goal of web scraping is converting data from one structured format — HTML, the language of web pages — into another structured format suitable for analysis: typically a pandas DataFrame, a CSV file, or a JSON document. This chapter teaches three strategies for that conversion, each appropriate for different situations and progressively more general.

HTML looks a great deal like the XML you learned to parse in Chapter 4, and the two are best understood as siblings: both descend from an older markup standard called SGML. But HTML is not well-formed XML — only its stricter cousin XHTML is — and browsers deliberately tolerate unclosed tags, missing quotes, and improperly nested elements. That tolerance for malformed markup is precisely why lenient HTML parsers like the html.parser backend you hand to BeautifulSoup exist, and it means the BeautifulSoup skills you already have transfer directly. The difference is that real-world HTML is far messier than the clean XML examples from the previous chapter. Web pages contain navigation bars, advertisements, tracking scripts, and deeply nested <div> tags that obscure the data you care about. Learning to cut through this noise is the core skill of this chapter.

6.2 Strategy 1: Manual Table Parsing

Many interesting datasets live in HTML tables — economic data, sports statistics, legislative records, demographic breakdowns. Let us extract box office revenue data from The Numbers, a popular source of movie financial data. Our target is the weekly chart for late December 2018 — the lucrative holiday corridor between Christmas and New Year’s. One hedge before we begin: sites reorganize their URLs and page structures over time, so the code below targets the page as it existed in mid-2026 (this chapter’s closing “A Note on Fragility” has more to say about living with that reality).

6.2.1 Retrieving and Inspecting

import requests
from bs4 import BeautifulSoup

url = "https://www.the-numbers.com/box-office-chart/weekly/2018/12/28"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

Before writing parsing code, inspect the page in your browser’s developer tools. Right-click on the table and select “Inspect” to see the HTML structure. You are looking for the <table> tag and, within it, <tr> (table row) and <td> (table data) tags.

# Find all tables on the page
tables = soup.find_all("table")
print(f"Number of tables found: {len(tables)}")
# Often there are navigation tables mixed in with data tables

# The main data table is usually the largest one
# Inspect each to find the right one
for i, table in enumerate(tables):
    rows = table.find_all("tr")
    print(f"Table {i}: {len(rows)} rows")

6.2.2 Extracting Row Data

Once you have identified the correct table, the next step is to navigate its structure tag by tag. Every <tr> (table row) contains a sequence of <td> (table data) or <th> (table header) cells, and BeautifulSoup lets you walk that hierarchy directly:

# Assuming the data table is tables[1] (verify by inspection)
data_table = tables[1]
rows = data_table.find_all("tr")

# The first row holds the column headers
header_row = rows[0]
header_cells = header_row.find_all(["td", "th"])
header = [cell.get_text(strip=True) for cell in header_cells]
print(header)
# ['Rank', '', 'Movie', 'Distributor', 'Gross', 'Change', ...]

# Inspect a data row the same way
data_row = rows[1]
cells = data_row.find_all(["td", "th"])
print([cell.get_text(strip=True) for cell in cells])
# ['1', '(1)', 'Aquaman', 'Warner Bros.', '$105,455,062', ...]

You might be tempted to shortcut this with data_row.text.split("\n") — grab all the text in the row at once and split it apart. Resist the temptation. Notice the empty string in the header list above: the site leaves the previous-rank column’s header blank. Splitting on newlines and filtering out empty strings silently drops cells like that one, and every value after it shifts one column to the left — your “Movie” column quietly fills with distributor names. Navigating cell by cell with find_all(["td", "th"]) preserves empty cells as empty strings, so each value stays aligned with its column. The extra line of code buys you correctness.

6.2.3 Building the DataFrame

# Loop through the data rows, extracting each cell's text
all_rows = []
for row in rows[1:]:  # Skip the header row
    cells = row.find_all(["td", "th"])
    values = [cell.get_text(strip=True) for cell in cells]

    # Guard against ragged rows: tables often embed spacer, ad, or
    # subtotal rows with a different number of cells. Skip any row
    # that doesn't match the header — and print it, so you can check
    # that you aren't discarding real data.
    if len(values) != len(header):
        print(f"Skipping row with {len(values)} cells: {values[:3]}")
        continue

    all_rows.append(values)

import pandas as pd

df = pd.DataFrame(all_rows, columns=header)
print(df.head())

This manual approach gives you complete control over the extraction process. The tradeoff is that it requires more code and careful attention to the specific HTML structure of each page.

6.3 Strategy 2: pandas.read_html()

For well-structured HTML tables, pandas provides a powerful shortcut. The read_html() function finds all tables on a page and converts each one to a DataFrame:

import pandas as pd

# Point read_html directly at a URL — no requests or BeautifulSoup needed
url = "https://en.wikipedia.org/wiki/2018_in_film"
tables = pd.read_html(url)

print(f"Number of tables found: {len(tables)}")
# Wikipedia pages often have many tables

You can specify which row contains the column headers and select the table you want by index:

# The highest-grossing films table might be at index 3 (verify by inspection)
films_df = pd.read_html(url, header=0)[3]
print(films_df.head())

The read_html function is remarkably powerful for its simplicity, but it has limitations. It struggles with tables that use colspan or rowspan attributes to merge cells — for example, a table where “Disney” spans two rows to cover multiple films. In those cases, you may need to fall back to manual parsing or clean up the DataFrame after extraction.

6.3.1 Cleaning Up read_html Results

In practice, read_html rarely produces a perfect DataFrame. Column names may contain whitespace or be split across multiple header rows. Numeric columns often parse as strings because of dollar signs, commas, or percentage symbols. Merged cells create NaN values that need to be filled. Here is a typical cleanup workflow:

# Retrieve and extract the table
tables = pd.read_html(url, header=0)
df = tables[3]  # Verify the correct index by inspection

# Drop rows that repeat the header (common in long Wikipedia tables).
# Do this BEFORE renaming: the embedded rows contain the *original*
# header text, so they must be compared against the original columns —
# after lowercasing, "Rank" in the data would never match "rank"
df = df[df.iloc[:, 0] != df.columns[0]]

# Rename columns to something usable
df.columns = [col.strip().lower().replace(" ", "_") for col in df.columns]

# Convert numeric columns that parsed as strings
df["worldwide_gross"] = pd.to_numeric(
    df["worldwide_gross"].str.replace(r"[\$,]", "", regex=True),
    errors="coerce"
)

# Fill NaN values from merged cells (forward-fill)
df["studio"] = df["studio"].ffill()

print(df.dtypes)
print(df.head())

The errors="coerce" argument in pd.to_numeric() is essential: it converts unparseable values to NaN instead of raising an error, which lets your cleanup pipeline continue even when some cells contain unexpected text. The forward-fill (ffill) operation handles the common Wikipedia pattern where a studio name spans multiple rows — the name appears once, and subsequent rows are blank until the next studio. You should expect to spend as much time cleaning read_html output as you spent extracting it — this is normal and not a sign that something went wrong. The function handles the hard part (parsing HTML tables); the cleanup is your job.

6.4 Strategy 3: Writing a Custom Parser

Many web pages present structured data without using HTML tables. Award nominees, product listings, news articles, and directory entries are often structured with <div> tags and CSS classes. Extracting this data requires navigating the HTML tree to find the right containers.

6.4.1 The Oscars Example

The Academy Awards website lists nominees and winners organized by category. The data is structured with nested <div> tags rather than <table> elements. Let us build a parser for it.

import requests
from bs4 import BeautifulSoup

url = "https://www.oscars.org/oscars/ceremonies/2024"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

6.4.2 Finding Reliable Anchors

The key challenge with non-tabular scraping is finding HTML elements that reliably contain your target data. Start by using the browser’s Inspector tool to examine the structure around one category:

# Look for grouping elements — the structure may use classes like
# "view-grouping" or similar containers for each award category
groupings = soup.find_all("div", class_="view-grouping")
print(f"Found {len(groupings)} category groupings")

# If the count doesn't match expectations (24 Oscar categories),
# you may need to navigate to a more specific parent element

6.4.3 Navigating the Tree for Specificity

If your initial find_all returns too many or too few results, you need to navigate the tree structure to find a more specific anchor. This involves going up to a unique parent and then searching down within it:

# Find the main content area first
content_area = soup.find("div", class_="field--name-field-award-categories")

if content_area:
    # Now search only within this specific container
    categories = content_area.find_all("div", class_="view-grouping")
    print(f"Categories within content area: {len(categories)}")
else:
    # Guard: if the anchor is missing, fall back to an empty list so
    # the code below still runs instead of crashing with a NameError.
    # An empty result is your cue to re-inspect the page structure.
    categories = []
    print("Content area not found — re-inspect the page in dev tools")

6.4.4 Extracting Category Data

For each category, extract the award name and the list of nominees:

results = []

for category in categories:
    # Extract the category name
    name_tag = category.find("div", class_="view-grouping-header")
    if not name_tag:
        continue
    category_name = name_tag.text.strip()
    
    # Extract individual nominees
    nominees = category.find_all("div", class_="views-row")
    for nominee in nominees:
        nominee_name = nominee.text.strip()
        results.append({
            "category": category_name,
            "nominee": nominee_name
        })

df = pd.DataFrame(results)
print(df.head(10))

6.4.5 CSS Selectors with select()

So far you have navigated the tree with find() and find_all(). BeautifulSoup offers a second navigation language you already half-know from browsing the web: CSS selectors, via the select() method. If you have ever written a stylesheet — or copied a selector out of your browser’s Inspector — the syntax will feel familiar. A bare name matches a tag, a leading . matches a class, a leading # matches an id, and whitespace between selectors matches descendants:

# A tag name selects all matching elements — like find_all("table")
tables = soup.select("table")

# .class selects by CSS class — like find_all(class_="view-grouping")
groupings = soup.select(".view-grouping")

# #id selects the element with that id — like find(id="main-content")
main = soup.select("#main-content")

# Descendant combinators chain a whole path in one expression.
# This single line is equivalent to the find() + find_all() pair
# we used to narrow down to categories within the content area:
selected = soup.select("div.field--name-field-award-categories div.view-grouping")
print(f"Categories via CSS selector: {len(selected)}")
# Categories via CSS selector: 24 — same result as before

select() always returns a list, like find_all(); its sibling select_one() returns the first match, like find(). Where select() shines is exactly the situation you just worked through: expressing “this element, inside that container” as one readable expression rather than a nested sequence of finds — and because the Inspector’s right-click “Copy selector” command produces CSS selectors, you can often paste the browser’s own description of an element straight into your code.

6.4.6 Abstracting into Functions

Once your parser works for one page, abstract it into a function you can reuse:

import time

def scrape_oscar_year(year, headers):
    """Scrape Oscar nominees for a given ceremony year.

    Parameters
    ----------
    year : int
        The ceremony year (e.g., 2024)
    headers : dict
        HTTP headers including User-Agent

    Returns
    -------
    pd.DataFrame
        DataFrame with columns: category, nominee, year
    """
    url = f"https://www.oscars.org/oscars/ceremonies/{year}"

    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()  # Raise exception for bad status codes
    except requests.RequestException as e:
        print(f"Error fetching {year}: {e}")
        return pd.DataFrame()

    soup = BeautifulSoup(response.text, "html.parser")

    # The same parsing logic as above, now bundled for reuse
    content_area = soup.find("div", class_="field--name-field-award-categories")
    if not content_area:
        print(f"No categories found for {year} — page structure may differ")
        return pd.DataFrame()

    rows = []
    for category in content_area.find_all("div", class_="view-grouping"):
        name_tag = category.find("div", class_="view-grouping-header")
        if not name_tag:
            continue
        for nominee in category.find_all("div", class_="views-row"):
            rows.append({
                "category": name_tag.get_text(strip=True),
                "nominee": nominee.get_text(strip=True),
                "year": year,
            })

    df = pd.DataFrame(rows)
    return df

Note what the function does not do: it never sleeps. Rate limiting belongs in the loop that calls the function — that way a single test call returns immediately, and the delay happens between requests, where it actually matters:

# Collect five ceremonies' worth of nominations
all_years = []
for year in range(2020, 2025):
    print(f"Scraping {year}...")
    year_df = scrape_oscar_year(year, headers)
    all_years.append(year_df)
    time.sleep(1)  # Pause between requests, not inside the function

oscars = pd.concat(all_years, ignore_index=True)
print(f"Collected {len(oscars)} nominations across {oscars['year'].nunique()} ceremonies")

A loop like this is exactly where the principles of Chapter 2 stop being abstract. Before pointing it at oscars.org — or any other site — check the site’s robots.txt, just as you practiced there: those rules govern the scraping you do in this chapter and the ones that follow, not just the theory you read in that one. And rather than re-implementing polite behavior each time, reuse the responsible_get() function you built in that chapter:

# From @sec-ethics: responsible_get() sets the course User-Agent
# "WebDataScience/1.0 (INFO 4617; brian.keegan@colorado.edu)", waits
# `delay` seconds after each request, and returns None on network
# errors. Swap it in for requests.get() + time.sleep() in any loop:
response = responsible_get(url, delay=1.0)

6.4.7 A Second Parser: Colorado Legislators

Let us build a second parser for a very different page structure. The Colorado General Assembly website at leg.colorado.gov/legislators lists all current state legislators with their names, party affiliations, districts, and chambers:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = "https://leg.colorado.gov/legislators"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

# Inspect the page structure — legislators are typically in repeating containers
# The exact class names may change; always verify with the Inspector
containers = soup.find_all("div", class_="views-row")
print(f"Found {len(containers)} legislator entries")

For each container, extract the relevant fields. The exact tag structure will vary, so the key skill here is using the Inspector to trace from the visible text on the page back to the HTML elements that contain it:

records = []
for container in containers:
    name_tag = container.find("a")
    party_tag = container.find("span", class_="party-initial")
    district_tag = container.find("span", class_="district-number")

    if name_tag:
        records.append({
            "name": name_tag.text.strip(),
            "party": party_tag.text.strip() if party_tag else None,
            "district": district_tag.text.strip() if district_tag else None,
        })

df = pd.DataFrame(records)
print(f"Extracted {len(df)} legislators")
print(df.groupby("party").size())

This parser illustrates an important principle: start simple and iterate. Your first attempt at identifying the right containers may return too many or too few results. Compare your count against what you see on the page. If you find 100 containers but the page shows 65 legislators, you are matching extra elements — narrow your search with a more specific class or a parent container. If you find 0, the page may use different class names than you expected, or the content may be loaded dynamically with JavaScript (in which case, see Chapter 8).

Note that the page may have changed since this was written — that is the fundamental challenge of web scraping. If the class names above do not match what you see in the Inspector, that is not a bug in this code; it is a feature of how the web works. The skill you are building is not memorizing specific class names but learning the process of inspection, hypothesis, and verification that lets you adapt to any page structure. Think of each parser as a hypothesis about the page’s structure — you test it, compare the output to what you expect, and refine until they match.

6.4.8 Error Handling Patterns

As you scrape more pages, you will encounter failures: pages that do not load, elements that are missing, or structures that do not match your expectations. A robust scraper handles these gracefully:

def safe_scrape(url, headers, parse_func):
    """Scrape a URL with comprehensive error handling.

    Parameters
    ----------
    url : str
        The URL to scrape
    headers : dict
        HTTP headers
    parse_func : callable
        A function that takes a BeautifulSoup object and returns data

    Returns
    -------
    list or None
        Parsed data, or None if the scrape failed
    """
    try:
        response = requests.get(url, headers=headers, timeout=30)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        print(f"Request failed for {url}: {e}")
        return None

    soup = BeautifulSoup(response.text, "html.parser")

    try:
        return parse_func(soup)
    except AttributeError as e:
        print(f"Parsing failed for {url}: {e}")
        print("The page structure may have changed — re-inspect with dev tools.")
        return None
    except Exception as e:
        print(f"Unexpected error for {url}: {e}")
        return None

The AttributeError catch is especially important: it fires when .find() returns None and you try to call .text on it — the single most common scraping error. By catching it explicitly, you get an informative message (“the page structure may have changed”) instead of a cryptic traceback. The broader Exception catch ensures your scraping loop continues even if an individual page has an unexpected problem.

This pattern separates the what (your parsing logic in parse_func) from the how (request handling, error recovery). You can write a clean parsing function that assumes everything works, then wrap it in safe_scrape for production use. This is the same separation-of-concerns principle you saw with the responsible_get() function in Chapter 2, and you will use it again in Chapter 7 when building pipelines that retrieve dozens or hundreds of pages.

WarningA Note on Fragility

Websites redesign their HTML regularly. The specific CSS classes and tag structures described here were accurate at the time of writing but may have changed since. If your parser stops working, the first step is always to re-inspect the page in your browser’s developer tools and identify what changed. This fragility is a fundamental characteristic of web scraping and a motivation for preferring APIs when they are available (see Chapter 3).

6.5 Exercises

1. Wikipedia table extraction. Use pd.read_html() to extract a table of your choice from Wikipedia (e.g., a list of countries by population, U.S. states by area, or planets in the solar system). Clean up the resulting DataFrame and create a visualization.

2. Custom parser. Write a custom BeautifulSoup parser for a non-tabular web page. Good candidates include a course catalog, a restaurant menu, or a list of elected officials. Document your element selection strategy: why did you choose these specific tags and classes?

3. Multi-page scraping. Extend your parser from Exercise 2 to handle multiple pages (e.g., multiple years, multiple categories, or paginated results). Use time.sleep() between requests, try/except for error handling, and a function to abstract the repeated logic.

4. Non-English Wikipedia. Use pd.read_html() to scrape a data table from a non-English Wikipedia edition (e.g., the French, Spanish, or Japanese Wikipedia). Does read_html handle non-ASCII characters correctly? What cleanup is needed compared to the English version? Write a brief analysis of any encoding issues you encounter.

5. Cache first, parse offline. Build a two-stage scraper. Stage one downloads the raw HTML for several pages (e.g., five Oscar ceremony years or five weekly box office charts) exactly once, saving each response to a local cache/ directory with a filename derived from the URL slug (e.g., cache/ceremonies-2024.html) and sleeping between downloads. Stage two is your parser, which reads only from the cached files and never touches the network. In a short paragraph, explain why this save-then-parse pattern matters: consider politeness (how many times does each server get hit while you debug?), reproducibility (what is your analysis anchored to after the site changes?), and iteration speed.

6. Graduate extension (INFO 5617). Read Fiesler et al. (2020), a study of how social media platforms’ Terms of Service regulate automated data collection. Then audit two websites you might realistically scrape for a research project: locate and read each site’s Terms of Service and robots.txt, and classify what each document says about scraping using the paper’s categories. Write a ~500-word memo comparing what the two sites permit, where the ToS and robots.txt agree or conflict, and how you would proceed as a researcher — including what the paper’s findings suggest about the gap between written policy and ethical practice.

TipMissing Manual Reference

For a refresher on writing functions, handling exceptions with try/except, and structuring reusable code, see Missing Manual Chapter 17: Scripting.

6.6 Social History and Public Interest

Web scraping has deep roots in data journalism. Organizations like ProPublica, The Markup, and the Investigative Reporters and Editors (IRE) have built some of their most important investigations on data scraped from public websites — court records, school inspection reports, environmental compliance databases, campaign finance disclosures. The skill of turning a web page into a dataset is, in this tradition, a form of civic empowerment: it enables citizens and journalists to analyze information that is nominally public but practically inaccessible in its raw HTML form.

NotePublic Interest Connection

Turning a web page into a dataset is a form of oversight (Chapter 3). When government agencies publish information as web pages but not as downloadable data, scraping bridges the gap between nominal transparency and practical accountability. The ability to parse a legislative directory into a DataFrame enables analysis of representation, diversity, and political geography that the agency itself may not provide.

The Colorado General Assembly website (leg.colorado.gov) provides a concrete example. Biographical information about legislators — their districts, party affiliations, committee assignments, and contact information — is published on the web, but not in a structured format you can download and analyze. Scraping this data into a DataFrame enables research on representation, diversity, and legislative behavior that would be impossible otherwise.

6.7 Common Issues to Debug

• Multiple tables on a page: read_html and find_all("table") return all tables, including navigation elements. Inspect each to find the one with your data.
• Character encoding: Watch for \xa0 (non-breaking spaces) and other Unicode characters in names and text. Use .strip() and .replace('\xa0', ' ').
• Elements that look unique but are not: Always verify your find_all count against the expected number of results before building your parser.
• Dynamic content: If requests + BeautifulSoup returns a nearly-empty page, the content may be loaded by JavaScript. See Chapter 8 for Selenium-based solutions.

6.8 Key Takeaways

HTML scraping follows a consistent pattern: retrieve the page, inspect it with developer tools, identify your target elements, extract the data, and convert it to a DataFrame. Start with read_html() for tables — it handles most simple cases. Fall back to manual BeautifulSoup parsing when you need more control. Write custom parsers for non-tabular data by navigating the DOM tree to find reliable anchors. Always abstract your parsing logic into functions, handle errors gracefully, and rate-limit your requests. Web pages change, so code defensively.

6.9 Further Reading

• pandas read_html documentation: https://pandas.pydata.org/docs/reference/api/pandas.read_html.html
• Mitchell (2018) — Chapters 2–5 on HTML scraping patterns
• Vanden Broucke and Baesens (2018) — best practices for production web scraping

Fiesler, Casey, Nathan Beard, and Brian C. Keegan. 2020. “No Robots, Spiders, or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social Media Terms of Service.” Proceedings of the International AAAI Conference on Web and Social Media 14: 187–96.

Mitchell, Ryan. 2018. Web Scraping with Python: Collecting More Data from the Modern Web. 2nd ed. O’Reilly Media.

Vanden Broucke, Seppe, and Bart Baesens. 2018. Practical Web Scraping for Data Science: Best Practices and Examples with Python. Apress.