6  Parsing Static Web Pages

TipLearning Objectives
  • Place a scraping project on a 2×2 spectrum of single vs. multiple sites and consistent vs. inconsistent design
  • Extract a page’s single well-formed table with pandas.read_html(), and pick the right table by hand when decoys share the page
  • Clean up a scraped table’s stray footnote markers, merged cells, and unconverted numeric columns
  • Parse a Wikipedia infobox’s label/value structure with BeautifulSoup, then re-verify and port that same strategy to another language edition
  • Write a custom parser for repeating card-style containers, and recognize when a second site’s cards require a second parser
  • Abstract repeated scraping logic into reusable functions with error handling
  • Recognize when a page’s content is not actually static, and know where to pick up from there
TipCompanion Notebook

Run this chapter’s code as you read: open the companion notebook (see Appendix A for all of them).

6.1 From HTML to Data

The overall goal of web scraping is converting data from one structured format — HTML, the language of web pages — into another structured format suitable for analysis: typically a pandas DataFrame, a CSV file, or a JSON document. How hard that conversion is depends less on how much data you want than on the shape of the page hiding it, and this chapter organizes that variety into eight strategies, each matched to a shape you will meet in the wild.

HTML looks a great deal like the XML you learned to parse in Chapter 4, and the two are best understood as siblings: both descend from an older markup standard called SGML. But HTML is not well-formed XML — only its stricter cousin XHTML is — and browsers deliberately tolerate unclosed tags, missing quotes, and improperly nested elements. That tolerance for malformed markup is precisely why lenient HTML parsers like the html.parser backend you hand to BeautifulSoup exist, and it means the BeautifulSoup skills you already have transfer directly. The difference is that real-world HTML is far messier than the clean XML examples from the previous chapter. Web pages contain navigation bars, advertisements, tracking scripts, and deeply nested <div> tags that obscure the data you care about. Learning to cut through this noise is the core skill of this chapter.

6.2 A Spectrum of Scraping Problems

Before you write a line of parsing code, it helps to place your project on two independent axes. The first is how many sites your data lives on: a single site, or several. The second is how consistent that site’s markup is: does every page you need follow the same template, or does the same page mix pieces built to different templates? Crossing those two axes gives you four quadrants, each with a real example:

Single site Multiple sites
Consistent design Box Office Mojo’s weekly box office chart — the same table shape every week English and French Wikipedia’s film infoboxes — the same MediaWiki platform and the same key/value layout, with different labels
Inconsistent design oscars.org’s ceremony pages — a winner’s card and a nominee’s card on the same page are not shaped the same The Numbers, Box Office Mojo, and Rotten Tomatoes — three real sites tracking the same movies with three unrelated HTML structures

Where your project lands changes how much code you write and how long that code keeps working. A single site with a consistent template needs one parser, written once, that covers every page you will ever fetch from it. Multiple sites with inconsistent designs need one parser per site, each translating that site’s own shape into a schema you define, reconciled together afterward.

This chapter works through eight concrete strategies, moving roughly from the easiest quadrant toward the hardest: the clean single table and the table hiding among decoys, for single, consistent pages that ship your data as a literal <table>; the table that needs cleanup, for a table whose values need work once you have it; the infobox pattern and porting the pattern to a new site, for the consistent-design quadrants on one Wikipedia edition and then another; cards that need to become rows, for a single site whose repeating containers were never tables at all; inconsistent cards, multiple sites, for the quadrant that forces you to write more than one parser and reconcile the results; and when “static” isn’t, the strategy that is really a boundary, marking where the other seven strategies stop applying because the page was never fully there to parse in the first place.

6.3 Strategy 1: The Clean Single Table

Start with the easiest case on the spectrum: a single site, a consistent template, and a page whose data lives in exactly one <table> with nothing else on the page competing for it. Box Office Mojo’s weekly box office charts are built this way, and the chart for the week ending December 28, 2018 — the lucrative corridor between Christmas and New Year’s — sits at a URL that follows the same pattern every week:

import pandas as pd

url = "https://www.boxofficemojo.com/weekend/2018W52/"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
tables = pd.read_html(url, storage_options=headers)

print(f"Number of tables found: {len(tables)}")
# Number of tables found: 1

df = tables[0]
print(df.head())

One call to pd.read_html(), no requests, no BeautifulSoup, no navigating a tree by hand — because there is exactly one table on the page, tables[0] is unambiguous, and the DataFrame it returns needs little more than a numeric-type pass before you can plot it. Hold onto how little code that took, because most pages are not this generous. The next strategy shows what the same weekly-chart idea looks like on a site that ships two tables on the same page instead of one.

6.4 Strategy 2: The Table Hiding Among Decoys

The Numbers publishes the same kind of weekly box office chart as Box Office Mojo, for the same weeks — but the page ships two <table> elements, not one: a full desktop layout and a separate mobile layout stacked beneath it. Pick tables[0] on faith and you have a fifty-fifty chance of parsing the wrong one; you have to inspect the page and confirm which table is the one you want before you write a line of extraction code. Let us extract box office revenue data from The Numbers for the same late-December 2018 chart. One hedge before we begin: sites reorganize their URLs and page structures over time, so the code below targets the page as it existed in mid-2026 (this chapter’s closing “A Note on Fragility” has more to say about living with that reality).

6.4.1 Retrieving and Inspecting

import requests
from bs4 import BeautifulSoup

url = "https://www.the-numbers.com/box-office-chart/weekly/2018/12/28"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

Before writing parsing code, inspect the page in your browser’s developer tools. Right-click on the table and select “Inspect” to see the HTML structure. You are looking for the <table> tag and, within it, <tr> (table row) and <td> (table data) tags.

# Find all tables on the page
tables = soup.find_all("table")
print(f"Number of tables found: {len(tables)}")
# Number of tables found: 2 -- a full desktop layout and a mobile layout

# The main data table is usually the largest one
# Inspect each to find the right one
for i, table in enumerate(tables):
    rows = table.find_all("tr")
    print(f"Table {i}: {len(rows)} rows")

6.4.2 Extracting Row Data

Once you have identified the correct table, the next step is to navigate its structure tag by tag. Every <tr> (table row) contains a sequence of <td> (table data) or <th> (table header) cells, and BeautifulSoup lets you walk that hierarchy directly:

# Assuming the data table is tables[1] (verify by inspection)
data_table = tables[1]
rows = data_table.find_all("tr")

# The first row holds the column headers
header_row = rows[0]
header_cells = header_row.find_all(["td", "th"])
header = [cell.get_text(strip=True) for cell in header_cells]
print(header)
# ['Rank', '', 'Movie', 'Distributor', 'Gross', 'Change', ...]

# Inspect a data row the same way
data_row = rows[1]
cells = data_row.find_all(["td", "th"])
print([cell.get_text(strip=True) for cell in cells])
# ['1', '(1)', 'Aquaman', 'Warner Bros.', '$105,455,062', ...]

You might be tempted to shortcut this with data_row.text.split("\n") — grab all the text in the row at once and split it apart. Resist the temptation. Notice the empty string in the header list above: the site leaves the previous-rank column’s header blank. Splitting on newlines and filtering out empty strings silently drops cells like that one, and every value after it shifts one column to the left — your “Movie” column quietly fills with distributor names. Navigating cell by cell with find_all(["td", "th"]) preserves empty cells as empty strings, so each value stays aligned with its column. The extra line of code buys you correctness.

6.4.3 Building the DataFrame

# Loop through the data rows, extracting each cell's text
all_rows = []
for row in rows[1:]:  # Skip the header row
    cells = row.find_all(["td", "th"])
    values = [cell.get_text(strip=True) for cell in cells]

    # Guard against ragged rows: tables often embed spacer, ad, or subtotal rows with a different number of cells. Skip any row that doesn't match the header — and print it, so you can check that you aren't discarding real data.
    if len(values) != len(header):
        print(f"Skipping row with {len(values)} cells: {values[:3]}")
        continue

    all_rows.append(values)

import pandas as pd

df = pd.DataFrame(all_rows, columns=header)
print(df.head())

This manual approach gives you complete control over the extraction process. The tradeoff is that it requires more code and careful attention to the specific HTML structure of each page. The next strategy goes back to pandas.read_html() — but this time the challenge is not finding the right table, since there is only one; it is cleaning up the one you found.

6.5 Strategy 3: The Table That Needs Cleanup

For well-structured HTML tables, pandas provides a powerful shortcut. The read_html() function finds all tables on a page and converts each one to a DataFrame:

import pandas as pd

# Point read_html directly at a URL — no requests or BeautifulSoup needed
url = "https://en.wikipedia.org/wiki/List_of_highest-grossing_films"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
tables = pd.read_html(url, header=0, storage_options=headers)

print(f"Number of tables found: {len(tables)}")
# Wikipedia pages often have many tables

Notice storage_options: without it, read_html requests the page anonymously, and Wikipedia answers an anonymous request with HTTP 403. It is the same courtesy Chapter 2 asked of every request you send, identifying yourself by name, passed through the one parameter read_html offers for it. Since that single call already fetched every table on the page, select the one you want by index instead of requesting the page again:

# The highest-grossing films table is at index 0
films_df = tables[0]
print(films_df.head())
# Columns: Rank, Peak, Title, Worldwide gross, Year, Ref
# Row 0 in October 2026: 1, 1, Avengers: Endgame †, $2,925,713,609, 2019, [# 1][# 2]

read_html is remarkably powerful for its simplicity, but it is not magic: it still struggles with colspan and rowspan — cells merged across rows or columns to avoid repeating a value, which read_html cannot always split back apart — and it never guesses at what a cell’s text is supposed to mean. This table shows a more common version of the second problem: every value it returned is technically present but not yet usable for arithmetic or plotting.

6.5.1 Cleaning Up read_html Results

Look at the raw Worldwide gross column before you clean anything, so you know exactly what you are stripping out:

print(films_df["Worldwide gross"].unique()[:10])

Most entries look like $2,923,710,708 — a dollar sign, digits, and commas. A handful carry a stray letter glued onto the front, left over from a footnote marker that never got separated from the number when the table was rendered: Titanic’s gross reads T$2,257,906,828, and Ne Zha 2’s reads NZ$2,217,758,382. The Ref column has its own quirk — footnote markers like [# 1][# 2] that read_html leaves as literal bracketed text rather than resolving into anything numeric. Here is a cleanup pass that handles both:

# Strip everything except digits -- removes the $ and the stray footnote letters in one pass
films_df["gross"] = pd.to_numeric(
    films_df["Worldwide gross"].str.replace(r"[^0-9]", "", regex=True),
    errors="coerce"
)

# Ref holds bracketed footnote markers like [# 3][# 4] -- count them instead of discarding them
films_df["footnote_count"] = films_df["Ref"].str.count(r"\[")

print(films_df[["Title", "gross", "footnote_count"]].head())

The errors="coerce" argument in pd.to_numeric() is essential: it converts anything the regex missed into NaN instead of raising an exception, so your cleanup pipeline continues past a handful of unexpected cells rather than crashing on the first one. Counting the footnote markers instead of throwing them away keeps a signal — how heavily annotated a given figure is — that a straight str.replace() would have discarded along with the brackets. This table’s columns happen not to need it, but the same .ffill() you would reach for on a merged-cell “Studio” column elsewhere works the same way here: propagate the last non-null value forward until the next one appears. You should expect to spend as much time cleaning read_html output as you spent extracting it — the function handles the hard part of parsing HTML tables; the cleanup is your job.

6.6 Strategy 4: The Infobox Pattern

Not every <table> holds rows of records. Wikipedia’s film infoboxes — the shaded box in the upper right of an article like The Godfather — are a single record laid out as a labeled key/value list: one row per fact, a <th> holding the label and a <td> holding the value. pandas.read_html() will happily parse an infobox into a two-column DataFrame, but that discards the very thing that makes it useful — which label belongs to which value — so this shape calls for BeautifulSoup instead:

import requests
from bs4 import BeautifulSoup

url = "https://en.wikipedia.org/wiki/The_Godfather"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

infobox = soup.find("table", class_="infobox")
facts = {}
for row in infobox.find_all("tr"):
    label, value = row.find("th"), row.find("td")
    if label and value:
        facts[label.get_text(strip=True)] = value.get_text(" ", strip=True)

print(facts["Directed by"])
# Francis Ford Coppola

The if label and value guard exists because not every <tr> in an infobox carries both cells: the top row is usually a single <th> spanning the film’s title with no matching <td>, and image rows carry a <td> with no <th> beside it. Skipping any row missing either half keeps facts limited to genuine key/value pairs. Once built, facts reads like the infobox looks: facts["Produced by"] is "Albert S. Ruddy", facts["Cinematography"] is "Gordon Willis", and facts["Screenplay by"] runs Mario Puzo and Francis Ford Coppola together into one string, because get_text(" ", strip=True) joins everything inside the cell with a single space rather than preserving whatever line breaks separated the names visually. facts["Starring"] does the same across the entire cast list — Marlon Brando, Al Pacino, James Caan, Richard Castellano, Robert Duvall, Sterling Hayden, and more — worth remembering if you later want to split it back into individual names.

6.7 Strategy 5: Porting the Pattern to a New Site

The loop above is barely ten lines, and the next step reuses it almost verbatim — the site changes, the labels change, but the shape does not. Point it at the French Wikipedia article for the same film and choose your URL carefully: https://fr.wikipedia.org/wiki/Le_Parrain (without (film)) is a disambiguation page, a “page d’homonymie” with no infobox and no tables at all. The article with the infobox lives at https://fr.wikipedia.org/wiki/Le_Parrain_(film).

Reach for the same class name you used in Strategy 4 there and it fails you in an unexpected way. The page does carry class="infobox_v3" — inspect it in a live browser and you can see it right there on the infobox — but soup.select("table.infobox_v3") still comes back empty. The class is real; it is just not on the <table>. French Wikipedia’s infobox template wraps the actual <table> in a <div class="infobox_v3 infobox infobox--frwiki ...">, and the table itself carries no class at all. Selecting by a class you saw in the Inspector only works if you also confirm which element that class sits on — the visual “infobox” you are looking at and the <table> holding its rows are not always the same node. What holds up here is the table’s own <caption>, one level down from the div you might have selected instead, which BeautifulSoup’s CSS-selector support can target directly:

url = "https://fr.wikipedia.org/wiki/Le_Parrain_(film)"
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

infobox = soup.select("table:has(caption)")[0]

facts = {}
for row in infobox.find_all("tr"):
    label, value = row.find("th"), row.find("td")
    if label and value:
        facts[label.get_text(strip=True)] = value.get_text(" ", strip=True)

print(facts["Réalisation"])
# Francis Ford Coppola

The loop is identical to Strategy 4 — same guard, same two-cell extraction — because the shape of the problem, a table of label/value pairs, is identical. What changed is the anchor and the labels: facts["Titre original"] is "The Godfather", facts["Scénario"] is "Mario Puzo Francis Ford Coppola", facts["Musique"] is "Nino Rota", facts["Acteurs principaux"] runs together Marlon Brando, Al Pacino, James Caan, Richard S. Castellano, and Robert Duvall, facts["Sociétés de production"] lists Paramount Pictures and Alfran Productions, facts["Pays de production"] is "États-Unis", and facts["Genre"] is "gangsters". Every one of those had to be re-verified against this specific page — the strategy transfers from one Wikipedia edition to another; the selector that anchors it does not.

6.8 Strategy 6: Cards That Need to Become Rows

Many web pages present structured data without using HTML tables at all. Award nominees, product listings, news articles, and directory entries are often structured as repeating <div> containers — cards, in the loose sense the industry uses the word — where each card is conceptually a row and the page as a whole is conceptually a table. Extracting this data means navigating the HTML tree to find the right containers instead of a <table> tag.

6.8.1 The Oscars Example

The Academy Awards website lists nominees and winners organized by category, structured with nested <div> tags rather than <table> elements — a good candidate for this strategy, with one complication before you can even start. As of this writing, oscars.org’s bot protection rejects a plain requests.get() with HTTP 403 — it also rejects a User-Agent spoofed to look like Chrome or Firefox, the same 403 either way. What it does not reject is a User-Agent that identifies you honestly, the way Chapter 2 asked you to identify yourself in the first place:

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get("https://www.oscars.org/oscars/ceremonies/2026", headers=headers)
print(response.status_code)
# 200

soup = BeautifulSoup(response.text, "html.parser")

Notice what the difference is not: it is not a more convincing disguise. A UA string that pretends to be a browser gets the same 403 as no UA at all — the WAF’s whole job is catching exactly that pretense. A UA string that plainly says who you are and how to reach you gets through. That is not a coincidence a scraper can rely on everywhere, and it is not a trick — it is the actual point of Chapter 2’s advice to identify yourself: a site operator evaluating traffic can tell a named, reachable requester from an anonymous one pretending to be a real browser, and can choose to treat them differently. Sometimes, as here, that choice runs in your favor.

6.8.2 Finding Reliable Anchors

The key challenge with non-tabular scraping is finding HTML elements that reliably contain your target data. Oscars.org runs on Drupal, a content-management system whose class names describe Drupal’s own data model rather than anything about awards — field--name-field-award-categories names a field, not a category — which is worth knowing before you go hunting the Inspector for something friendlier:

# The outer container that holds every award category on the page
area = soup.find("div", class_="field--name-field-award-categories")

# Each category is its own paragraph-type container within it
categories = area.find_all("div", class_="paragraph--type--award-category")
print(f"Found {len(categories)} categories")
# Found 24 categories

6.8.4 Extracting Category Data

For each category, extract the award name and every honoree’s type, entity, and film:

results = []

for cat in categories:
    name_tag = cat.find(class_="field--name-field-award-category-oscars")
    if not name_tag:
        continue
    category_name = name_tag.get_text(strip=True)

    for h in cat.find_all("div", class_="paragraph--type--award-honoree"):
        type_tag = h.find(class_="field--name-field-honoree-type")
        entity_tag = h.find(class_="field--name-field-award-entities")
        film_tag = h.find(class_="field--name-field-award-film")  # None if a card has no film field

        results.append({
            "category": category_name,
            "honoree_type": type_tag.get_text(strip=True) if type_tag else None,
            "entity": entity_tag.get_text(strip=True) if entity_tag else None,
            "film": film_tag.get_text(strip=True) if film_tag else None,
        })

df = pd.DataFrame(results)
print(df[df["category"] == "Actor in a Leading Role"].iloc[0])
# honoree_type              Winner
# entity        Michael B. Jordan
# film                     Sinners

The field--name-field-award-film lookup pairs each honoree with a film: Michael B. Jordan’s win names both the actor and the film he won it for, and Best Picture names the producers as the entity and the film, “One Battle after Another,” in the film field. The field names stay the same from category to category, but what they hold does not. International Feature Film puts the film, “Sentimental Value,” in the entity field and its country, Norway, in the film field, so a column called film can hold a country; Original Song puts the song’s title in the film field and its film and songwriters in the entity field. Every card on the 2026 page has all three fields, but nothing on the page promises that, so the guard lets film_tag come back None instead of crashing on a card without one.

6.8.5 CSS Selectors with select()

So far you have navigated the tree with find() and find_all(). BeautifulSoup offers a second navigation language you already half-know from browsing the web: CSS selectors, via the select() method. If you have ever written a stylesheet — or copied a selector out of your browser’s Inspector — the syntax will feel familiar. A bare name matches a tag, a leading . matches a class, a leading # matches an id, and whitespace between selectors matches descendants:

# A tag name selects all matching elements — like find_all("div")
divs = soup.select("div")

# .class selects by CSS class — like find_all(class_="paragraph--type--award-category")
categories_by_css = soup.select(".paragraph--type--award-category")

# #id selects the element with that id — like find(id="main-content")
main = soup.select("#main-content")

# Descendant combinators chain a whole path in one expression.
# This single line is equivalent to the find() + find_all() pair we used to narrow down to categories within the content area:
selected = soup.select("div.field--name-field-award-categories div.paragraph--type--award-category")
print(f"Categories via CSS selector: {len(selected)}")
# Categories via CSS selector: 24 — same result as before

select() always returns a list, like find_all(); its sibling select_one() returns the first match, like find(). Where select() shines is exactly the situation you just worked through: expressing “this element, inside that container” as one readable expression rather than a nested sequence of finds — and because the Inspector’s right-click “Copy selector” command produces CSS selectors, you can often paste the browser’s own description of an element straight into your code.

6.8.6 Abstracting into Functions

Once your parser works for one ceremony, abstract it into a function you can reuse across every year you want. This is exactly the situation responsible_get() from Chapter 2 was built for: a named, identified requester, one page at a time, with a pause built in between calls. Here it is again so this chapter’s code runs on its own:

import requests
import time
import logging

def responsible_get(url, headers=None, delay=1.0):
    """Make an HTTP GET request with responsible defaults.

    Parameters
    ----------
    url : str
        The URL to request
    headers : dict, optional
        Custom headers; a default User-Agent is added if none provided
    delay : float
        Seconds to wait after the request (default: 1.0)

    Returns
    -------
    requests.Response or None
        The response object, or None if the request failed
    """
    default_headers = {
        "User-Agent": "WebDataScience/1.0 (student-project; your-email@colorado.edu)"
    }
    if headers:
        default_headers.update(headers)

    try:
        response = requests.get(url, headers=default_headers, timeout=30)
        response.raise_for_status()  # Treat 4xx/5xx statuses as failures
        logging.info(f"{response.status_code} {url}")
        time.sleep(delay)
        return response
    except requests.exceptions.HTTPError as e:
        logging.error(f"HTTP error: {e}")
        time.sleep(delay)  # A failed request still counts against the server
        return None
    except requests.exceptions.RequestException as e:
        logging.error(f"Request failed ({type(e).__name__}): {url}")
        return None

With that defined, wrap the parser around it:

def parse_oscar_ceremony(year):
    """Fetch and parse one Oscars ceremony page into one row per honoree.

    Parameters
    ----------
    year : int
        The ceremony year to fetch and parse

    Returns
    -------
    pd.DataFrame
        DataFrame with columns: category, honoree_type, entity, film, year
    """
    url = f"https://www.oscars.org/oscars/ceremonies/{year}"
    response = responsible_get(url)
    if response is None:
        print(f"Request failed for {year}")
        return pd.DataFrame()

    soup = BeautifulSoup(response.text, "html.parser")

    area = soup.find("div", class_="field--name-field-award-categories")
    if not area:
        print(f"No categories found for {year} — page structure may differ")
        return pd.DataFrame()

    rows = []
    for cat in area.find_all("div", class_="paragraph--type--award-category"):
        name_tag = cat.find(class_="field--name-field-award-category-oscars")
        if not name_tag:
            continue
        category_name = name_tag.get_text(strip=True)

        for h in cat.find_all("div", class_="paragraph--type--award-honoree"):
            type_tag = h.find(class_="field--name-field-honoree-type")
            entity_tag = h.find(class_="field--name-field-award-entities")
            film_tag = h.find(class_="field--name-field-award-film")

            rows.append({
                "category": category_name,
                "honoree_type": type_tag.get_text(strip=True) if type_tag else None,
                "entity": entity_tag.get_text(strip=True) if entity_tag else None,
                "film": film_tag.get_text(strip=True) if film_tag else None,
                "year": year,
            })

    return pd.DataFrame(rows)

Collecting several ceremonies together is an ordinary loop over years, one request per iteration, with responsible_get()’s own delay doing the rate-limiting rather than a manual time.sleep() scattered through the loop:

all_years = [parse_oscar_ceremony(year) for year in range(2024, 2027)]
oscars = pd.concat(all_years, ignore_index=True)
print(f"Collected {len(oscars)} honorees across {oscars['year'].nunique()} ceremonies")

Nothing about this loop is special to oscars.org — it is the same shape as the Colorado Legislators parser below, and the same shape Chapter 2 used for its own examples. The only thing this site asked of you that a fully open site would not is honesty: identify yourself, and it lets you in.

6.8.7 A Second Parser: Colorado Legislators

The Oscars example is one instance of this strategy — cards standing in for rows — on one site. Let us apply the same strategy to a very different page, still on a single site, to see how much of it is the pattern and how much was specific to Drupal’s field names. The Colorado General Assembly website at leg.colorado.gov/legislators lists all current state legislators with their names, party affiliations, districts, and chambers, and — unlike oscars.org — does not run a WAF that cares who is asking; a bare requests.get() with no headers at all works here just as well as an identified one:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = "https://leg.colorado.gov/legislators"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

# Inspect the page structure — legislators are typically in repeating containers
# The exact class names may change; always verify with the Inspector
containers = soup.find_all("div", class_="views-row")
print(f"Found {len(containers)} legislator entries")

For each container, extract the relevant fields. The exact tag structure will vary, so the key skill here is using the Inspector to trace from the visible text on the page back to the HTML elements that contain it:

records = []
for container in containers:
    name_tag = container.find("a")
    party_tag = container.find("span", class_="party-initial")
    district_tag = container.find("span", class_="district-number")

    if name_tag:
        records.append({
            "name": name_tag.text.strip(),
            "party": party_tag.text.strip() if party_tag else None,
            "district": district_tag.text.strip() if district_tag else None,
        })

df = pd.DataFrame(records)
print(f"Extracted {len(df)} legislators")
print(df.groupby("party").size())

This parser illustrates an important principle: start simple and iterate. Your first attempt at identifying the right containers may return too many or too few results. Compare your count against what you see on the page. If you find 100 containers but the page shows 65 legislators, you are matching extra elements — narrow your search with a more specific class or a parent container. If you find 0, the page may use different class names than you expected, or the content may be loaded dynamically with JavaScript (in which case, see Chapter 8).

Note that the page may have changed since this was written — that is the fundamental challenge of web scraping. If the class names above do not match what you see in the Inspector, that is not a bug in this code; it is a feature of how the web works. The skill you are building is not memorizing specific class names but learning the process of inspection, hypothesis, and verification that lets you adapt to any page structure. Think of each parser as a hypothesis about the page’s structure — you test it, compare the output to what you expect, and refine until they match.

6.9 Strategy 7: Inconsistent Cards, Multiple Sites

Oscars’ award cards and the Colorado legislators’ cards are both single-site problems: write the parser once, and it covers the whole site. Move to a second site entirely and the same underlying idea — repeating containers standing in for table rows — reappears in a structurally unrelated shape. Rotten Tomatoes’ “Movies in Theaters” grid at https://www.rottentomatoes.com/browse/movies_in_theaters/ looks, at a glance, like the same kind of card layout as an Oscars category. It is not built the same way at all: Rotten Tomatoes’ grid uses custom HTML elements — tags the site defines itself, not <div> — so find_all("div", ...) returns nothing useful there:

import requests
from bs4 import BeautifulSoup

url = "https://www.rottentomatoes.com/browse/movies_in_theaters/"
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")

tiles = soup.find_all("media-info-tile")
print(f"Found {len(tiles)} tiles")
# Found 28 tiles (October 2026)

for tile in tiles:
    title = tile.find("rt-text", attrs={"data-qa": "discovery-media-list-item-title"})
    score = tile.find("rt-text", attrs={"slot": "criticsScore"})
    critics = score.get_text(strip=True) if score else ""
    print(title.get_text(strip=True), critics or "no Tomatometer yet")
# Casino Royale 94%

Each <media-info-tile> carries a poster, an <rt-text data-qa="discovery-media-list-item-title"> holding the title, and two more <rt-text> elements told apart by their slot attribute: criticsScore holds the Tomatometer as a percentage string, and audienceScore the audience’s. A film too new to have enough reviews has an empty criticsScore; on the October 2026 page, 16 of the 28 did. The page has other custom tiles too: find_all("tile-dynamic") matches a “Certified Fresh” list and news links, not the grid. Name the grid’s own element, and confirm in your browser’s Inspector that what you matched is what you meant. None of this resembles paragraph--type--award-honoree or the Colorado legislators’ views-row, and BeautifulSoup does not care: it navigates tile-dynamic and rt-text exactly as it would <div> and <span>, because it works on tag structure, not on a fixed vocabulary of “real” HTML tags.

This is where the 2×2 spectrum from the start of the chapter stops being an abstraction. A research question that needs both box-office performance and critical reception — whether a film’s opening-weekend take tracks its Tomatometer score, say — cannot be answered from one site: The Numbers and Box Office Mojo have the money figures, and Rotten Tomatoes has the reviews. This is the multiple-sites, inconsistent-design quadrant from the start of the chapter, and it is where most real scraping projects end up living. The fix is not a cleverer, more general parser. It is one parser per site, each translating that site’s own shape into a shared handful of columns — title, plus whatever that site alone provides — joined afterward on the one field every source shares: the movie’s title.

6.10 Strategy 8: When “Static” Isn’t

Every strategy so far assumes the HTML your code receives is the HTML a human sees — that the page is static in the sense this chapter needs, meaning fully formed before your request finishes. That assumption fails, and it fails without raising any error at all:

import requests

url = "https://www.imdb.com/title/tt0068646/"  # The Godfather
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(url, headers=headers)

print(response.status_code, len(response.text))
# 202 0

Status code 202 means the request was accepted, and the body that came back is zero characters long — not a truncated cast list, not a missing infobox, nothing at all for BeautifulSoup to parse. Open the same URL in an actual browser and the page is there in full: title, cast, ratings, everything. The difference is what runs between the request and the pixels on your screen. IMDb builds this page with JavaScript that executes after the initial HTML arrives, and requests never runs JavaScript — it only ever sees whatever the server sent in that first response, which here is next to nothing.

None of the eight strategies in this chapter close this gap, because none of them touch JavaScript execution; every one of them only ever operates on HTML that already exists in the response body. The tell is a response that comes back too fast, too short, or too empty relative to what the browser shows, and the fix is not a different BeautifulSoup pattern but a different tool: Chapter 8 covers Selenium and other approaches that drive an actual browser, wait for JavaScript to run, and hand you the HTML the human sees instead of the HTML the server first sent.

6.11 Error Handling Patterns

As you scrape more pages, you will encounter failures: pages that do not load, elements that are missing, or structures that do not match your expectations. A robust scraper handles these gracefully:

def safe_scrape(url, headers, parse_func):
    """Scrape a URL with comprehensive error handling.

    Parameters
    ----------
    url : str
        The URL to scrape
    headers : dict
        HTTP headers
    parse_func : callable
        A function that takes a BeautifulSoup object and returns data

    Returns
    -------
    list or None
        Parsed data, or None if the scrape failed
    """
    try:
        response = requests.get(url, headers=headers, timeout=30)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        print(f"Request failed for {url}: {e}")
        return None

    soup = BeautifulSoup(response.text, "html.parser")

    try:
        return parse_func(soup)
    except AttributeError as e:
        print(f"Parsing failed for {url}: {e}")
        print("The page structure may have changed — re-inspect with dev tools.")
        return None
    except Exception as e:
        print(f"Unexpected error for {url}: {e}")
        return None

The AttributeError catch is especially important: it fires when .find() returns None and you try to call .text on it — the single most common scraping error. By catching it explicitly, you get an informative message (“the page structure may have changed”) instead of a cryptic traceback. The broader Exception catch ensures your scraping loop continues even if an individual page has an unexpected problem.

This pattern separates the what (your parsing logic in parse_func) from the how (request handling, error recovery). You can write a clean parsing function that assumes everything works, then wrap it in safe_scrape for production use. This is the same separation-of-concerns principle you saw with the responsible_get() function in Chapter 2, and you will use it again in Chapter 7 when building pipelines that retrieve dozens or hundreds of pages.

WarningA Note on Fragility

Websites redesign their HTML regularly. The specific CSS classes and tag structures described here were accurate at the time of writing but may have changed since. If your parser stops working, the first step is always to re-inspect the page in your browser’s developer tools and identify what changed. The Oscars class names in this chapter were re-verified directly against the live 2026 ceremony page — a reminder that fragility is not only markup shifting under you, but sometimes a site’s own bot protection changing what “asking nicely” requires: oscars.org rejects a bare requests.get() and rejects a browser-spoofed one, but accepts a request that identifies itself honestly. If a site you are scraping starts returning 403 where it once returned 200, check whether the fix is a more honest User-Agent before assuming the door is closed for good. This fragility is a fundamental characteristic of web scraping and a motivation for preferring APIs when they are available (see Chapter 3).

6.13 Additional Exercises

These are open-ended extensions — no scaffold, no fixed path. Use them for further practice or deeper exploration.

  1. Custom parser. Write a custom BeautifulSoup parser for a non-tabular web page. Good candidates include a course catalog, a restaurant menu, or a list of elected officials. Document your element selection strategy: why did you choose these specific tags and classes?

  2. Multi-page scraping. Extend your parser from Exercise 1 to handle multiple pages (e.g., multiple years, multiple categories, or paginated results). Use time.sleep() between requests, try/except for error handling, and a function to abstract the repeated logic.

  3. Non-English Wikipedia. Use pd.read_html() to scrape a data table from a non-English Wikipedia edition (e.g., the French, Spanish, or Japanese Wikipedia). Does read_html handle non-ASCII characters correctly? What cleanup is needed compared to the English version? Write a brief analysis of any encoding issues you encounter.

  4. Cache first, parse offline. Build a two-stage scraper. Stage one downloads the raw HTML for several pages (e.g., five weekly box office charts, or five Oscars ceremony years) exactly once, saving each response to a local cache/ directory with a filename derived from the URL slug (e.g., cache/weekend-2018w52.html) and sleeping between downloads. Stage two is your parser, which reads only from the cached files and never touches the network. In a short paragraph, explain how this save-then-parse pattern serves politeness (how many times does each server get hit while you debug?), reproducibility (what is your analysis anchored to after the site changes?), and iteration speed.

  5. Reconcile two sites. Pick two sites that describe the same underlying things through unrelated HTML — box office figures from The Numbers or Box Office Mojo, paired with critical reception from Rotten Tomatoes, or a pairing of your own choosing. Write one parser per site, extract a field both can produce (a title, a name, a date), and join the two results into a single DataFrame. Report where the join fails to match rows cleanly, and explain why — mismatched capitalization, alternate titles, or one site covering entries the other does not.

  6. Graduate extension (INFO 5617). Read Fiesler et al. (2020), a study of how social media platforms’ Terms of Service regulate automated data collection. Then audit two websites you might realistically scrape for a research project: locate and read each site’s Terms of Service and robots.txt, and classify what each document says about scraping using the paper’s categories. Write a ~500-word memo comparing what the two sites permit, where the ToS and robots.txt agree or conflict, and how you would proceed as a researcher — including what the paper’s findings suggest about the gap between written policy and ethical practice.

TipMissing Manual Reference

For a refresher on writing functions, handling exceptions with try/except, and structuring reusable code, see Missing Manual Chapter 17: Scripting.

6.14 Social History and Public Interest

Web scraping has deep roots in data journalism. Organizations like ProPublica, The Markup, and the Investigative Reporters and Editors (IRE) have built some of their most important investigations on data scraped from public websites — court records, school inspection reports, environmental compliance databases, campaign finance disclosures. The skill of turning a web page into a dataset is, in this tradition, a form of civic empowerment: it enables citizens and journalists to analyze information that is nominally public but practically inaccessible in its raw HTML form.

NotePublic Interest Connection

Turning a web page into a dataset is a form of oversight (Chapter 3). When government agencies publish information as web pages but not as downloadable data, scraping bridges the gap between nominal transparency and practical accountability. The ability to parse a legislative directory into a DataFrame enables analysis of representation, diversity, and political geography that the agency itself may not provide.

The Colorado General Assembly website (leg.colorado.gov) provides a concrete example. Biographical information about legislators — their districts, party affiliations, committee assignments, and contact information — is published on the web, but not in a structured format you can download and analyze. Scraping this data into a DataFrame enables research on representation, diversity, and legislative behavior that would be impossible otherwise.

6.15 Common Issues to Debug

  • Multiple tables on a page: read_html and find_all("table") return all tables, including navigation elements. Inspect each to find the one with your data.
  • Character encoding: Watch for \xa0 (non-breaking spaces) and other Unicode characters in names and text. Use .strip() and .replace('\xa0', ' ').
  • Elements that look unique but are not: Always verify your find_all count against the expected number of results before building your parser.
  • Dynamic content: If requests + BeautifulSoup returns a nearly-empty page — or, as in Strategy 8, no page at all — the content is loaded by JavaScript after the initial response. See “Strategy 8: When ‘Static’ Isn’t” above and Chapter 8 for Selenium-based solutions.

6.16 Key Takeaways

HTML scraping follows a consistent pattern: retrieve the page, inspect it with developer tools, identify your target elements, extract the data, and convert it to a DataFrame. Where you start within that pattern depends on where your page sits on the chapter’s 2×2 spectrum. A single, consistent table calls for pd.read_html() and little else; a table hiding among decoys, or one whose values need cleanup after extraction, calls for the same tools plus more inspection and cleanup code. A labeled infobox calls for a key/value loop that ports from one language edition to another once you re-verify the anchor and labels. Repeating card-style containers — consistent on a single site, or inconsistent across several — call for a custom parser, one per site when the sites disagree, joined afterward into a shared schema. Always abstract your parsing logic into functions, handle errors gracefully, and rate-limit your requests. And when a page comes back nearly or entirely empty, these eight strategies have reached their edge — Chapter 8 picks up from there.

6.17 Further Reading

Fiesler, Casey, Nathan Beard, and Brian C. Keegan. 2020. “No Robots, Spiders, or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social Media Terms of Service.” Proceedings of the International AAAI Conference on Web and Social Media 14: 187–96.
Mitchell, Ryan. 2018. Web Scraping with Python: Collecting More Data from the Modern Web. 2nd ed. O’Reilly Media.
Vanden Broucke, Seppe, and Bart Baesens. 2018. Practical Web Scraping for Data Science: Best Practices and Examples with Python. Apress.