8  Dynamic Web Pages with Selenium

TipLearning Objectives
  • Explain why JavaScript-rendered content is invisible to requests + BeautifulSoup
  • Install and configure Selenium WebDriver for Chrome, Firefox, or Safari
  • Locate elements using multiple strategies (ID, name, XPath, CSS selector)
  • Simulate user interactions — clicking, typing, and scrolling — programmatically
  • Pass fully-rendered page source from Selenium back to BeautifulSoup for parsing
  • Assess the fragility and ethical implications of browser automation
TipCompanion Notebook

Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).

8.1 When Static Scraping Fails

In Chapter 6, you learned to retrieve web pages with requests and parse them with BeautifulSoup. This works well for pages whose content is fully present in the initial HTML — what we call static pages. But most modern websites are dynamic: their content is loaded, modified, or entirely rendered by JavaScript after the initial HTML arrives.

When you use requests.get() on a dynamic page, you get the HTML skeleton before JavaScript has run. The data you see in your browser may simply not exist in the response that requests receives. You can verify this by comparing requests.get(url).text with what you see in the browser — if significant content is missing from the requests version, the page is dynamic.

The solution is to use a tool that can execute JavaScript: a real web browser, controlled programmatically. Selenium is the standard tool for this.

8.2 Setting Up Selenium

Selenium requires two components: the selenium Python library and a browser driver — a small program that lets Python control a specific browser. Fortunately, you only need to install the library: since version 4.6 (late 2022), Selenium has shipped with Selenium Manager, which automatically finds or downloads the correct driver for whatever browser you have installed.

# Install from a terminal:
# pip install selenium

Creating a driver is then a one-liner:

from selenium import webdriver

# For Chrome:
driver = webdriver.Chrome()

# For Firefox:
# driver = webdriver.Firefox()

If driver resolution fails on your machine — an unusual browser install location or a very new browser release can confuse it — check Selenium’s driver documentation for manual setup instructions.

When you create the driver, a browser window opens. This is your programmable browser — every command you issue through the driver object happens in that window.

8.4 Extracting Data

xkcd is famous for its hidden alt-text messages. You can extract element attributes:

# Find the comic image
img = driver.find_element(By.XPATH, "//div[@id='comic']//img")

# Get the alt-text and title attributes
print(f"Alt text: {img.get_attribute('alt')}")
print(f"Title (hover text): {img.get_attribute('title')}")

8.5 Simulating Interactions

Selenium can simulate clicks, keyboard input, and scrolling — anything a human user would do:

8.5.1 Clicking

# Click the "Random" button to navigate to a random comic
random_button = driver.find_element(By.XPATH, "//ul[@class='comicNav']//a[contains(text(),'Random')]")
random_button.click()

# Extract the alt-text from the random page
import time
time.sleep(1)  # Wait for the page to load

img = driver.find_element(By.XPATH, "//div[@id='comic']//img")
print(f"Random comic title: {img.get_attribute('title')}")

8.5.2 Typing and Searching

Selenium can type into forms and submit them. We will demonstrate with Wikipedia’s search box: Wikipedia’s robots policy is permissive toward automated access, and its markup is stable enough that this example should keep working for years. (Search engines like Google, by contrast, explicitly prohibit automated queries in their Terms of Service — where you point your automation matters as much as how you write it.)

from selenium.webdriver.common.keys import Keys

# Navigate to the Wikipedia portal
driver.get("https://www.wikipedia.org")

# Find the search box by its name attribute
search_box = driver.find_element(By.NAME, "search")

# Type a query
search_box.send_keys("Colorado Buffaloes")
time.sleep(1)  # Observe autocomplete suggestions

# Press Enter to submit the search
search_box.send_keys(Keys.RETURN)
time.sleep(2)  # Wait for the results page to load

# Read the resulting page
print(driver.title)
# Colorado Buffaloes - Wikipedia

heading = driver.find_element(By.ID, "firstHeading")
print(heading.text)
# Colorado Buffaloes

When your query matches an article title exactly, Wikipedia takes you straight to that article; otherwise you land on a search results page listing candidate matches. Either way, the pattern is the one you will reuse on any site with a search form: find the input element, type with send_keys(), submit with Keys.RETURN, wait for the new page, and read the results.

WarningAutomation Is Not a Loophole

Everything from Chapter 2 applies with full force to browser automation. A robots.txt disallow rule does not stop mattering because a real browser is doing the requesting, and a site’s Terms of Service does not stop applying because a script is doing the clicking. Legal risk concentrates precisely where Selenium is most tempting: automating logged-in accounts and ToS-restricted content. Check the policies of any site before you automate interactions with it.

8.5.3 Scrolling

Some pages load content as you scroll (infinite scroll). You can simulate this:

from selenium.webdriver.common.keys import Keys

body = driver.find_element(By.TAG_NAME, "body")

# Scroll down 5 times
for i in range(5):
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(1)  # Wait for new content to load

8.5.4 Waiting for Dynamic Content

The time.sleep() calls in the examples above are a crude approach to waiting for content to load. A better strategy is explicit waits, which pause execution until a specific condition is met:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Navigate to a page with dynamically loaded content
driver.get("https://example-dynamic-site.com")

# Wait up to 10 seconds for a specific element to appear
element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "results-container"))
)
print(f"Element found: {element.tag_name}")

WebDriverWait checks the condition repeatedly (every 0.5 seconds by default) and returns the element as soon as it appears, or raises a TimeoutException if the timeout expires. This is superior to time.sleep() in two ways: it does not wait longer than necessary (if the element appears in 0.2 seconds, it proceeds immediately), and it fails loudly if the element never appears (rather than silently returning an empty page).

The most common expected conditions you will use are:

  • presence_of_element_located — the element exists in the DOM (even if not visible)
  • visibility_of_element_located — the element is both present and visible on the page
  • element_to_be_clickable — the element is visible, enabled, and can receive clicks
  • text_to_be_present_in_element — specific text has appeared inside an element

Use these instead of time.sleep() whenever possible. The combination of WebDriverWait with explicit conditions makes your Selenium scripts both more reliable (they wait for exactly what they need) and faster (they proceed as soon as the condition is met rather than waiting a fixed duration). You will see this pattern used in the practical workflow section later in this chapter.

8.5.5 Scraping Infinite Scroll Pages

Many modern websites load content progressively as you scroll — social media feeds, image galleries, and product listings all use this pattern. The technique for scraping these pages involves scrolling to the bottom, waiting for new content to load, and repeating until no more content appears:

def scrape_infinite_scroll(driver, max_scrolls=20, scroll_pause=2):
    """Scroll an infinite-scroll page and collect all loaded content."""
    last_height = driver.execute_script("return document.body.scrollHeight")

    for i in range(max_scrolls):
        # Scroll to the bottom of the page
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(scroll_pause)

        # Check if the page grew
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            print(f"No new content after scroll {i+1}. Stopping.")
            break
        last_height = new_height
        print(f"Scroll {i+1}: page height grew to {new_height}px")

    # Return the fully-loaded page source
    return driver.page_source

The key insight is checking document.body.scrollHeight before and after each scroll. When the height stops growing, you have reached the end of the available content. Some sites use a “Load More” button instead of infinite scroll — for those, replace the scroll with a click on the button and watch for the button to disappear or become disabled.

A practical consideration: infinite scroll pages can contain thousands of items, and loading them all consumes significant memory in the browser process. Set a reasonable max_scrolls limit based on how much data you actually need. If you are collecting data for a class project, you rarely need more than a few hundred items — scrolling through an entire social media feed of 10,000 posts is both unnecessary and discourteous to the server. Remember the proportionality principle from Chapter 2: collect only the data you need for your research question.

8.6 Passing to BeautifulSoup

Once the page is fully rendered in the browser, you can extract the complete HTML and parse it with the familiar BeautifulSoup tools:

from bs4 import BeautifulSoup

# Get the fully-rendered page source from Selenium
html = driver.page_source
soup = BeautifulSoup(html, "html.parser")

# Now use BeautifulSoup as usual
links = soup.find_all("a", href=True)
print(f"Total links on page: {len(links)}")

This hybrid approach — Selenium for rendering, BeautifulSoup for parsing — gives you the best of both worlds. Selenium excels at navigating, clicking, scrolling, and waiting for JavaScript to execute. BeautifulSoup excels at searching and extracting data from HTML. By combining them, you avoid the awkwardness of using Selenium’s relatively limited element-finding API for complex extraction tasks while still getting access to the fully-rendered page content.

The workflow is always the same: use Selenium to get the page into the state you need (scrolled, clicked, searched, logged in), then hand off the rendered HTML to BeautifulSoup for extraction. Think of Selenium as the hands that navigate the browser and BeautifulSoup as the eyes that read the content. This division of labor keeps your code cleaner and more maintainable than trying to do everything with Selenium alone.

8.7 Always Close the Driver

When you are done, close the browser window:

driver.quit()

After calling quit(), any subsequent calls to the driver will raise an error. Use try/finally blocks to ensure cleanup:

try:
    driver.get("https://example.com")
    # ... scraping logic ...
finally:
    driver.quit()

8.8 The Fragility Problem

WarningWarning

Screen-scraping with Selenium is inherently fragile. The Twitter scraping examples that worked in the 2019 version of this course broke completely by 2024 because Twitter (now X) redesigned its HTML to use dynamically-generated class names and require authentication for basic browsing. This is not unusual — platforms regularly change their front-end code, sometimes specifically to resist automated access.

This fragility reinforces a principle from Chapter 3: when an API is available, use it. Selenium is a powerful last resort for cases where you ethically need data that no API provides, but it is slow, resource-intensive, and brittle compared to API access. The decision tree should be: API first → static scraping second → Selenium third.

Selenium is also not the only browser automation tool. Playwright, developed by Microsoft, is the modern alternative: it waits for elements automatically (eliminating much of the explicit-wait boilerplate you saw earlier), runs noticeably faster, and supports asynchronous operation for parallel scraping. If browser automation becomes a regular part of your work, Playwright is worth learning. That said, Selenium remains far more widespread, has a deeper reservoir of documentation and community answers, and is entirely adequate for coursework — which is why this chapter teaches it.

8.8.1 Headless Mode

For production scraping, you do not need a visible browser window. Headless mode runs the browser without a graphical interface, which is faster and can run on servers without a display:

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")

driver = webdriver.Chrome(options=options)
driver.get("https://example.com")
print(f"Title: {driver.title}")
driver.quit()

The =new suffix selects Chrome’s new headless implementation (Chrome 109 and later), which behaves much more like the regular browser than the legacy --headless mode did. The --no-sandbox and --disable-dev-shm-usage flags resolve common issues when running Chrome in containerized or server environments (like GitHub Actions, which you will use in Chapter 14). Headless mode is essential for automated data collection pipelines where no one is watching the browser window.

One important caveat: some websites detect headless browsers and serve different content or block access entirely. They do this by checking for browser properties that headless mode handles differently (like the navigator.webdriver flag). If you get different results in headless mode than in a visible browser, this is likely the cause. For educational purposes, switching back to headed mode usually resolves the issue. In production settings, there are legitimate workarounds, but circumventing bot detection raises the ethical questions discussed in Chapter 2 — the fact that you can evade detection does not mean you should.

8.9 A Practical Selenium Workflow

Fragility is an argument for writing Selenium code defensively, not for avoiding Selenium. Here is a complete workflow that pulls together everything this chapter has covered. We will automate a multi-page interaction: navigating to a website, performing a search, waiting for results to load, extracting the data, and repeating across multiple pages.

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import pandas as pd

def scrape_with_selenium(search_term, max_pages=3):
    """Scrape search results from a dynamic website.

    This demonstrates the full Selenium workflow:
    1. Launch a headless browser
    2. Navigate and interact with the page
    3. Wait for dynamic content to load
    4. Extract data via BeautifulSoup
    5. Handle pagination
    6. Clean up
    """
    options = Options()
    options.add_argument("--headless=new")
    driver = webdriver.Chrome(options=options)

    all_results = []

    try:
        driver.get("https://example-search-site.com")

        # Find and fill the search box
        search_box = driver.find_element(By.NAME, "q")
        search_box.send_keys(search_term)
        search_box.submit()

        for page in range(max_pages):
            # Wait for results to load (up to 10 seconds)
            WebDriverWait(driver, 10).until(
                EC.presence_of_element_located((By.CLASS_NAME, "result-item"))
            )

            # Pass rendered HTML to BeautifulSoup
            soup = BeautifulSoup(driver.page_source, "html.parser")

            # Extract data from this page
            for item in soup.find_all("div", class_="result-item"):
                # Check that both elements exist before calling .text —
                # the same defensive pattern you learned for static scraping
                title_tag = item.find("h3")
                desc_tag = item.find("p")
                if title_tag is None or desc_tag is None:
                    continue  # Skip malformed result items rather than crash
                all_results.append({
                    "title": title_tag.text.strip(),
                    "description": desc_tag.text.strip(),
                    "page": page + 1
                })

            # Try to click "Next" for pagination
            try:
                next_button = driver.find_element(By.LINK_TEXT, "Next")
                next_button.click()
                time.sleep(2)  # Wait for next page to load
            except Exception:
                break  # No more pages

    finally:
        driver.quit()  # Always clean up

    return pd.DataFrame(all_results)

Notice the use of WebDriverWait with expected_conditions — this is superior to time.sleep() because it waits only as long as needed for the element to appear, rather than waiting a fixed amount of time that might be too short (element not loaded) or too long (wasted time). The try/finally block ensures the browser is closed even if an error occurs. And notice the None checks before calling .text: dynamic pages serve malformed or incomplete items just as often as static ones do, and the defensive parsing habits from Chapter 6 carry over unchanged.

8.10 When to Use Selenium vs. APIs

With the full toolkit in hand, here is the decision tree for choosing your data access method:

  1. Is there an API? Use it. APIs are faster, more reliable, and more structured (see Chapter 10 through Chapter 13).
  2. Is the content in the static HTML? Use requests + BeautifulSoup (see Chapter 6). It is simpler and faster than Selenium.
  3. Is the content loaded by JavaScript? Use browser automation — Selenium, or Playwright if auto-waiting and speed matter for your project. Either way, expect it to be slower, more fragile, and more resource-intensive than the options above.
  4. Is the content behind a login? This is where ethical judgment matters most. Automating access to login-gated content may violate Terms of Service. See Chapter 2.

The exercises below give you practice at every branch of this tree.

8.11 Exercises

  1. Dynamic content extraction. Find a website that loads content dynamically (via JavaScript, infinite scroll, or AJAX). Use Selenium to load the full page, then pass the source to BeautifulSoup and extract the dynamically-loaded data. Compare what you get with Selenium to what requests.get() returns.

  2. Multi-step interaction. Use Selenium to automate a multi-step process: navigate to a site with a search form, enter a query, submit it, and extract results from the response page.

  3. Static vs. dynamic comparison. For three websites of your choice, compare the HTML returned by requests.get() with the driver.page_source from Selenium. Categorize each site as static, partially dynamic, or fully dynamic.

  4. Load More button. Find a website that uses a “Load More” button (rather than infinite scroll) to reveal additional content. Write a Selenium script that clicks the button in a loop until it disappears or becomes disabled, then extracts all the loaded data. How many items were hidden behind the button?

  5. Performance comparison. For the same URL, measure the time taken by three approaches: requests.get(), headed Selenium (with a visible browser), and headless Selenium. Use time.time() to measure each. Create a table comparing the three approaches on speed, completeness of data retrieved, and resource usage. When is the speed tradeoff of Selenium worth it?

  6. Graduate extension (INFO 5617). Choose a JavaScript-heavy site relevant to your own research interests and instrument it with both approaches from this part of the book: static requests + BeautifulSoup and Selenium. Quantify what each approach sees — count of elements retrieved, payload size in bytes, and wall-clock time — across at least five pages. Then write a roughly 500-word memo on when the added cost of browser automation is justified, engaging with the treatment of JavaScript scraping in Mitchell (2018). Your memo should articulate a defensible general rule for other researchers, not just describe your particular site.

TipMissing Manual Reference

For background on debugging strategies when Selenium scripts fail, see Missing Manual Chapter 6: Debugging and Chapter 7: Reading Python Tracebacks.

8.12 Social History and Public Interest

The shift from server-rendered to client-rendered web pages represents a fundamental change in the web’s architecture. JavaScript frameworks like React, Angular, and Vue have made the web more interactive but also more opaque to automated observation. The ability to “view source” — once a straightforward way to see exactly what a page contained — now often reveals only a JavaScript bootstrap that fetches and renders content dynamically.

This architectural shift interacts with the enclosure dynamics described in Chapter 3. When platforms render content client-side, they gain another layer of control over automated access. Anti-scraping measures — CAPTCHAs, bot detection, dynamically-generated class names — become easier to implement. Researchers who once could rely on simple HTTP requests increasingly need browser automation to access the same data, raising both the technical barrier and the ethical stakes. The tools in this chapter are a response to that architectural shift — they exist because the simpler approaches taught in Chapter 6 are no longer sufficient for much of the modern web.

NotePublic Interest Connection

When platforms close their APIs and render content exclusively through JavaScript, browser automation becomes the last-resort path to data that serves the public interest. The tools in this chapter exist because of the enclosure dynamic from Chapter 3 — the progressive restriction of access to data that was once openly available. Researchers studying algorithmic bias, misinformation, and platform governance increasingly depend on Selenium-based methods precisely because the APIs that once served these research needs have been shut down or restricted beyond practical use.

8.13 Common Issues to Debug

  • NoSuchElementException: The element does not exist yet because the page has not finished loading. Add time.sleep() or use explicit waits with WebDriverWait.
  • StaleElementReferenceException: The DOM changed between when you found the element and when you tried to interact with it. Re-find the element.
  • Driver/browser version mismatch: Selenium Manager usually resolves this automatically, but brand-new browser releases occasionally break compatibility for a few days. Updating the selenium package typically fixes it.
  • Memory issues: Each Selenium driver runs a full browser process. Close drivers promptly with driver.quit().

8.14 Key Takeaways

The modern web is dynamic: JavaScript renders content after the initial HTML loads, making it invisible to static scraping tools. Selenium controls a real browser programmatically, letting you access fully-rendered pages, simulate user interactions, and extract content that requests cannot see. But Selenium is slow, fragile, and resource-intensive — a last resort when APIs and static scraping are unavailable. The hybrid approach of Selenium for rendering plus BeautifulSoup for parsing gives you full access to the modern web while keeping your parsing code familiar.

8.15 Further Reading

Mitchell, Ryan. 2018. Web Scraping with Python: Collecting More Data from the Modern Web. 2nd ed. O’Reilly Media.