14  Automating Data Collection

TipLearning Objectives
  • Convert a Jupyter Notebook into a standalone Python script
  • Configure headless Selenium for server-side browser automation
  • Write GitHub Actions workflow files (YAML) to schedule scraping jobs
  • Manage secrets (API keys, credentials) in GitHub repository settings
  • Implement end-to-end automated scraping pipelines
  • Assess the maintenance commitments of sustained data collection
TipCompanion Notebook

Run this chapter’s code as you read: open the companion notebook (see Appendix A for all of them).

14.1 From Exploration to Production

Interactive notebooks are for exploration; production data collection requires automation. This chapter moves from the interactive scraping techniques of Parts II and III into scheduled, unattended data pipelines that run in the cloud.

The progression is: notebook (where you develop and test your code) → script (a standalone .py file that runs without interaction) → scheduled pipeline (a script that runs automatically on a regular schedule). By the end of this chapter, you will have data collection infrastructure that runs every hour without your intervention.

TipMissing Manual Reference

For a full treatment of converting notebooks to scripts, command-line arguments, and if __name__ == "__main__" patterns, see Missing Manual Chapter 17: Scripting. For version control with Git, see Missing Manual Chapter 31: Version Control.

14.2 Why Automate?

The techniques you have learned in earlier chapters — scraping web pages, querying APIs, parsing documents — are powerful for capturing a snapshot of data at a single point in time. But many of the most interesting research questions in web data science are fundamentally temporal. How do Rotten Tomatoes scores evolve during a film’s theatrical release? How does the composition of a Spotify playlist shift with the seasons? Which Wikipedia articles surge in pageviews when a news event breaks, and how quickly does attention decay? These questions require not a single extraction but a time series: the same data collected at regular intervals over days, weeks, or months.

Automation transforms web data science from a one-time extraction exercise into a longitudinal research methodology. A scraper that runs once gives you a cross-sectional dataset. A scraper that runs every six hours for three months gives you a panel dataset that can reveal trends, cycles, and responses to external events. The infrastructure cost is modest — GitHub Actions provides free compute for public repositories — but the analytical payoff is substantial.

This chapter teaches you how to build that infrastructure from the tools you already know. The progression mirrors a real development workflow: you start with a working notebook, convert it to a standalone script, wrap it in a GitHub Actions workflow, and then monitor the pipeline over time. Each step introduces new skills, but the underlying logic remains the same code you have already written. The goal is not to learn a new programming paradigm but to move your existing skills from the interactive notebook environment into a production context where they run unattended. By the end, you will have built data collection infrastructure that connects directly to the research design considerations in Chapter 15.

14.3 Step 1: Notebook to Script

Let us convert a Rotten Tomatoes score scraper from notebook format to a standalone script. The key changes: remove interactive exploration, add a main() function, handle imports and configuration at the top, and write output to a file.

# scraper.py — Rotten Tomatoes Score Scraper
"""Scrape Rotten Tomatoes audience and critic scores for a film."""

import csv
import os
from datetime import datetime, timezone
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

# Fixed CSV schema, used for both the header and every appended row
FIELDNAMES = ["timestamp", "url", "critic_score", "audience_score"]

def get_scores(url):
    """Retrieve audience and critic scores from a Rotten Tomatoes page.
    
    Parameters
    ----------
    url : str
        The Rotten Tomatoes movie URL
    
    Returns
    -------
    dict
        Dictionary with 'timestamp', 'url', 'critic_score', 'audience_score'
    """
    # Configure headless Chrome
    options = Options()
    options.add_argument("--headless")
    options.add_argument("--disable-dev-shm-usage")
    
    driver = webdriver.Chrome(options=options)
    
    try:
        driver.get(url)
        # Wait for the score elements to render -- reading page_source immediately on a JS-heavy page yields intermittent empty scores
        WebDriverWait(driver, 10).until(
            EC.presence_of_element_located(
                (By.TAG_NAME, "score-icon-critic-deprecated")
            )
        )
        soup = BeautifulSoup(driver.page_source, "html.parser")

        # Extract scores from the page's custom score elements
        scores = {}
        for tag in soup.find_all("score-icon-critic-deprecated"):
            scores["critic_score"] = tag.get("percentage", "")
        for tag in soup.find_all("score-icon-audience-deprecated"):
            scores["audience_score"] = tag.get("percentage", "")

        scores["timestamp"] = datetime.now(timezone.utc).isoformat()
        scores["url"] = url
        return scores
    finally:
        driver.quit()


def save_scores(scores, filepath="scores.csv"):
    """Append scores to a CSV file, creating it if needed."""
    file_exists = os.path.exists(filepath)

    with open(filepath, "a", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=FIELDNAMES)
        if not file_exists:
            writer.writeheader()
        writer.writerow(scores)


def main():
    url = "https://www.rottentomatoes.com/m/the_wild_robot"
    scores = get_scores(url)
    save_scores(scores)
    print(f"Saved scores at {scores['timestamp']}")


if __name__ == "__main__":
    main()

Notice the headless Chrome configuration: --headless runs the browser without a visible window (Chapter 8, “Headless Mode”), and --disable-dev-shm-usage keeps Chrome from running out of shared memory in a container. The script leaves out --no-sandbox, which many tutorials add. That flag turns off the sandbox that keeps a page’s code from changing the machine or reading its files. On GitHub’s Ubuntu runners, Chrome should start without it: they run your job as an ordinary user, not root, and their image includes Google Chrome (version 154 in October 2026), which Ubuntu lets use its sandbox. Add --no-sandbox only where Chrome refuses to start without it, such as a container that runs as root.

Two other design choices deserve a comment. First, the CSV schema is pinned in the FIELDNAMES constant rather than derived from scores.keys(): a fixed schema keeps the column order stable across every run, whereas a keys-derived header locks in whatever the first run happened to produce — and a later scrape with a missing or extra key would then corrupt or crash the append. Second, the elements this scraper targets — score-icon-critic-deprecated and its audience twin — are not metadata tags but custom web-component elements defined by Rotten Tomatoes’ front-end code. The -deprecated suffix is a broad hint that these element names change frequently, and when they do, the scraper will start returning empty results. That fragility is exactly why Chapter 8 emphasized inspecting the live page in your browser’s developer tools: when the site changes, the fix starts there, not in this book.

14.3.1 Testing Your Script Locally

Before deploying your script to GitHub Actions, always test it locally from the command line. Running a script in the terminal is different from running cells in a notebook — import errors, missing dependencies, and file path issues that were invisible in Jupyter will surface immediately.

# From your terminal, run the script directly:
# $ python scraper.py
# Saved scores at 2024-11-15T14:23:07.482361+00:00

# Verify the output file was created and contains data:
# $ head scores.csv
# timestamp,url,critic_score,audience_score
# 2024-11-15T14:23:07.482361+00:00,https://www.rottentomatoes.com/m/the_wild_robot,97,95

# Check the file size to confirm it is not empty:
# $ ls -la scores.csv
# -rw-r--r--  1 user  staff  148 Nov 15 14:23 scores.csv

Debugging a Python script on your own machine is straightforward: you can add print() statements, inspect variables, and re-run instantly. Debugging the same script inside a GitHub Actions YAML workflow is far more painful — each attempt requires a commit, a push, and waiting for the runner to spin up, install dependencies, and execute. The few minutes you spend testing locally will save you hours of frustrating CI debugging later. If the script runs cleanly on your machine and produces the expected output file, you can be reasonably confident it will work in the Actions environment, provided the dependencies match.

14.4 Step 2: GitHub Actions

GitHub Actions lets you run scripts on a schedule using GitHub’s cloud infrastructure. You define a workflow in a YAML file placed in .github/workflows/:

# .github/workflows/scraper-action.yml
name: Scrape Rotten Tomatoes Scores

on:
  schedule:
    - cron: "0 */6 * * *"  # Every 6 hours
  workflow_dispatch:  # Allow manual triggering

permissions:
  contents: write  # The commit-and-push step below needs write access

jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Install Chrome
        uses: browser-actions/setup-chrome@v1
      
      - name: Run scraper
        run: python scraper.py
      
      - name: Commit and push results
        run: |
          git config user.name "GitHub Actions Bot"
          git config user.email "actions@github.com"
          git add scores.csv
          git diff --staged --quiet || git commit -m "Update scores $(date -u)"
          git pull --rebase origin main  # Integrate any commits pushed since checkout
          git push

Do not skip the permissions block: the automatic GITHUB_TOKEN that workflows use to authenticate is often read-only by default, and without contents: write the final git push step fails with a 403 error.

14.4.1 Python-Free Scraping

Not every automated scraper needs Python. The Wikimedia REST API returns JSON, so you can retrieve data with curl directly in the YAML workflow:

# .github/workflows/wiki-top-pages.yml
name: Top Wikipedia Pages

on:
  schedule:
    - cron: "0 12 * * *"  # Daily at noon UTC

permissions:
  contents: write  # Required for the push step below

jobs:
  fetch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Fetch top pages
        run: |
          mkdir -p data  # curl writes the file but will not create the directory
          DATE=$(date -u -d "yesterday" +%Y/%m/%d)
          curl -s "https://wikimedia.org/api/rest_v1/metrics/pageviews/top/en.wikipedia/all-access/$DATE" \
            -H "User-Agent: WebDataScience/1.0" \
            -o "data/top_pages_$(date -u -d yesterday +%Y%m%d).json"

      - name: Commit results
        run: |
          git config user.name "GitHub Actions Bot"
          git config user.email "actions@github.com"
          git add data/
          git diff --staged --quiet || git commit -m "Top pages $(date -u -d yesterday +%Y-%m-%d)"
          git pull --rebase origin main  # Integrate any commits pushed since checkout
          git push

14.4.2 Understanding Cron Syntax

The cron field in your workflow file controls when the scraper runs. Cron expressions use five fields separated by spaces: minute hour day-of-month month day-of-week. Each field can be a specific number, a range, a list, or a wildcard (* for “every”). The */N syntax means “every N units.” All times are in UTC, not your local timezone.

Expression Meaning
0 */6 * * * Every 6 hours (at minute 0)
0 12 * * 1-5 Weekdays at noon UTC
30 8 * * * Daily at 8:30 AM UTC
0 0 1 * * First day of each month at midnight
0 0 * * 0 Weekly on Sunday at midnight

The interactive tool at crontab.guru lets you build and verify cron expressions visually — type in an expression and it will show you in plain English when the job will run. Bookmark it; you will use it more often than you expect. Note that GitHub Actions does not guarantee exact execution times for scheduled workflows. Your job may run several minutes after the scheduled time, especially during periods of high demand on GitHub’s infrastructure. For most data collection tasks, this jitter is inconsequential.

14.4.3 Secrets Management

For authenticated APIs (like Spotify from Chapter 12), store credentials as GitHub repository secrets:

      - name: Run Spotify scraper
        env:
          SPOTIPY_CLIENT_ID: ${{ secrets.SPOTIPY_CLIENT_ID }}
          SPOTIPY_CLIENT_SECRET: ${{ secrets.SPOTIPY_CLIENT_SECRET }}
        run: python spotify_scraper.py
TipMissing Manual Reference

For a full treatment of automation with GitHub Actions, cron syntax, and CI/CD concepts, see Missing Manual Chapter 33: Automation. For secrets management, see Missing Manual Chapter 34: Environment Variables and Secrets.

14.4.4 Monitoring and Debugging Actions

Once your workflow is running on a schedule, you need to monitor it. GitHub’s Actions tab shows the status of every workflow run — green check for success, red X for failure. Click into a failed run to read the step-by-step log output, which will show you exactly where the error occurred. Common failure modes include expired API keys (the scraper worked for weeks, then the token expired), package version conflicts (a dependency updated and broke your import), and Chrome driver version mismatches (Selenium’s ChromeDriver must match the installed Chrome version, and both update frequently).

A particularly insidious failure mode is the “silent success”: the workflow runs to completion without errors, but the scraper did not actually collect any data — perhaps the target website changed its HTML structure, or an API endpoint started returning empty results. Your workflow reports success because no Python exception was raised, but your CSV file has a new row full of empty values. To catch this, add a validation step to your workflow that checks whether the data actually grew.

      - name: Validate scrape results
        run: |
          # Count rows in the CSV (excluding the header)
          ROW_COUNT=$(tail -n +2 scores.csv | wc -l)
          echo "Current row count: $ROW_COUNT"

          # Check if the latest row has non-empty score fields
          LAST_LINE=$(tail -1 scores.csv)
          if echo "$LAST_LINE" | grep -q ",,"; then
            echo "ERROR: Latest row contains empty fields"
            exit 1
          fi

          echo "Validation passed"

Adding this step after your scraping step means the workflow will fail with a clear error message when the data is incomplete, rather than silently committing garbage to your repository. You can also configure GitHub to send email notifications on workflow failures, so you learn about problems promptly rather than discovering weeks of missing data when you finally sit down to analyze your dataset.

14.4.5 Handling Failures Gracefully

Even with validation in place, your scraper will occasionally fail due to circumstances beyond your control: the target website goes down for maintenance, a rate limit is temporarily enforced, or a network timeout occurs. A well-designed scraping pipeline handles these failures without losing previously collected data or crashing the workflow.

The simplest approach is to wrap your scraping logic in a try/except block that logs the error and exits gracefully rather than raising an unhandled exception:

import sys
import traceback

def main():
    url = "https://www.rottentomatoes.com/m/the_wild_robot"
    try:
        scores = get_scores(url)
        save_scores(scores)
        print(f"Saved scores at {scores['timestamp']}")
    except Exception as e:
        print(f"Scraper failed: {e}", file=sys.stderr)
        traceback.print_exc()
        # Exit with code 1 so the GitHub Actions step reports failure
        sys.exit(1)

This pattern ensures that a single failed scrape does not corrupt your data file. The error is logged, the workflow step is marked as failed, and the next scheduled run will try again. Over a multi-week collection period, you should expect a small number of failures — the important thing is that they are visible, documented, and do not cascade into data loss.

14.4.6 Hardening the Workflow

A few more lines of YAML make a scheduled workflow noticeably more robust. A timeout-minutes limit kills a hung job (a stalled Selenium session would otherwise occupy the runner for the six-hour default), a concurrency group prevents a delayed run from overlapping with the next scheduled one, and caching pip’s downloads speeds up every run:

# Additions to .github/workflows/scraper-action.yml
concurrency:
  group: rt-scraper
  cancel-in-progress: false  # Let the current run finish; queue the next one

jobs:
  scrape:
    runs-on: ubuntu-latest
    timeout-minutes: 15  # Fail fast instead of hanging for the 6-hour default
    steps:
      # ... after the setup-python step ...
      - name: Cache pip dependencies
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-${{ hashFiles('requirements.txt') }}

Inside the scraper itself, the single biggest reliability improvement is retrying failed requests with exponential backoff. You already know this pattern — Chapter 5 introduced retries and backoff for exactly this situation — so apply it in your scraping functions rather than letting one transient 503 waste an entire scheduled run. A scraper that retries a couple of times with increasing delays will ride out the vast majority of momentary network and server hiccups that would otherwise appear as gaps in your time series.

14.5 Step 3: Analyze Collected Data

After running your scraper for a period, you will have accumulated data to analyze:

import pandas as pd

df = pd.read_csv("scores.csv")
df["timestamp"] = pd.to_datetime(df["timestamp"])

import matplotlib.pyplot as plt

plt.figure(figsize=(12, 5))
plt.plot(df["timestamp"], df["critic_score"], label="Critics", marker="o")
plt.plot(df["timestamp"], df["audience_score"], label="Audience", marker="s")
plt.title("Rotten Tomatoes Score Tracking")
plt.ylabel("Score (%)")
plt.legend()
plt.tight_layout()
plt.show()

14.7 Additional Exercises

These are open-ended extensions — no scaffold, no fixed path. Use them for further practice or deeper exploration. The first two are where the recommended exercise’s pieces get deployed for real.

  1. Build a scraper. Set up a GitHub Actions scraper for a data source of your choice — the script and workflow you drafted above are your starting point. Run it for at least three days and analyze the results. Good candidates include weather APIs, news site headlines, government dashboards, or e-commerce prices.

  2. Government data collector. Set up a daily GitHub Actions workflow that retrieves data from a free API (Open-Meteo weather, OpenAQ air quality, or USGS earthquake data). Run it for at least one week. Download the collected data and create a visualization showing how the measurements changed over the collection period.

  3. Graduate extension (INFO 5617). Deploy a scheduled collector of your own design and keep it running for at least two weeks. Then write an infrastructure post-mortem in the spirit of the “good enough practices” described by Wilson et al. (2017): quantify how many scheduled runs were missed or delayed, categorize every failure (selector drift, rate limiting, runner environment changes, silent successes), and measure any drift in the collected data itself over the window. Conclude with the two or three changes that would most improve reliability if the pipeline had to run for six months. The deliverable is the post-mortem, not the scraper.

NotePublic Interest Connection

Automated data collection is how you build the independent infrastructure that Chapter 3 describes as ownership. A GitHub Actions scraper does not depend on platform goodwill — it runs on your terms, on your schedule, collecting data you control. But ownership comes with responsibility: the ongoing labor of monitoring, maintaining, and updating your scrapers is itself a form of stewardship.

14.8 Social History and Public Interest

Automated data collection powers some of the most important public interest projects in data science. Air quality monitoring networks, transit performance trackers, and legislative activity dashboards all depend on regular, automated data retrieval. Community data projects — where citizens build their own monitoring infrastructure rather than depending on platforms or government agencies — embody the ownership value described in Chapter 3.

But automation also raises the ethical stakes from Chapter 2. A manual scraper that you run once makes a few requests. An automated scraper running every hour, indefinitely, accumulates a much larger footprint. The commitment to maintaining an automated scraper responsibly — monitoring for errors, respecting rate limits, adjusting to site changes — is a form of ongoing stewardship.

The Internet Archive’s Wayback Machine is perhaps the most ambitious automated data collection project in history — it has been systematically crawling and archiving the web since 1996, preserving billions of pages that would otherwise be lost as websites change and disappear. On a smaller scale, projects like the GDELT Project monitor global news media in real time, and the Pushshift archive (before its shutdown) collected every public Reddit post and comment for over a decade. These projects demonstrate both the power and the responsibility of automated collection: the data they preserved has enabled thousands of research studies, but maintaining the infrastructure required sustained funding, technical labor, and institutional commitment. When you build your own automated scraper, you are participating in this same tradition of data stewardship — on a smaller scale, but with the same obligations.

14.9 Common Issues to Debug

  • YAML indentation errors: YAML is whitespace-sensitive. Use a YAML validator before committing.
  • GitHub Actions runner environment: The Ubuntu runner may differ from your local setup. Pin Python and Chrome versions explicitly.
  • Git conflicts: If the Action runs while you are also pushing changes, you will get merge conflicts. This is why the workflow examples in this chapter run git pull --rebase before git push.
  • Cron syntax: The five fields are minute hour day-of-month month day-of-week. Use crontab.guru to verify your schedule.
  • Stale ChromeDriver: Selenium requires a ChromeDriver version that matches your Chrome version. The browser-actions/setup-chrome@v1 action in GitHub Actions handles this automatically, but local setups may drift. Use webdriver-manager for automatic local driver management.

14.10 Key Takeaways

Interactive notebooks are for exploration; production data collection requires automation. The progression is notebook → script → scheduled pipeline. GitHub Actions provides free cloud-based scheduling for public repositories. But automation is not “set and forget” — scrapers break, APIs change, and data quality drifts. Building your own data infrastructure means accepting responsibility for its upkeep. That ongoing commitment is a form of the stewardship and ownership that defines public interest data science.

14.11 Further Reading

Wilson, Greg, Jennifer Bryan, Karen Cranston, Justin Kitzes, Lex Nederbragt, and Tracy K. Teal. 2017. “Good Enough Practices in Scientific Computing.” PLOS Computational Biology 13 (6): e1005510. https://doi.org/10.1371/journal.pcbi.1005510.