9  Extracting Data from PDFs

TipLearning Objectives
  • Explain why PDF text extraction is difficult and how PDFs differ from structured formats
  • Use PyPDF to open PDF files, access metadata, and extract page-level text
  • Apply string methods and regular expressions to clean and structure extracted text
  • Build a data pipeline that processes multiple PDF files to extract structured information
  • Visualize trends across a corpus of PDF documents
TipCompanion Notebook

Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).

9.1 The Web’s Filing Cabinets

PDFs are everywhere in institutional record-keeping: city council meeting minutes, legislative bill texts, financial reports, court filings, academic papers, and environmental impact assessments. They are the format of choice for “official” documents precisely because they preserve visual layout across different devices and printers. But that design priority — visual fidelity over semantic structure — makes them hostile to programmatic data extraction. The records locked inside them are what Salganik (2018) calls readymade data: created by institutions for administrative purposes rather than for research, but repurposable — with effort — to answer questions their creators never anticipated.

When you read a PDF, you see paragraphs, tables, headers, and page numbers. When a computer reads a PDF, it sees a sequence of drawing instructions: place this character at coordinates (x, y), draw a line from here to there, embed this image at this location. Text extraction from PDFs is reconstruction, not reading — the computer must infer the logical structure from the spatial layout.

This chapter teaches you to work within these constraints. The results will not be perfect, but they can be good enough to turn inaccessible institutional records into analyzable datasets.

NotePublic Interest Connection

When governments publish meeting minutes, budgets, and regulatory filings only as PDFs, they create what Chapter 3 describes as “soft enclosure” — records that are nominally public but practically inaccessible for analysis. Building the skills to extract data from PDFs is itself a form of the oversight value: it enables independent analysis of institutional records.

9.2 PyPDF Basics

The pypdf library provides page-level text extraction and metadata access:

# Install if needed:
# pip install pypdf

from pypdf import PdfReader

# Open a PDF file (download Boulder City Council minutes first)
reader = PdfReader("council_minutes_2024_01.pdf")

# Document-level metadata
print(f"Number of pages: {len(reader.pages)}")
print(f"Metadata: {reader.metadata}")

9.2.1 Extracting Text

# Extract text from the first page
page = reader.pages[0]
text = page.extract_text()
print(text[:500])  # Print the first 500 characters

The output will likely be messy: line breaks may not correspond to paragraph boundaries, headers and footers are mixed in with content, and table data may be garbled. This is normal. Cleaning up extracted text is a core part of the workflow.

9.2.2 Cleanup Strategies

import re

def clean_page_text(raw_text):
    """Clean text extracted from a PDF page.
    
    Parameters
    ----------
    raw_text : str
        Raw text from PdfReader.pages[n].extract_text()
    
    Returns
    -------
    str
        Cleaned text with normalized spacing, line breaks preserved
    """
    # Remove page numbers (common patterns)
    text = re.sub(r'Page \d+ of \d+', '', raw_text)

    # Remove common headers/footers (customize per document)
    text = re.sub(r'CITY COUNCIL MEETING MINUTES', '', text)

    # Collapse runs of spaces and tabs — but NOT newlines. Line
    # structure is real information in extracted PDF text (headings
    # sit on their own lines), so we keep it
    text = re.sub(r'[ \t]+', ' ', text)

    # Strip each line and drop lines left empty by the removals above
    lines = [line.strip() for line in text.split('\n')]
    return '\n'.join(line for line in lines if line)

cleaned = clean_page_text(text)
print(cleaned[:300])

The most important choice in this function is what not to collapse. A tempting one-liner is re.sub(r'\s+', ' ', raw_text), which normalizes all whitespace in a single pass — but \s matches newlines too, and flattening the page into one long line destroys structure you will need later. Section headings, for example, are recognizable precisely because they sit alone on their own lines; erase the newlines and no heading detector can find them. Collapsing only spaces and tabs ([ \t]+) tidies the text while keeping its shape.

9.2.3 Regular Expressions for PDF Text

Regular expressions are your primary tool for imposing structure on the messy text that comes out of PDF extraction. Where structured data formats give you fields and columns, PDF text gives you a wall of characters — and regex is how you carve that wall into usable pieces. The patterns below cover the most common extraction tasks you will encounter when working with government and institutional PDFs.

Each pattern targets a specific kind of structured information that appears in unstructured text. Learning to recognize these patterns and adapt them to your specific documents is a skill that improves with practice. No single regex will work across all PDFs, but these five cover a wide range of common scenarios.

import re

# 1. Dates in slash-separated format (e.g., "1/15/2024", "12/03/23")
date_pattern = r'\d{1,2}/\d{1,2}/\d{2,4}'
dates = re.findall(date_pattern, cleaned)
# Returns: ['1/15/2024', '2/6/2024', ...]

# 2. Dollar amounts (e.g., "$1,250.00", "$50", "$3,400,000")
dollar_pattern = r'\$[\d,]+\.?\d*'
amounts = re.findall(dollar_pattern, cleaned)
# Returns: ['$1,250.00', '$50', '$3,400,000']

# 3. Repeating headers that PDF extraction duplicates on every page
header_pattern = r'CITY COUNCIL.*?MINUTES'
cleaned = re.sub(header_pattern, '', cleaned)

# 4. Section headings in ALL CAPS on their own line (at least 6
#    characters, allowing digits and light punctuation for headings
#    like "AGENDA ITEM 5A"). The class matches a literal space, not
#    \s — \s would match newlines too, letting a single match
#    swallow several consecutive heading lines
heading_pattern = r'^[A-Z][A-Z0-9 .:&-]{5,}$'
headings = re.findall(heading_pattern, cleaned, re.MULTILINE)
# Returns: ['CALL TO ORDER', 'PUBLIC COMMENT', 'CONSENT AGENDA', 'ADJOURNMENT', ...]

# 5. Email addresses embedded in document text
email_pattern = r'[\w.+-]+@[\w-]+\.[\w.-]+'
emails = re.findall(email_pattern, cleaned)
# Returns: ['clerk@bouldercolorado.gov', ...]

Notice that each of these patterns makes assumptions about the document format. The date pattern assumes slash-separated dates; if your documents use “January 15, 2024” instead, you need a different pattern. The dollar amount pattern assumes a leading $ sign; budget documents that use parentheses for negative values need additional handling. The section heading pattern assumes uppercase formatting; some documents use title case or bold formatting that does not survive text extraction. Always test your patterns against multiple pages and multiple documents from the same source before committing to a pipeline.

TipMissing Manual Reference

For a thorough introduction to regular expressions — the pattern-matching tool you will use extensively for text cleanup — see Missing Manual Chapter 18: Regular Expressions.

9.3 Case Study: Boulder City Council Minutes

Boulder’s City Council publishes signed minutes of its public meetings online. These PDFs contain structured information — council member attendance, public comment speakers, agenda items, voting records — that can be extracted and analyzed.

9.3.1 Measuring Council Attendance

import os
from pypdf import PdfReader

# Assuming you have downloaded several PDF files into a directory
pdf_dir = "council_minutes/"
pdf_files = sorted([f for f in os.listdir(pdf_dir) if f.endswith(".pdf")])

def split_names(match):
    """Split a regex match of comma-separated names into a clean list."""
    if not match:
        return []
    return [name.strip() for name in match.group(1).split(",") if name.strip()]

def extract_attendance(pdf_path):
    """Extract council member attendance from a meeting minutes PDF.

    Parameters
    ----------
    pdf_path : str
        Path to the PDF file

    Returns
    -------
    dict
        Dictionary with 'file', 'date', 'present', and 'absent' —
        the last two are lists of council member names
    """
    reader = PdfReader(pdf_path)

    # Attendance is typically on the first page
    first_page_text = reader.pages[0].extract_text()

    # Cheap date extraction: filenames like council_minutes_2024_01.pdf
    # encode the year and month — parse them rather than the PDF text
    date_match = re.search(r'(\d{4})[_-](\d{2})', os.path.basename(pdf_path))
    meeting_date = f"{date_match.group(1)}-{date_match.group(2)}" if date_match else None

    # Names follow "Present:" — capture up to "Absent:" if it appears
    # on the same line, otherwise up to the end of the line
    present_match = re.search(r'Present:\s*(.*?)\s*(?:Absent:|$)',
                              first_page_text, re.IGNORECASE | re.MULTILINE)

    # Names after "Absent:" run to the end of that line
    absent_match = re.search(r'Absent:\s*(.*)$',
                             first_page_text, re.IGNORECASE | re.MULTILINE)

    return {
        "file": os.path.basename(pdf_path),
        "date": meeting_date,
        "present": split_names(present_match),
        "absent": split_names(absent_match),
    }

Be honest with yourself about what this function can and cannot do. It assumes the clerk wrote a line beginning “Present:” followed by comma-separated names — a format that holds for many Boulder minutes, but not all. Name formats vary (“Council Member Smith” versus “Smith,” with or without titles), some documents separate names with semicolons or line breaks, and minutes where everyone attended often read “Absent: None,” which this code will dutifully report as a council member named None. Treat the output as a first draft: spot-check it against a handful of PDFs, patch the patterns to match the formats you actually encounter, and document the failure cases you decide not to handle.

9.3.2 Building the Pipeline

results = []
for pdf_file in pdf_files:
    try:
        pdf_path = os.path.join(pdf_dir, pdf_file)
        result = extract_attendance(pdf_path)
        results.append(result)
        print(f"Processed: {pdf_file}")
    except Exception as e:
        print(f"Error processing {pdf_file}: {e}")

import pandas as pd

df = pd.DataFrame(results)

# Turn the name lists into counts for analysis
df["n_present"] = df["present"].apply(len)
df["n_absent"] = df["absent"].apply(len)
print(df.head())

9.3.4 Measuring Public Comment Frequency

One of the most civically valuable pieces of information buried in meeting minutes is how many members of the public showed up to speak. Public comment sections are where residents voice concerns about zoning changes, police oversight, budget priorities, and development projects. Extracting this data requires detective work: you need to identify the textual markers that signal a public comment section and then count the speakers within it.

The challenge is that different municipalities format public comment sections differently, and even within the same city, the format may shift over time as clerks change or recording practices evolve. You are looking for patterns — section headers like “PUBLIC COMMENT” or “PUBLIC HEARING,” followed by names or numbered items indicating individual speakers. This is the kind of work where you iterate: try a pattern on one document, inspect the results manually, refine the pattern, test it on a second document, and adjust again. Start by printing the raw text of several public comment sections to understand the format before writing any extraction code.

def count_public_comments(text):
    """Count the number of public comment speakers in meeting text.

    Parameters
    ----------
    text : str
        Full text extracted from a meeting minutes PDF

    Returns
    -------
    int
        Estimated number of public comment speakers
    """
    # Find the public comment section
    comment_section = re.search(
        r'(?:PUBLIC COMMENT|PUBLIC HEARING)(.*?)(?:CONSENT AGENDA|OLD BUSINESS|NEW BUSINESS)',
        text, re.DOTALL | re.IGNORECASE
    )

    if not comment_section:
        return 0

    section_text = comment_section.group(1)

    # Count speakers by looking for common patterns
    # Pattern 1: "spoke regarding" or "spoke about" or "spoke in favor"
    spoke_count = len(re.findall(r'spoke\s+(?:regarding|about|in|on|to)',
                                  section_text, re.IGNORECASE))

    # Pattern 2: numbered items like "1." "2." "3."
    numbered_count = len(re.findall(r'^\s*\d+\.', section_text, re.MULTILINE))

    # Return the higher count (different formats capture different patterns)
    return max(spoke_count, numbered_count)
# Apply across all meeting files
comment_counts = []
for pdf_file in pdf_files:
    pdf_path = os.path.join(pdf_dir, pdf_file)
    try:
        reader = PdfReader(pdf_path)
        full_text = " ".join([p.extract_text() for p in reader.pages])
        count = count_public_comments(full_text)
        comment_counts.append({"file": pdf_file, "speakers": count})
    except Exception as e:
        comment_counts.append({"file": pdf_file, "speakers": 0})

df_comments = pd.DataFrame(comment_counts)

plt.figure(figsize=(12, 5))
plt.bar(range(len(df_comments)), df_comments["speakers"])
plt.xlabel("Meeting")
plt.ylabel("Public Comment Speakers")
plt.title("Public Comment Participation in Boulder City Council Meetings")
plt.tight_layout()
plt.show()

What does this data reveal? Spikes in public comment frequency often correspond to controversial agenda items — a proposed development project, a change in policing policy, or a budget reallocation. Periods of low participation may indicate either public satisfaction or public disengagement. Neither interpretation is self-evident from the data alone, which is why combining extracted counts with knowledge of the agenda (also extractable from the same PDFs) produces richer analysis.

This kind of measurement is inherently imperfect. Your regex patterns will miss some speakers and occasionally count non-speakers. But even approximate counts, applied consistently across dozens of meetings, reveal patterns that would be invisible to someone reading a single set of minutes. The imprecision of PDF extraction is the price of scale — and scale is what transforms individual documents into a dataset.

9.4 Building a Reusable PDF Pipeline

When you work with PDFs at scale — processing dozens or hundreds of documents from the same source — you need a structured pipeline. Here is a pattern that works for most government document collections:

import os
import re
import pandas as pd
from pypdf import PdfReader

class PDFCorpusAnalyzer:
    """A reusable pipeline for extracting and analyzing PDF corpora.
    
    This class encapsulates the full workflow:
    1. Discover PDF files in a directory
    2. Extract text from each file
    3. Apply document-specific cleanup rules
    4. Extract structured fields
    5. Compile results into a DataFrame
    """
    
    def __init__(self, directory, cleanup_patterns=None):
        self.directory = directory
        self.cleanup_patterns = cleanup_patterns or []
        self.files = sorted([
            f for f in os.listdir(directory) if f.lower().endswith(".pdf")
        ])
    
    def extract_text(self, filepath, pages=None):
        """Extract text from specified pages of a PDF."""
        reader = PdfReader(filepath)
        
        if pages is None:
            pages = range(len(reader.pages))
        
        text = ""
        for page_num in pages:
            if page_num < len(reader.pages):
                text += reader.pages[page_num].extract_text() + "\n"
        
        # Apply cleanup patterns
        for pattern, replacement in self.cleanup_patterns:
            text = re.sub(pattern, replacement, text)
        
        return text.strip()
    
    def process_all(self, extract_fn):
        """Process all PDFs using a custom extraction function.
        
        Parameters
        ----------
        extract_fn : callable
            Function that takes (filepath, text) and returns a dict
        """
        results = []
        for filename in self.files:
            filepath = os.path.join(self.directory, filename)
            try:
                text = self.extract_text(filepath)
                result = extract_fn(filepath, text)
                result["filename"] = filename
                results.append(result)
            except Exception as e:
                print(f"Error processing {filename}: {e}")
                results.append({"filename": filename, "error": str(e)})
        
        return pd.DataFrame(results)

# Usage example: analyze Boulder City Council meeting lengths
def extract_meeting_info(filepath, text):
    """Extract meeting metadata from council minutes text."""
    # Count approximate words
    word_count = len(text.split())
    
    # Try to find the meeting date (format varies by document)
    date_match = re.search(
        r'(January|February|March|April|May|June|July|August|'
        r'September|October|November|December)\s+\d{1,2},\s+\d{4}',
        text
    )
    
    return {
        "word_count": word_count,
        "date": date_match.group(0) if date_match else None,
        "page_count": len(PdfReader(filepath).pages)
    }

analyzer = PDFCorpusAnalyzer(
    "council_minutes/",
    cleanup_patterns=[
        (r'[ \t]+', ' '),               # Collapse spaces/tabs, keep line breaks
        (r'Page \d+ of \d+', ''),       # Remove page numbers
    ]
)

df = analyzer.process_all(extract_meeting_info)
print(df.head())

9.4.1 When PyPDF Is Not Enough

PyPDF handles text-based PDFs well but struggles with two common scenarios:

Tables in PDFs: PyPDF extracts tables as streams of text, losing the column-row structure. For tabular PDF data, use pdfplumber, which preserves spatial relationships:

# pip install pdfplumber
import pdfplumber

with pdfplumber.open("report.pdf") as pdf:
    page = pdf.pages[0]
    table = page.extract_table()
    
    # table is a list of lists (rows of cells)
    df = pd.DataFrame(table[1:], columns=table[0])

Scanned PDFs: If the PDF was created by scanning a physical document (no text layer), PyPDF will extract empty or garbled text. You need OCR (Optical Character Recognition). The Tesseract OCR engine, accessible through the pytesseract library, can process scanned pages, though accuracy varies with scan quality.

9.5 Exercises

  1. Find and extract. Locate a set of PDF documents from a government source — city council agendas, regulatory filings, or budget summaries. Extract a consistent data element across at least ten documents and build a DataFrame.

  2. Extraction quality comparison. Compare the text extraction quality of PyPDF on three different types of PDFs: a text-based report, a document with complex tables, and a slide deck exported to PDF. What works well and what fails?

  3. Tables, not text. Find a government PDF that contains a budget or financial table — a city budget summary, a department expenditure report, or a school district financial statement. First extract the page with pypdf and observe what happens to the table’s structure. Then extract the same table with pdfplumber’s extract_table() and build a proper DataFrame with numeric columns. Briefly describe what spatial information pdfplumber uses that plain text extraction throws away, and when that extra dependency is worth it.

  4. Topic tracking. Download a year’s worth of city council meeting minutes (roughly 12–24 PDFs). Choose a topic term relevant to local politics — “housing,” “police,” “flood,” “budget” — and count how often it appears in each meeting’s extracted text. Plot the frequency over time and identify the meetings where the topic spiked. For a measure more robust than raw counts, apply the bag-of-words preprocessing from Chapter 7 (tokenization, stopword removal) before counting, and normalize by each document’s length. What events or agenda items explain the spikes?

  5. Tool comparison. Install pdfplumber alongside pypdf. Extract text from the same PDF using both tools. Compare the quality of text extraction — line breaks, whitespace handling, table detection. Which tool handles your specific PDF better, and why?

  6. Graduate extension (INFO 5617). Sample at least ten municipal-meeting PDFs from each of two different cities and quantify extraction quality across the two corpora: characters extracted per page, the share of pages yielding little or no text, and whether any documents are scanned images that would require OCR. Then write a data-quality memo organized around the datasheet categories of Gebru et al. (2021) — motivation, composition, collection process, preprocessing, and recommended uses — documenting what a downstream researcher would need to know before trusting analyses built on these extractions.

9.6 Social History and Public Interest

The PDF format was created by Adobe in 1993 to solve a real problem: documents needed to look the same regardless of what printer or screen displayed them. But the format’s emphasis on visual fidelity came at the cost of semantic accessibility. Organizations like DocumentCloud, MuckRock, and the Sunlight Foundation have built tools to extract structured data from the flood of PDFs that government agencies produce — bridging the gap between nominally public records and practically accessible data.

DocumentCloud, built by ProPublica and IRE (Investigative Reporters and Editors), emerged as a platform where journalists could upload, analyze, annotate, and share document collections. It has been used in investigations of police misconduct records, corporate fraud filings, and government secrecy — cases where the raw documents are public but the volume makes manual analysis impractical. DocumentCloud’s contribution was not just technical but organizational: it created a shared infrastructure where one journalist’s document processing work could benefit others investigating the same institutions.

MuckRock took a different approach to the document access problem by automating the Freedom of Information Act (FOIA) request process itself. The platform helped journalists and citizens file thousands of public records requests, tracked agency response times, and published the results openly. The Sunlight Foundation, active from 2006 to 2020, advocated for government agencies to publish documents in machine-readable formats rather than PDFs in the first place — arguing that true transparency requires not just publication but usability. Together, these tools and organizations represent the infrastructure of transparency: community-built resources that bridge the gap between nominally public documents and practically accessible data. Their work embodies the ownership value from Chapter 3 — the idea that public records belong to the public, and that technical barriers to access are themselves a form of enclosure that undermines democratic accountability.

The Boulder City Council case illustrates this tension at the local level. The same meeting information could be published as structured data (allowing citizens to easily track attendance, voting patterns, and public comment frequency) or as scanned PDFs (requiring significant technical effort to analyze). The format choice is itself a policy decision with consequences for transparency and accountability. When you build a pipeline that extracts structured data from PDFs, you are not just doing data science — you are doing the work that these organizations have championed for decades.

9.7 Common Issues to Debug

  • Garbled text from scanned PDFs: PyPDF extracts text only from PDFs that have a text layer. Truly scanned documents require OCR tools like Tesseract.
  • Table data extracted as a text stream: PyPDF is not a table extraction tool. For tabular data in PDFs, consider pdfplumber or camelot.
  • Regular expressions that work on one document but fail on another: PDF formatting varies even within documents from the same source. Test your patterns on multiple files.
  • Encoding issues: Some PDFs use non-standard character encodings. Use .encode('utf-8', errors='replace') if needed.

9.8 Key Takeaways

PDFs are designed for visual fidelity, not data extraction. PyPDF gives you page-level text as a starting point; string methods and regex let you clean and structure what you extract; and function abstractions let you process many documents consistently. Expect messiness — PDF extraction is reconstruction, not reading. The broader lesson connects to Chapter 3: when institutions publish data only in PDF form, they create a practical barrier to the analysis that nominally public records are meant to enable.

9.9 Further Reading

Gebru, Timnit, Jamie Morgenstern, Brenda Vecchione, et al. 2021. “Datasheets for Datasets.” Communications of the ACM 64 (12): 86–92. https://doi.org/10.1145/3458723.
Salganik, Matthew J. 2018. Bit by Bit: Social Research in the Digital Age. Princeton University Press. https://www.bitbybitbook.com/.