4 Public Interest Data Science

4.1 Learning Objectives

By the end of this chapter, you will be able to:

Identify three dynamics — enclosure, erosion, and exemption — that shape what public data looks like and what it omits.
Recognize the telltale symptoms of each dynamic in real datasets (title blocks, format mismatches, suppression codes, absent records).
Apply the three pillars of public interest practice — openness, oversight, and ownership — using the Known Issues Log, join-quality reports, and Data Dictionary.
Frame your analytical conclusions in language that matches what the data can actually support.

4.2 Why This Chapter Matters

Data analysis is never neutral. Every dataset reflects choices about what to count, how to count it, who gets to count, and who is counted. This chapter introduces a framework for thinking about those choices — not to make you suspicious of data, but to make you a more careful reader of it.

The datasets in this book come from government agencies, public records, and open archives. They are about cannabis sales, voter registration, crime, mortality, and population. They are, in a meaningful sense, about people. Understanding how they were produced — and what was left out — is part of doing the analysis well.

4.3 Three Dynamics That Shape Public Data

Public data is shaped by three recurring dynamics that appear, in different forms, in nearly every dataset you will work with in this book.

4.3.1 Enclosure: Data Designed for Humans, Not Machines

Enclosure happens when data that is technically public is formatted in ways that make it difficult to use programmatically. The data exists, but accessing it requires significant effort — and that effort is unevenly distributed.

The Colorado cannabis sales file is a clear example. It is published by a state agency and available to download, but it arrives as an Excel export with a two-row title block at the top, eight footnote rows at the bottom, and sales figures formatted as currency strings ($1,234,567). None of this matters if you are reading the spreadsheet in Excel. All of it matters if you are trying to load it into pandas.

The CDC WONDER database is a more extreme example. Mortality data is available, but it must be requested through a web form that limits what you can query at once. The results are delivered as HTML tables — not CSV files, not an API — which means extracting the data requires additional parsing steps that are not documented or supported.

These are not accidents. They are the result of publishing workflows optimized for human readers in browsers and spreadsheets. The practical effect is that the people who can most easily use this data are those with the technical skills to work around its formatting — a form of enclosure through interface design.

What to look for

When you load a new dataset, the problems you encounter in the first few cells — extra header rows, currency strings, aggregated totals, merged cells — are often symptoms of enclosure. The dataset was designed for a different audience than you.

4.3.2 Erosion: Infrastructure Decay Made Concrete

Erosion happens when data infrastructure degrades over time, typically because no single agency is responsible for maintaining consistency across systems.

The county name mismatch in the course datasets is a concrete example. Three Colorado government agencies each publish data about Colorado counties:

The CDC WONDER database formats county names as "Adams County, CO".
The Colorado Bureau of Investigation uses "Adams County".
The Colorado State Demography Office uses bare names: "Adams".

No one decided that these formats should differ. They are the accumulated result of three agencies building their own systems, at different times, with different conventions and no coordination mandate. The mismatch is invisible until you try to join the tables — at which point it silently drops every row, because "Adams County, CO" does not equal "Adams County" does not equal "Adams".

This is infrastructure erosion made concrete: a technical inconvenience that embodies a real governance problem. The cost is borne by anyone who tries to analyze these datasets together — which is exactly the analysis you would want to do if you were studying, say, the relationship between crime rates, death rates, and population in Colorado counties.

What to look for

When joining tables from different agencies, always audit your key columns before joining. SELECT DISTINCT county FROM deaths ORDER BY county; run against all three tables will immediately reveal format mismatches that would otherwise produce a silent empty result.

4.3.3 Exemption: What Data Does Not Exist

Exemption is the hardest dynamic to see, because it involves absence rather than presence. It happens when data that would be useful for public accountability is not collected, not retained, or not released.

The CDC suppression rule is a visible example of exemption by policy. When death counts in a county-year-cause cell are below 10, the CDC reports the value as "Suppressed" rather than a number. The stated reason is privacy protection — individuals in a small cell might be identifiable. This is a legitimate concern, but the effect is that the counties with the smallest populations — often rural counties, often with higher rates of specific causes of death — have the most missing data. The places where data would be most informative are the places where data is least available.

Less visible forms of exemption include data that is never collected at all: housing displacement records at the parcel level, detention facility population data, stop-and-frisk incident logs in jurisdictions that do not publish them. These absences are not accidents either. They reflect choices about what is worth measuring and who is worth measuring it for.

What to look for

When a column has missing values, ask why. "Suppressed" in the CDC data is not a data quality problem — it is a policy decision. Understanding why data is missing is different from knowing that it is missing, and it matters for how you handle it analytically and how you communicate about it.

4.4 Three Pillars of Public Interest Data Practice

Understanding these dynamics does not make data analysis impossible. It makes it more honest. Three practices, which appear throughout this book, help translate awareness into better work.

4.4.1 Openness — The Known Issues Log

Openness means making your data problems visible rather than hiding them. The Known Issues Log introduced in Chapter 2 is a transparency tool: a running record of every quality problem you found, what you decided to do about it, and why.

Publishing a Known Issues Log alongside your analysis lets readers evaluate your decisions rather than taking your conclusions on faith. It also protects you: documenting that you knew about a data problem and addressed it deliberately is very different from having it discovered later as an unacknowledged flaw.

Openness is not about being embarrassed by data problems. Every real-world dataset has them. It is about treating data quality as a fact to communicate rather than a weakness to conceal.

4.4.2 Oversight — The Join-Quality Report

Oversight means auditing your data pipelines to catch silent failures. The join-quality report introduced in Chapter 12 is an audit trail: a structured check of what matched, what was dropped, and whether the result is plausible.

A join that silently drops 40% of rows because of a county name format mismatch is not a software bug — it is a logical error that produces a confident-looking wrong answer. Oversight practices — checking row counts before and after joins, validating totals, auditing unmatched records — are what prevent silent errors from becoming invisible conclusions.

In a public interest context, the stakes are higher. An analysis of county-level health outcomes that silently drops rural counties is not only technically wrong — it is systematically biased against the populations that are hardest to count.

4.4.3 Ownership — The Data Dictionary

Ownership means tracking provenance: where data came from, who collected it, when it was accessed, and what decisions were made about it. The Data Dictionary introduced in Chapter 2 is a provenance record.

Provenance matters for reproducibility — someone else should be able to reconstruct your analysis from the same sources. It also matters for accountability. Knowing that a dataset comes from the Colorado Department of Revenue’s Marijuana Enforcement Division, downloaded on a specific date from a specific URL, is different from knowing only that you “got the cannabis data from the state.”

In an era of frequently updated government datasets, provenance records also let you know whether your analysis is based on data that has since been revised or retracted.

4.5 How These Pillars Appear in This Book

The three pillars are not add-ons — they are woven into the recurring practices that appear in every chapter of this book.

Recurring practice	Pillar	Where introduced
Known Issues Log	Openness	Chapter 2
Data Dictionary	Ownership	Chapter 2
Post-load checklist	Openness	Chapter 5
Validate after every transformation	Oversight	throughout
Join-quality report	Oversight	Chapter 12
County name standardization	Oversight (erosion response)	Chapter 10
`na_values` documentation	Ownership	Chapter 5
Documenting why data is missing	Openness	Chapter 6

Reading the technical chapters with these pillars in mind transforms a series of data manipulation exercises into a coherent practice. You are not only learning how to rename columns and join tables. You are learning how to be accountable for what you find.

4.6 Common Issues in This Chapter

You treat missing values as random

What it means: Code that drops NaNs or fills them with a mean assumes the missingness is random. When the missingness is systematic — the CDC suppression rule, for example — that assumption biases your analysis.

How to fix it: Before any missing-value imputation, ask why the value is missing. Document the answer in your Known Issues Log. Treat policy-driven suppression differently from data-entry omissions.

A join silently drops rows because of format mismatches

What it means: You joined two agency tables on county, but one uses "Adams County, CO" and the other uses "Adams". The result is confident-looking, systematically wrong.

How to fix it: Run a join-quality report (Chapter 12): compare row counts before and after the join, and list any unmatched keys. Standardize keys in a cleaning step before joining.

Your analysis does not account for absent data

What it means: The counties or demographic groups with the most missing rows are often the ones your analysis would most benefit from including. Silence about absence is not the same as representativeness.

How to fix it: When you drop or exclude rows, document what was excluded and speculate about what the exclusion means for your conclusions. A limitation section in your writeup is not weakness — it is credibility.

4.7 Key Takeaways

Public data is shaped by three recurring dynamics — enclosure, erosion, and exemption — that are invisible until you know to look for them.
Three practices — openness (Known Issues Log), oversight (join-quality reports), and ownership (Data Dictionary) — translate awareness into better analysis.
The technical habits in this book (inspect before acting, validate after every transform, document your decisions) are not separate from ethics. They are what ethical data practice looks like in code.
Good data work makes absent data and uncertain decisions visible, not invisible.

4.8 Further Reading

📚 Going deeper on public interest data practice

“Data Feminism” by D’Ignazio and Klein — a free online edition at data-feminism.mitpress.mit.edu develops this book’s themes at length.
“Weapons of Math Destruction” by Cathy O’Neil — an accessible survey of how opaque algorithms and missing data harm communities.
Ada Lovelace Institute: Data justice — adalovelaceinstitute.org publishes short, readable reports on data governance and equity.
Investigative Reporters and Editors: Data library — ire.org has public-interest datasets and story pitches.
CDC WONDER documentation — wonder.cdc.gov/wonder/help/main.html documents the suppression rules referenced above.

4.9 Exercises

Exercise 1: Identify the dynamic

For each of the following data problems, identify which dynamic (enclosure, erosion, or exemption) it represents and explain your reasoning:

A county health department publishes monthly reports as PDF scans of printed tables.
Two state agencies use different ZIP code formats — one includes the ZIP+4 suffix, the other does not — making it impossible to join their records directly.
A police department publishes use-of-force data only in aggregate, not at the incident level.
A state unemployment database uses 999 to indicate “not applicable” in some columns and null in others.

Exercise 2: The absent dataset

Look at the datasets described in Appendix A. For each dataset, identify one question you would like to answer that the dataset cannot answer — either because the relevant data is not collected, is suppressed, or is not in this dataset. For each gap, suggest what additional data source might fill it and whether that source is likely to be publicly available.

Exercise 3: Known Issues Log entry

Load the Colorado cannabis sales dataset as described in Chapter 5. Write a Known Issues Log entry for the suppressed NR values in the sales columns. Your entry should document: (1) what the problem is, (2) which rows are affected, (3) what you decided to do about it, and (4) why that decision is defensible.

# Public Interest Data Science {#sec-public-interest} ## Learning Objectives By the end of this chapter, you will be able to: - Identify three dynamics — **enclosure**, **erosion**, and **exemption** — that shape what public data looks like and what it omits. - Recognize the telltale symptoms of each dynamic in real datasets (title blocks, format mismatches, suppression codes, absent records). - Apply the three pillars of public interest practice — **openness**, **oversight**, and **ownership** — using the Known Issues Log, join-quality reports, and Data Dictionary. - Frame your analytical conclusions in language that matches what the data can actually support. ## Why This Chapter Matters Data analysis is never neutral. Every dataset reflects choices about what to count, how to count it, who gets to count, and who is counted. This chapter introduces a framework for thinking about those choices — not to make you suspicious of data, but to make you a more careful reader of it. The datasets in this book come from government agencies, public records, and open archives. They are about cannabis sales, voter registration, crime, mortality, and population. They are, in a meaningful sense, *about people*. Understanding how they were produced — and what was left out — is part of doing the analysis well. ## Three Dynamics That Shape Public Data Public data is shaped by three recurring dynamics that appear, in different forms, in nearly every dataset you will work with in this book. ### Enclosure: Data Designed for Humans, Not Machines **Enclosure** happens when data that is technically public is formatted in ways that make it difficult to use programmatically. The data exists, but accessing it requires significant effort — and that effort is unevenly distributed. The Colorado cannabis sales file is a clear example. It is published by a state agency and available to download, but it arrives as an Excel export with a two-row title block at the top, eight footnote rows at the bottom, and sales figures formatted as currency strings (`$1,234,567`). None of this matters if you are reading the spreadsheet in Excel. All of it matters if you are trying to load it into pandas. The CDC WONDER database is a more extreme example. Mortality data is available, but it must be requested through a web form that limits what you can query at once. The results are delivered as HTML tables — not CSV files, not an API — which means extracting the data requires additional parsing steps that are not documented or supported. These are not accidents. They are the result of publishing workflows optimized for human readers in browsers and spreadsheets. The practical effect is that the people who can most easily use this data are those with the technical skills to work around its formatting — a form of enclosure through interface design. ::: {.callout-note} ## What to look for When you load a new dataset, the problems you encounter in the first few cells — extra header rows, currency strings, aggregated totals, merged cells — are often symptoms of enclosure. The dataset was designed for a different audience than you. ::: ### Erosion: Infrastructure Decay Made Concrete **Erosion** happens when data infrastructure degrades over time, typically because no single agency is responsible for maintaining consistency across systems. The county name mismatch in the course datasets is a concrete example. Three Colorado government agencies each publish data about Colorado counties: - The CDC WONDER database formats county names as `"Adams County, CO"`. - The Colorado Bureau of Investigation uses `"Adams County"`. - The Colorado State Demography Office uses bare names: `"Adams"`. No one decided that these formats should differ. They are the accumulated result of three agencies building their own systems, at different times, with different conventions and no coordination mandate. The mismatch is invisible until you try to join the tables — at which point it silently drops every row, because `"Adams County, CO"` does not equal `"Adams County"` does not equal `"Adams"`. This is infrastructure erosion made concrete: a technical inconvenience that embodies a real governance problem. The cost is borne by anyone who tries to analyze these datasets together — which is exactly the analysis you would want to do if you were studying, say, the relationship between crime rates, death rates, and population in Colorado counties. ::: {.callout-note} ## What to look for When joining tables from different agencies, always audit your key columns before joining. `SELECT DISTINCT county FROM deaths ORDER BY county;` run against all three tables will immediately reveal format mismatches that would otherwise produce a silent empty result. ::: ### Exemption: What Data Does Not Exist **Exemption** is the hardest dynamic to see, because it involves absence rather than presence. It happens when data that would be useful for public accountability is not collected, not retained, or not released. The CDC suppression rule is a visible example of exemption by policy. When death counts in a county-year-cause cell are below 10, the CDC reports the value as `"Suppressed"` rather than a number. The stated reason is privacy protection — individuals in a small cell might be identifiable. This is a legitimate concern, but the effect is that the counties with the smallest populations — often rural counties, often with higher rates of specific causes of death — have the most missing data. The places where data would be most informative are the places where data is least available. Less visible forms of exemption include data that is never collected at all: housing displacement records at the parcel level, detention facility population data, stop-and-frisk incident logs in jurisdictions that do not publish them. These absences are not accidents either. They reflect choices about what is worth measuring and who is worth measuring it for. ::: {.callout-note} ## What to look for When a column has missing values, ask why. `"Suppressed"` in the CDC data is not a data quality problem — it is a policy decision. Understanding *why* data is missing is different from knowing *that* it is missing, and it matters for how you handle it analytically and how you communicate about it. ::: ## Three Pillars of Public Interest Data Practice Understanding these dynamics does not make data analysis impossible. It makes it more honest. Three practices, which appear throughout this book, help translate awareness into better work. ### Openness — The Known Issues Log **Openness** means making your data problems visible rather than hiding them. The Known Issues Log introduced in @sec-documentation is a transparency tool: a running record of every quality problem you found, what you decided to do about it, and why. Publishing a Known Issues Log alongside your analysis lets readers evaluate your decisions rather than taking your conclusions on faith. It also protects you: documenting that you knew about a data problem and addressed it deliberately is very different from having it discovered later as an unacknowledged flaw. Openness is not about being embarrassed by data problems. Every real-world dataset has them. It is about treating data quality as a fact to communicate rather than a weakness to conceal. ### Oversight — The Join-Quality Report **Oversight** means auditing your data pipelines to catch silent failures. The join-quality report introduced in @sec-sql-reshaping is an audit trail: a structured check of what matched, what was dropped, and whether the result is plausible. A join that silently drops 40% of rows because of a county name format mismatch is not a software bug — it is a logical error that produces a confident-looking wrong answer. Oversight practices — checking row counts before and after joins, validating totals, auditing unmatched records — are what prevent silent errors from becoming invisible conclusions. In a public interest context, the stakes are higher. An analysis of county-level health outcomes that silently drops rural counties is not only technically wrong — it is systematically biased against the populations that are hardest to count. ### Ownership — The Data Dictionary **Ownership** means tracking provenance: where data came from, who collected it, when it was accessed, and what decisions were made about it. The Data Dictionary introduced in @sec-documentation is a provenance record. Provenance matters for reproducibility — someone else should be able to reconstruct your analysis from the same sources. It also matters for accountability. Knowing that a dataset comes from the Colorado Department of Revenue's Marijuana Enforcement Division, downloaded on a specific date from a specific URL, is different from knowing only that you "got the cannabis data from the state." In an era of frequently updated government datasets, provenance records also let you know whether your analysis is based on data that has since been revised or retracted. ## How These Pillars Appear in This Book The three pillars are not add-ons — they are woven into the recurring practices that appear in every chapter of this book. | Recurring practice | Pillar | Where introduced | |--------------------|--------|-----------------| | Known Issues Log | Openness | @sec-documentation | | Data Dictionary | Ownership | @sec-documentation | | Post-load checklist | Openness | @sec-py-loading | | Validate after every transformation | Oversight | throughout | | Join-quality report | Oversight | @sec-sql-reshaping | | County name standardization | Oversight (erosion response) | @sec-sql-cleaning | | `na_values` documentation | Ownership | @sec-py-loading | | Documenting why data is missing | Openness | @sec-py-cleaning | Reading the technical chapters with these pillars in mind transforms a series of data manipulation exercises into a coherent practice. You are not only learning how to rename columns and join tables. You are learning how to be accountable for what you find. ## Common Issues in This Chapter ::: {.callout-warning} ## You treat missing values as random **What it means:** Code that drops NaNs or fills them with a mean assumes the missingness is random. When the missingness is systematic — the CDC suppression rule, for example — that assumption biases your analysis. **How to fix it:** Before any missing-value imputation, ask *why* the value is missing. Document the answer in your Known Issues Log. Treat policy-driven suppression differently from data-entry omissions. ::: ::: {.callout-warning} ## A join silently drops rows because of format mismatches **What it means:** You joined two agency tables on `county`, but one uses `"Adams County, CO"` and the other uses `"Adams"`. The result is confident-looking, systematically wrong. **How to fix it:** Run a join-quality report (@sec-sql-reshaping): compare row counts before and after the join, and list any unmatched keys. Standardize keys in a cleaning step before joining. ::: ::: {.callout-warning} ## Your analysis does not account for absent data **What it means:** The counties or demographic groups with the most missing rows are often the ones your analysis would most benefit from including. Silence about absence is not the same as representativeness. **How to fix it:** When you drop or exclude rows, document what was excluded and speculate about what the exclusion means for your conclusions. A limitation section in your writeup is not weakness — it is credibility. ::: ## Key Takeaways - Public data is shaped by three recurring dynamics — **enclosure**, **erosion**, and **exemption** — that are invisible until you know to look for them. - Three practices — **openness** (Known Issues Log), **oversight** (join-quality reports), and **ownership** (Data Dictionary) — translate awareness into better analysis. - The technical habits in this book (inspect before acting, validate after every transform, document your decisions) are not separate from ethics. They are what ethical data practice looks like in code. - Good data work makes absent data and uncertain decisions visible, not invisible. ## Further Reading ::: {.callout-tip collapse="true"} ## 📚 Going deeper on public interest data practice - **"Data Feminism" by D'Ignazio and Klein** — a free online edition at [data-feminism.mitpress.mit.edu](https://data-feminism.mitpress.mit.edu/) develops this book's themes at length. - **"Weapons of Math Destruction" by Cathy O'Neil** — an accessible survey of how opaque algorithms and missing data harm communities. - **Ada Lovelace Institute: Data justice** — [adalovelaceinstitute.org](https://www.adalovelaceinstitute.org/) publishes short, readable reports on data governance and equity. - **Investigative Reporters and Editors: Data library** — [ire.org](https://www.ire.org/) has public-interest datasets and story pitches. - **CDC WONDER documentation** — [wonder.cdc.gov/wonder/help/main.html](https://wonder.cdc.gov/wonder/help/main.html) documents the suppression rules referenced above. ::: ## Exercises ::: {.callout-note collapse="true"} ## Exercise 1: Identify the dynamic For each of the following data problems, identify which dynamic (enclosure, erosion, or exemption) it represents and explain your reasoning: 1. A county health department publishes monthly reports as PDF scans of printed tables. 2. Two state agencies use different ZIP code formats — one includes the ZIP+4 suffix, the other does not — making it impossible to join their records directly. 3. A police department publishes use-of-force data only in aggregate, not at the incident level. 4. A state unemployment database uses `999` to indicate "not applicable" in some columns and `null` in others. ::: ::: {.callout-note collapse="true"} ## Exercise 2: The absent dataset Look at the datasets described in @sec-datasets. For each dataset, identify one question you would like to answer that the dataset cannot answer — either because the relevant data is not collected, is suppressed, or is not in this dataset. For each gap, suggest what additional data source might fill it and whether that source is likely to be publicly available. ::: ::: {.callout-note collapse="true"} ## Exercise 3: Known Issues Log entry Load the Colorado cannabis sales dataset as described in @sec-py-loading. Write a Known Issues Log entry for the suppressed `NR` values in the sales columns. Your entry should document: (1) what the problem is, (2) which rows are affected, (3) what you decided to do about it, and (4) why that decision is defensible. :::