10 Oversight
In 2018, Joy Buolamwini and Timnit Gebru published “Gender Shades,” a disaggregated audit of three commercial gender-classification systems (Buolamwini and Gebru 2018). The systems misclassified darker-skinned women far more often than lighter-skinned men, and every vendor’s single aggregate accuracy number had hidden the gap. The paper mattered not only because of what it found but because of what followed: sustained pressure from advocates, company decisions to pause face-recognition sales, the end-to-end auditing framework that Raji and colleagues (2020) later formalized, and legislation that cited the work by name. An audit is a technique. Oversight is an arrangement. You will learn the technique of the disaggregated audit in Chapter 12, where the assurance lineage teaches it at the level of the state. This chapter is about the arrangement, and in particular about its last and most neglected link: remedy.
Keegan (2026) puts the point bluntly: “Evidence matters only if it can trigger remedy.” Oversight, in the installed-base framework, is the working configuration in which records, authority, and remedy meet. Remove authority and you have findings nobody can act on. Remove remedy and you have an authority that can document harm but cannot repair it. The failure mode this chapter makes visible is the remedy that exists on paper and nowhere else: an appeal right nobody knows about, a complaint channel that routes to an empty inbox, a court pathway whose cost exceeds the value of the record.
Data about exactly this failure is already public. Boulder County’s Public Records Archive posts Colorado Open Records Act requests that others filed and the county answered (Chapter 9): some in full, some partially, some late, some not at all. Each response is a small act of oversight or of exemption, and each carries a legal basis and a set of escalation options. In this chapter you turn the released requests you chose into an exemptions and remedy ledger: which exemptions were cited, how often, by which office, what pathway each left open, and, where the record shows it, what happened next.
10.1 Routine auditability versus episodic audits
Two arrangements get called “audits,” and they are not the same thing.
Routine auditability is built into a system. Logs are kept in a form an outsider can parse. Versions are archived. Metrics are recomputed and published on a schedule. When someone wants to check, the inputs already exist and nobody’s permission is needed. Public accounting aspires to this in principle, and Baker’s (2005) history of “the public interest” in American accounting is a warning that audit professions do not stay public-interested on their own.
Episodic audits are one-off events. A regulator commissions a report, a researcher publishes a paper, a reporter files a records request and runs the numbers. These can be powerful, but their power is bounded by the moment and by the audited party’s willingness to cooperate twice. Gender Shades was episodic. The follow-up that turned it into sustained pressure was closer to routine.
Much of the 2020s legislative wave sits uneasily between the two. New York City’s Local Law 144, which took effect in 2023, requires “bias audits” of automated employment decision tools and publication of impact ratios. It does not say what counts as an adequate auditor, how data must be collected, or what happens when an audit finds a problem, and an early empirical study found that few covered employers posted audits at all. The law created the appearance of routine auditability without authority to enforce it. Episodic audits pretending to be routine is what Goodman and Trehu (2023) call audit-washing. Records requests have the same structure. A single request is episodic. A class that codes the county’s released requests every term, logs every response in the same schema, and publishes the pattern is building something closer to routine auditability of a county’s records practice.
Oversight depends on two elements that no audit or records request can supply by itself: authority and remedy. When you read an audit report or a records denial from a county, your first two questions should not be about method. They should be: who has authority to act on this, and what remedy is available to the people affected? If the answers are “nobody” and “none,” you are reading audit-washing or its records equivalent. The specific claim of this chapter is that a remedy ledger advances oversight by converting each withheld record into a named pathway with a clock, so that a county’s exemptions become a pattern an authority can be asked to act on rather than a series of private disappointments.
10.2 Remedy as civic procedure
Keegan (2026) calls the relevant element remedy justice and treats escalation pathways (complaint channels, audit triggers, appeals, reporting cadences, enforcement hooks) as “usable civic procedure.” A pathway counts only if it is accessible in language, time, and cost; protective against retaliation; and consequential, with clear timelines and someone with authority to decide. A pathway that fails any of the three is decorative.
Apply that test to CORA. Colorado’s remedy ladder for a records denial is short and steep (Colorado Freedom of Information Coalition 2024):
- Ask. Request a written statement of the grounds for denial, citing the legal authority. For criminal justice records, ask the custodian to articulate the CCJRA public-interest balancing.
- Negotiate. Narrow the request, clarify it, or ask for an itemized fee estimate.
- Give notice. Before going to court, send the custodian written notice of intent to file, generally at least fourteen days ahead.
- Go to court. Apply to the district court, which must hold a hearing promptly. A requester who prevails is generally entitled to court costs and reasonable attorney fees.
There is no administrative appeal in between. That makes the ladder consequential (a court can order disclosure and fees) but not very accessible: the step after “negotiate” is litigation. In practice, most requesters stop at step two. Alongside the legal ladder runs a political one: the county’s own guidance invites requesters to call the County Attorney’s Office for help with the process, the Board of County Commissioners takes public comment, and a pattern of withholding is a story a local newsroom can tell. Chapter 11 is about using the second ladder.
10.3 Tutorial: an exemptions and remedy ledger
The tutorial uses a small fixture so that you can run it before you have coded your archived requests. The fixture is invented for teaching. Its offices are labeled A, B, and C, and its rows do not describe any real Boulder County request, office, or response. Once your tracker from Chapter 9 holds the released requests you chose, you replace the fixture with it and rerun everything.
Each row is one citation: a request can cite several grounds, and a request with no withholding still gets a row.
import sqlite3
import numpy as np
import pandas as pd
# ILLUSTRATIVE FIXTURE: invented for teaching. These rows do not describe
# any real Boulder County request, office, or response.
cols = ["request_id", "office", "regime", "submitted", "responded",
"outcome", "cited_as"]
citations = pd.DataFrame([
(1, "Office A", "CORA", "2027-01-28", "2027-02-02", "full", None),
(2, "Office A", "CORA", "2027-01-28", "2027-02-11", "partial", "Trade secret / confidential commercial info"),
(2, "Office A", "CORA", "2027-01-28", "2027-02-11", "partial", "personnel file"),
(3, "Office B", "CORA", "2027-01-28", "2027-02-11", "partial", "trade secrets"),
(4, "Office B", "CORA", "2027-01-28", None, "none", None),
(5, "Office C", "CCJRA", "2027-01-28", "2027-02-09", "denied", "CCJRA - contrary to public interest"),
(6, "Office C", "CCJRA", "2027-01-28", "2027-02-09", "partial", "Contrary to the public interest (CCJRA)"),
(6, "Office C", "CCJRA", "2027-01-28", "2027-02-09", "partial", "ongoing investigation"),
(7, "Office A", "CORA", "2027-01-28", "2027-02-04", "fee_hold", None),
(8, "Office B", "CORA", "2027-01-29", "2027-02-03", "partial", "attorney-client privileged"),
], columns=cols)
print(citations.shape)
# => (10, 7)Custodians do not write exemptions in a controlled vocabulary. “Trade secret / confidential commercial info” and “trade secrets” are the same ground in different words. Normalize them into categories, but keep the original text beside the category, because the custodian’s wording is the record and your category is an interpretation.
CATEGORY = {
"trade secret": "trade_secret",
"personnel": "personnel",
"public interest": "ccjra_balancing",
"investigation": "investigation",
"attorney-client": "privilege",
}
def categorize(text):
t = text.lower()
hits = [cat for key, cat in CATEGORY.items() if key in t]
return hits[0] if hits else "UNMAPPED"
citations["category"] = citations["cited_as"].map(categorize, na_action="ignore")
counts = (citations.dropna(subset=["category"])
.groupby("category")
.agg(times_cited=("request_id", "size"),
requests=("request_id", "nunique"),
offices=("office", "nunique"))
.sort_values("times_cited", ascending=False, kind="stable"))
print(counts)
# => times_cited requests offices
# => category
# => ccjra_balancing 2 2 1
# => trade_secret 2 2 2
# => investigation 1 1 1
# => personnel 1 1 1
# => privilege 1 1 1The "UNMAPPED" fallback is deliberate. A new phrasing you did not anticipate should show up loudly in the counts, not vanish into a category it does not belong to.
Next, collapse to one row per request and compute how many working days each took, measured against CORA’s ten-working-day outer limit from Chapter 9. An unanswered request is measured up to an “as of” date.
AS_OF = pd.Timestamp("2027-02-16")
req = (citations.groupby("request_id")
.agg(office=("office", "first"), regime=("regime", "first"),
submitted=("submitted", "first"),
responded=("responded", "first"),
outcome=("outcome", "first"),
exemptions=("category",
lambda s: ", ".join(sorted(s.dropna().unique()))))
.reset_index())
start = pd.to_datetime(req["submitted"]).values.astype("datetime64[D]")
end = (pd.to_datetime(req["responded"]).fillna(AS_OF)
.values.astype("datetime64[D]"))
req["workdays"] = np.busday_count(start, end)
req["past_clock"] = req["workdays"] > 10
print(req[["request_id", "office", "outcome", "exemptions",
"workdays", "past_clock"]].to_string(index=False))
# => request_id office outcome exemptions workdays past_clock
# => 1 Office A full 3 False
# => 2 Office A partial personnel, trade_secret 10 False
# => 3 Office B partial trade_secret 10 False
# => 4 Office B none 13 True
# => 5 Office C denied ccjra_balancing 8 False
# => 6 Office C partial ccjra_balancing, investigation 8 False
# => 7 Office A fee_hold 5 False
# => 8 Office B partial privilege 3 FalseFinally, attach a pathway to every outcome and store the three tables in SQLite, where they can be joined with the request tracker you built in Chapter 9.
PATHWAYS = pd.DataFrame([
("full", "none needed", "record and archive"),
("partial", "written grounds; narrow; 14-day notice; court", "ask for written grounds"),
("denied", "written grounds; 14-day notice; court", "ask for written grounds"),
("none", "reminder; 14-day notice; court", "send reminder citing clock"),
("fee_hold", "itemized estimate; narrow; ask for waiver", "ask for itemized estimate"),
], columns=["outcome", "pathway", "next_step"])
with sqlite3.connect("remedy_ledger.sqlite") as conn:
citations.to_sql("citations", conn, if_exists="replace", index=False)
req.to_sql("requests", conn, if_exists="replace", index=False)
PATHWAYS.to_sql("pathways", conn, if_exists="replace", index=False)
ledger = pd.read_sql("""
SELECT r.request_id, r.office, r.outcome, r.past_clock, p.next_step
FROM requests r JOIN pathways p USING (outcome)
WHERE r.outcome != 'full'
ORDER BY r.past_clock DESC, r.request_id
""", conn)
print(ledger.to_string(index=False))
# => request_id office outcome past_clock next_step
# => 4 Office B none 1 send reminder citing clock
# => 2 Office A partial 0 ask for written grounds
# => 3 Office B partial 0 ask for written grounds
# => 5 Office C denied 0 ask for written grounds
# => 6 Office C partial 0 ask for written grounds
# => 7 Office A fee_hold 0 ask for itemized estimateThe last table is the ledger’s working face: every request that did not come back whole, sorted so that the one past its statutory clock sits on top, with the next step named. Add an outcome_of_escalation column wherever the archive or other public records show a requester taking a step, and the ledger becomes a record of what the remedy ladder actually delivered.
Three choices in this tutorial look technical and are not. First, what you count: counting citations rather than requests makes an office that cites four grounds in one letter look four times as secretive as an office that denies outright on one. Report both, as times_cited and requests do. Second, which clock: the fixture applies CORA’s working-day clock to every row, including the CCJRA rows. Whether the same deadlines govern criminal justice records is a question to check against the statute, not to settle by default. Third, what “past the clock” means: np.busday_count skips weekends but not county holidays unless you pass them in. A ledger that accuses an office of lateness on the strength of a forgotten holiday has handed the office its rebuttal.
10.4 Reading the ledger
Even a fixture this small supports a few honest sentences. Trade-secret claims appeared at two of three offices, which suggests a contracting pattern rather than one office’s habit. The criminal-justice office leaned on the CCJRA balancing test, which means the remedy question there is whether the custodian articulated the balance, not whether an exemption exists. One request went unanswered past the outer limit, which is the single clearest fact in the table and the one a commissioner could act on fastest. And one request was held behind a fee estimate, which is not an exemption at all but functions like one if the requester cannot pay.
Notice what the ledger cannot say. It cannot tell you whether a withholding was lawful; only a court can. It cannot tell you whether eight requests represent the county’s practice; they do not. What it can do is make a pattern followable, in Keegan’s (2026) word, so that the next requester, reporter, or commissioner starts from your record instead of from nothing.
10.5 When the state is the adversary
Remedy looks different when the party you seek remedy from also controls the records. In SyRI, the Dutch coalition did not ask the ministry to fix its model. It went to court, and the remedy was structural: the system was stopped (District Court of The Hague 2020). In the toeslagenaffaire, remedy arrived only after journalists and a parliamentary inquiry reconstructed what the tax authority would not explain, and it took the political form of a cabinet resignation and the slow, contested financial form of compensation (Amnesty International 2021). Remedy can be restorative (reinstating a benefit), structural (decommissioning a tool), or financial (damages and fees), and each needs a different authority.
Allegheny County illustrates a different arrangement. The county has revised AFST over time and published evaluations and ethical reviews alongside it, which is closer to routine auditability than most counties attempt. But the pathways that matter most to a screened-in family run through the child-welfare process itself, and the most consequential external review, the federal civil-rights scrutiny reported in 2023, produced no public record (Ho and Burke 2023). Disclosure by the county is not the same as remedy for the family.
10.6 What survives political change
Three things distinguish oversight regimes that survive political change from those that collapse into theater. The first is authority: a named institution with a budget and a mandate that can demand evidence and act on it. For a county, that might be the Board of County Commissioners, a county auditor, the district attorney, or a state court. When no such authority is willing, a ledger is an artifact looking for a reader.
The second is remedy that reaches the affected. Local Law 144’s core defect is that it requires disclosure of disparate impact but gives no right of action to a candidate screened out by a tool with a poor impact ratio (Goodman and Trehu 2023). CORA’s strength is the reverse: a requester who wins in court can recover attorney fees. Its weakness is that nothing sits between a custodian’s denial and a lawsuit.
The third is scope protection. Regimes erode when their scope narrows and nobody defends the frame: “advisory” systems that are decisive in practice, “pilot programs” that run for a decade, records reclassified from administrative to criminal justice. Chapter 8 treated this as the logic of exemption. A regime whose scope is a negotiated floor will be negotiated downward by the entities it constrains. A ledger maintained across terms is one of the few instruments that can show the floor moving.
10.7 Exercises
Exercise 10.1 (Guided). Run the fixture tutorial end to end. Then add two rows of your own invention (still labeled as fixture), one of which uses a phrasing that falls through to "UNMAPPED". Fix the CATEGORY map, rerun, and write two sentences on why the fallback exists.
Exercise 10.2 (Piece 2 component). Replace the fixture with the released requests you chose from Boulder County’s Public Records Archive, and their comparison requests, from the tracker in Chapter 9. Record every ground exactly as the custodian wrote it, normalize it, compute clocks using the county’s holiday calendar, and attach the next step a requester could take. If a request came back in full or not at all, the ledger still has rows: a full release and a silence are both outcomes. Save as piece2/remedy_ledger.sqlite with a short piece2/remedy_ledger.md that states, in five sentences, the pattern and its limits. This is the exemptions and remedy ledger for Piece 2.
Exercise 10.3 (Runnable, real public data). Your Piece 2 ledger centers on one office. Sample at least ten released requests to other offices from the same Public Records Archive, code each one’s outcome and any grounds cited with the same schema, and run the counts. Compare the pattern with your Piece 2 ledger. What does the published archive leave out by design?
Exercise 10.4 (Analytic). Pick one rung of Colorado’s remedy ladder and evaluate it against Keegan’s three tests: accessible, protective, consequential. Use the Colorado Freedom of Information Coalition guide and at least one other source. In 500 words, name the rung’s strongest and weakest dimension and one change, by statute or by county policy, that would strengthen the weakest.
Exercise 10.5 (Open-ended). Design an escalation pathway that Boulder County could adopt without new state law: for example, a published log of CORA denials with grounds, a response-time report to the Board, or an ombuds contact for records disputes. In 600 words, specify who maintains it, who can trigger it, what happens when it is triggered, and how you would know in two years whether it worked. This is a candidate ask for your testimony.
10.8 Looking ahead
A remedy ledger is evidence. It becomes pressure only when someone with authority hears it. The next chapter, Chapter 11, closes Module 2 with the genre built for that moment: testimony before a Boulder County Board of County Commissioners hearing, delivered in a few minutes and preserved in a written statement for the record. After that, Part IV climbs to the state and to the assurance lineage, where Chapter 12 teaches the disaggregated audit that this chapter set aside.
10.9 Further Reading and Resources
- Keegan (2026), “Public interest data infrastructuring,” especially “Authority, trust, and contestable oversight” and “Remedy justice” (Keegan 2026).
- Colorado Freedom of Information Coalition, open government guide: https://coloradofoic.org/open-government-guide/. Deadlines, fees, and the court remedy under CORA and the CCJRA (Colorado Freedom of Information Coalition 2024).
- Boulder County, Colorado Open Records Act page: https://bouldercounty.gov/records/colorado-open-records-act/. Links to the Open Records Center and its archive of prior releases.
- Laura Beth Nielsen and Catherine Albiston (2006), “The organization of public interest practice” (Nielsen and Albiston 2006). How the institutions that pursue remedy are funded and staffed.
- District Court of The Hague (2020), SyRI judgment, ECLI:NL:RBDHA:2020:1878, https://uitspraken.rechtspraak.nl/details?id=ECLI:NL:RBDHA:2020:1878 (District Court of The Hague 2020).
- Amnesty International (2021), Xenophobic Machines, https://www.amnesty.org/en/documents/eur35/4686/2021/en/ (Amnesty International 2021).
- Inioluwa Deborah Raji et al. (2020), “Closing the AI accountability gap,” Proceedings of FAT* 2020 (Raji et al. 2020).
- Bryce Goodman and Julia Trehu, “AI audit washing and accountability” (2023) (Goodman and Trehu 2023).
- NYC Department of Consumer and Worker Protection, Local Law 144 guidance: https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page.
- C. Richard Baker (2005), “What is the meaning of ‘the public interest’?,” Accounting, Auditing & Accountability Journal (Baker 2005).