Web Data Science

Retrieving, Parsing, and Analyzing Data from the Web with Python

Author

Brian C. Keegan, Ph.D.

Published

July 2026

Preface

The internet makes many kinds of information easy to access. Public agencies publish data through APIs, social platforms expose behavioral traces through developer tools, news organizations structure their archives in parseable HTML, and government bodies release records as downloadable documents. The ability to retrieve, parse, and analyze this information is a valuable and increasingly contested skill.

This book provides an overview of computational tools and practices for transforming web documents and APIs into data for common research designs. It is designed for advanced undergraduates and early-career master’s students who have some experience with Python programming — you know loops, functions, lists, and dictionaries — but have not yet worked extensively with web protocols, HTML parsing, or API access.

What You Will Learn

By the end of this book, you will be able to:

Retrieve data from static and dynamic web pages, archived documents, and PDF files
Access and query web APIs including Wikipedia, the U.S. Census Bureau, FRED, the FEC, Reddit, Spotify, Bluesky, Mastodon, OpenAI, and Anthropic
Navigate the legal and ethical landscape of web data collection
Automate data collection pipelines using GitHub Actions
Design research projects that combine multiple web data sources
Reason critically about the structural forces — enclosure, exemption, and erosion — that are reshaping access to web data

How This Book Is Organized

The book is divided into four parts. Part I (Foundations) introduces the computing environment, ethical and legal frameworks, the changing landscape of data access, web data formats, and internet protocols. Part II (Documents) covers extracting data from static web pages, archived pages, dynamic pages, and PDFs. Part III (APIs) teaches you to work with Wikipedia, government, social media, and AI APIs. Part IV (Practice) addresses automation and research design.

Each chapter follows a consistent structure: learning objectives, conceptual motivation, a guided tutorial with narrative code, exercises, a discussion of social history and public interest themes, common debugging issues, key takeaways, and links to further reading. Because this book serves a combined undergraduate and graduate course (INFO 4617/5617), each chapter’s final exercise is a graduate extension — a scholarly reading paired with a more open-ended task — intended for the 5000-level section, though ambitious undergraduates are welcome to attempt them.

Every chapter also has a companion Jupyter Notebook containing the chapter’s code and narrative, ready to download and run as you read. See Appendix A — Companion Notebooks for the full list.

Prerequisites and Companion Resources

This book assumes you have Python 3.10 or above installed via the Anaconda distribution, and that you are comfortable working in Jupyter Notebooks. If you need to set up or troubleshoot your computing environment, the Missing Manual for Information Scientists is a companion resource that covers the hidden curriculum of computing technologies — from operating systems and file management to debugging, version control, and secrets management. This book references specific chapters of the Missing Manual throughout.

Code blocks in this book are narrative: they include comments explaining what each line does, but they are set to not execute automatically. You should run the code yourself in a Jupyter Notebook or Python script to see the results. This design choice reflects a pedagogical commitment: running code, seeing it fail, and debugging it is where the learning happens.

Acknowledgements

This book grew out of the INFO 4871/5871: Web Data Science course at the University of Colorado Boulder. It has benefited from the questions, frustrations, and insights of many cohorts of students. The author also gratefully acknowledges the journalists, researchers, and civic technologists whose work demonstrates why web data fluency matters for the public interest.

AI Disclosure

Portions of this book were drafted with the assistance of large language model tools. All content has been reviewed, edited, and verified by the author. For more on responsible AI disclosure in academic work, see Appendix B — AI Coauthorship and Responsible Disclosure.

# Preface {.unnumbered} The internet makes many kinds of information easy to access. Public agencies publish data through APIs, social platforms expose behavioral traces through developer tools, news organizations structure their archives in parseable HTML, and government bodies release records as downloadable documents. The ability to retrieve, parse, and analyze this information is a valuable and increasingly contested skill. This book provides an overview of computational tools and practices for transforming web documents and APIs into data for common research designs. It is designed for advanced undergraduates and early-career master's students who have some experience with Python programming — you know loops, functions, lists, and dictionaries — but have not yet worked extensively with web protocols, HTML parsing, or API access. ## What You Will Learn By the end of this book, you will be able to: - Retrieve data from static and dynamic web pages, archived documents, and PDF files - Access and query web APIs including Wikipedia, the U.S. Census Bureau, FRED, the FEC, Reddit, Spotify, Bluesky, Mastodon, OpenAI, and Anthropic - Navigate the legal and ethical landscape of web data collection - Automate data collection pipelines using GitHub Actions - Design research projects that combine multiple web data sources - Reason critically about the structural forces — enclosure, exemption, and erosion — that are reshaping access to web data ## How This Book Is Organized The book is divided into four parts. **Part I (Foundations)** introduces the computing environment, ethical and legal frameworks, the changing landscape of data access, web data formats, and internet protocols. **Part II (Documents)** covers extracting data from static web pages, archived pages, dynamic pages, and PDFs. **Part III (APIs)** teaches you to work with Wikipedia, government, social media, and AI APIs. **Part IV (Practice)** addresses automation and research design. Each chapter follows a consistent structure: learning objectives, conceptual motivation, a guided tutorial with narrative code, exercises, a discussion of social history and public interest themes, common debugging issues, key takeaways, and links to further reading. Because this book serves a combined undergraduate and graduate course (INFO 4617/5617), each chapter's final exercise is a **graduate extension** — a scholarly reading paired with a more open-ended task — intended for the 5000-level section, though ambitious undergraduates are welcome to attempt them. Every chapter also has a companion Jupyter Notebook containing the chapter's code and narrative, ready to download and run as you read. See @sec-appendix-notebooks for the full list. ## Prerequisites and Companion Resources This book assumes you have Python 3.10 or above installed via the Anaconda distribution, and that you are comfortable working in Jupyter Notebooks. If you need to set up or troubleshoot your computing environment, the [*Missing Manual for Information Scientists*](https://cuinfoscience.github.io/INFO-Missing-Manual/) is a companion resource that covers the hidden curriculum of computing technologies — from operating systems and file management to debugging, version control, and secrets management. This book references specific chapters of the *Missing Manual* throughout. Code blocks in this book are narrative: they include comments explaining what each line does, but they are set to not execute automatically. You should run the code yourself in a Jupyter Notebook or Python script to see the results. This design choice reflects a pedagogical commitment: running code, seeing it fail, and debugging it is where the learning happens. ## Acknowledgements This book grew out of the INFO 4871/5871: Web Data Science course at the University of Colorado Boulder. It has benefited from the questions, frustrations, and insights of many cohorts of students. The author also gratefully acknowledges the journalists, researchers, and civic technologists whose work demonstrates why web data fluency matters for the public interest. ::: {.callout-note} ## AI Disclosure Portions of this book were drafted with the assistance of large language model tools. All content has been reviewed, edited, and verified by the author. For more on responsible AI disclosure in academic work, see @sec-appendix-ai-disclosure. :::