15  Research Design with Web Data

TipLearning Objectives

• Develop a research question suited to web data and match it to appropriate data sources
• Design a multi-stage data collection plan combining scraping, APIs, and supplementary data
• Evaluate the sampling, representativeness, and validity challenges specific to web data
• Build a reproducible research pipeline from data collection through analysis
• Communicate findings to different audiences with appropriate evidentiary standards
• Apply the values of openness, oversight, and ownership to your own research practice

TipCompanion Notebook

Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).

15.1 Technical Skill Is Not Enough

You now command a substantial technical toolkit: you can scrape static and dynamic web pages, parse PDFs, query APIs for platforms from Wikipedia to Spotify to the Federal Reserve, apply NLP and LLM-based analysis, and automate your data collection with GitHub Actions. These are powerful capabilities. But technical skill without research design produces data, not knowledge.

This chapter steps back from code to address the questions that determine whether your technically-excellent scraping produces genuine insight: What is the right question for this data source? What can this data actually tell you? What are the threats to the validity of your conclusions? How do you communicate your findings responsibly?

15.2 From Question to Data Source

The best web data science projects start with a question, not a dataset. The data source serves the question, not the other way around.

15.2.1 The Proposal Arc

A research proposal — whether for a class project, a thesis, a journal article, or a grant application — follows a consistent structure:

1. Motivation: What is the question or problem? Why does it matter? Who benefits from the answer?
2. Data source: What specific web data could address this question? What constructs does the data measure? What are the gaps?
3. Methods: How will you collect the data? What tools and techniques from this book will you use? What analytical approach will you apply?
4. Expected results: What kinds of analyses or visualizations will you produce? What patterns do you expect to find?
5. Implications: What should stakeholders do if your findings hold? What are the practical consequences?
6. Limitations: What can this data not tell you? What are the threats to validity? What alternative explanations exist?

Notice that limitations are not an afterthought — they are a core component of the design. Acknowledging what your data cannot do is as important as demonstrating what it can.

15.2.2 Matching Questions to Methods

Research Design	Web Data Methods	Example
Longitudinal tracking	Wayback Machine (Chapter 7), GitHub Actions (Chapter 14)	How has a platform’s privacy policy changed over time?
Cross-sectional snapshot	API queries (Chapter 10, Chapter 11, Chapter 12)	How does Reddit moderation differ across subreddits?
Content analysis	HTML scraping (Chapter 6) + LLM coding (Chapter 13)	What topics dominate legislative activity in Colorado?
Network analysis	Link extraction (Chapter 10), social graphs (Chapter 12)	How connected are Wikipedia articles about climate change?
Natural experiment	Event-triggered collection (Chapter 14)	How did a policy change affect web search patterns?

15.3 Common Research Designs with Web Data

The table above compresses a lot of decisions into single rows. Four of these designs come up so often in web data science that they deserve a closer look; choosing among them depends on your question, your data source, and the claims you want to make.

15.3.1 Longitudinal Tracking

Question type: How does X change over time?

You observed something on the web and want to track how it evolves — the content of a policy document, the composition of a playlist, the frequency of a term in news coverage, or the price of a product. Longitudinal tracking requires either archival data (the Wayback Machine in Chapter 7) or forward-looking automated collection (GitHub Actions in Chapter 14).

Design considerations: The Wayback Machine captures snapshots opportunistically, not on a fixed schedule, so your temporal resolution is uneven. Automated scrapers give you consistent intervals but require maintenance. In both cases, you need to account for changes in the underlying structure (a website redesign, an API version change) that might affect your measurements without reflecting genuine change in the phenomenon you are studying.

15.3.2 Cross-Sectional Snapshot

Question type: How does X vary across cases at a point in time?

You want to compare legislative activity across states, subreddit moderation policies across communities, or audio features across genres. Cross-sectional designs collect data at one point in time and analyze variation across units.

Design considerations: The main threat is selection bias — you are limited to the platforms, communities, and data sources that exist and are accessible. If a platform’s API does not expose the variable you need, you cannot include it. If a community is private or a government website does not publish the data you want, it is missing from your sample.

15.3.3 Content Analysis

Question type: What is the substantive content of a corpus?

You have collected a body of text — scraped web pages, PDF documents, social media posts — and want to characterize what it contains. Content analysis can be manual (human coding), automated (NLP in Chapter 7, LLM-assisted coding in Chapter 13), or a combination.

Design considerations: The reliability of automated coding should be validated against manual coding on a random sample. LLM-based classification is powerful but non-deterministic — run the same prompt twice and you may get different results. Always report inter-coder reliability or human-AI agreement rates.

15.3.4 Natural Experiments

Question type: What was the causal effect of an event or policy change?

Sometimes external events — a platform policy change, a natural disaster, a legislative decision — create natural variation that allows you to estimate causal effects. For example, you might compare Wikipedia editing behavior before and after a major news event, or Reddit community dynamics before and after a moderation policy change.

Design considerations: Causal claims from observational data are always tentative. You need a clear “treatment” event, a defensible comparison group, and controls for confounding variables. Web data makes it easy to observe change but does not, by itself, establish causation.

15.4 Sampling and Representativeness

Whichever design you choose, it inherits a common problem: web data is not a random sample of anything. It is a convenience sample shaped by who uses the platform, what the API exposes, and what your scraper can reach. Understanding these biases is essential for interpreting your findings.

Who is missing? Reddit users skew young, male, and American. Wikipedia editors skew educated, male, and from the Global North. Bluesky’s early adopters skew toward journalists and technologists. The data you retrieve from any platform reflects its user base, not the broader population.

What is hidden? APIs expose what platforms choose to expose. Reddit’s API does not reveal deleted posts, shadow-banned users, or the internal ranking algorithms that determine what content becomes visible. Spotify’s audio features are proprietary measurements, not raw audio data. What you can measure is constrained by what the platform makes available.

What has been lost? Longitudinal studies face the erosion problem from Chapter 3. Articles are deleted, accounts are suspended, pages are redesigned. The data available today may not be available tomorrow, and the data available yesterday may already be gone.

Methodologists group many of these concerns under the formal term construct validity: the degree to which the thing you can measure (pageviews, posts, search queries) actually corresponds to the concept you claim to study (attention, public opinion, illness) (Salganik 2018). Whenever this book asks “does your variable actually measure what you claim?”, it is asking about construct validity — and weak construct validity is the most common flaw in web data research.

The canonical cautionary tale is Google Flu Trends, which promised to estimate influenza prevalence from search queries and instead overestimated it by roughly a factor of two for more than a hundred consecutive weeks (Lazer et al. 2014). Part of the failure was big data hubris — treating a large behavioral dataset as a substitute for, rather than a supplement to, traditional measurement. The rest was algorithm dynamics: Google kept redesigning its search and suggestion features, so the measurement instrument drifted underneath the model even as the underlying phenomenon stayed put. Every automated collector you build (Chapter 14) is exposed to both failure modes.

15.5 Human Subjects and the IRB

If your research involves data about people — and most web data does — you need to know when it counts as human subjects research. This is not a question you get to answer for yourself, and the answer is less intuitive than many researchers expect.

Under the Common Rule, the federal regulation governing research ethics in the United States, human subjects research involves obtaining information about living individuals through interaction or intervention, or obtaining their identifiable private information. Scraped web data complicates every clause of that definition. A public post is not exactly private, but the person who wrote it did not expect it to land in a research dataset (Fiesler et al. 2020). A username is not exactly identifiable, but linking it across platforms often makes it so.

Three principles should guide you:

• Publicly available does not mean automatically exempt. “It was on the public web” is a statement about access, not about ethics or regulation. Institutional Review Boards (IRBs) frequently determine that analyses of scraped public data are “not human subjects research” or qualify for an exemption — but that determination belongs to the IRB, not to the researcher. If you catch yourself reasoning “this is obviously fine, so I will not ask,” you have identified exactly the moment to ask.
• Platform terms of service are not an ethics review. Complying with a ToS is a legal and contractual matter, as discussed in Chapter 2; it neither substitutes for nor settles the human subjects question.
• Data minimization and de-identification are default practice, not extra credit. Collect only the fields your question requires, strip or hash identifiers you do not need, and store anything sensitive separately from your analysis dataset. These habits shrink both the ethical risk and the consequences of any future leak.

Consult your IRB early whenever your design involves interacting with users (even sending messages from a research account), collecting from private or semi-private spaces (closed groups, invite-only servers), identifying specific individuals, or measuring sensitive attributes such as health, sexuality, immigration status, or political affiliation.

Expectations differ by context. For a class project in this course, your instructor and your campus’s guidance on student research are the right checkpoints. For research you intend to publish, graduate students should engage the IRB early — a determination request is usually a short form, and “our IRB reviewed the protocol and determined it was not human subjects research” is a sentence reviewers like to see. The ethics chapter of Salganik (2018) (and its historical appendix) is the best single treatment of these questions, Fiesler et al. (2020) documents how poorly platform rules align with either law or ethics, and Chapter 2 covers the legal landscape in more depth.

15.6 Data Quality and Validation

Web data is messy. Scrapers fail silently, APIs return incomplete results, and the same information can appear in different formats across sources. Build validation into your pipeline:

def validate_dataset(df, expected_columns, min_rows=10):
    """Basic validation checks for a scraped dataset.
    
    Parameters
    ----------
    df : pd.DataFrame
        The dataset to validate
    expected_columns : list
        Columns that should be present
    min_rows : int
        Minimum expected number of rows
    
    Returns
    -------
    bool
        True if all checks pass
    """
    checks = []
    
    # Check column presence
    missing_cols = set(expected_columns) - set(df.columns)
    if missing_cols:
        print(f"WARNING: Missing columns: {missing_cols}")
        checks.append(False)
    else:
        checks.append(True)
    
    # Check minimum rows
    if len(df) < min_rows:
        print(f"WARNING: Only {len(df)} rows (expected >= {min_rows})")
        checks.append(False)
    else:
        checks.append(True)
    
    # Check for excessive missing data
    null_pct = df.isnull().mean()
    high_null = null_pct[null_pct > 0.5]
    if not high_null.empty:
        print(f"WARNING: High null rates: {dict(high_null)}")
        checks.append(False)
    else:
        checks.append(True)
    
    return all(checks)

15.7 Reproducibility and Documentation

A reproducible research pipeline means that someone else — or you, six months from now — can re-run your code and get the same results. This requires:

• Version control: Your code, configuration, and documentation should be in a git repository (more on this below).
• Data documentation: For each dataset, record what it contains, how it was collected, when it was collected, and what its known quality issues are (Gebru et al. 2021).
• Environment specification: A requirements.txt or environment.yml file that pins the exact versions of your dependencies.
• Collection parameters: Document not just what data you collected but how: the API endpoints, query parameters, rate limiting, date ranges, and any filters you applied.

15.8 Managing the Research Process

These principles become concrete when you organize an actual project. Two habits — disciplined version control and systematic documentation — separate projects that can be revisited, audited, and extended from projects that become black boxes the moment you stop working on them.

15.8.1 Version Control for Research Projects

Your code, data documentation, and analysis notebooks should be version-controlled with git from the start. This is not just good practice — it is essential for reproducibility. When you revise your analysis after a reviewer’s feedback, you need to be able to trace exactly what changed and when.

my-project/
├── .gitignore           # Exclude api_keys.json, large data files
├── README.md            # Project overview and setup instructions
├── requirements.txt     # Python dependencies with pinned versions
├── data/
│   ├── raw/             # Original scraped data (never modify)
│   ├── processed/       # Cleaned and transformed data
│   └── README.md        # Data documentation (provenance, variables)
├── notebooks/
│   ├── 01_collection.ipynb
│   ├── 02_cleaning.ipynb
│   └── 03_analysis.ipynb
├── scripts/
│   └── scraper.py       # Production scraping scripts
└── output/
    ├── figures/
    └── tables/

TipMissing Manual Reference

For a complete guide to version control with Git and GitHub — commits, branches, pull requests, and collaboration workflows — see Missing Manual Chapter 31: Version Control.

15.8.2 Data Documentation

For every dataset you create, write a datasheet (Gebru et al. 2021) that answers:

• What: What does each variable measure? What are the units? What do missing values mean?
• How: What tools did you use? What parameters and filters did you apply? What date range does the data cover?
• Limits: What is not captured? What biases are likely present? What quality issues are you aware of?
• Ethics: Does the data contain personally identifiable information? Was it collected in accordance with the platform’s terms? Are there privacy risks if the dataset is shared?

This documentation is not busywork — it is the difference between a dataset that can be used by others (including future-you) and one that is a black box.

15.9 Communicating Findings

Web data science serves different audiences, each with different expectations:

• Academic papers require methodological precision, explicit limitations, and engagement with prior scholarship. Your data collection methods should be replicable from the description alone.
• Policy briefs need actionable findings, clear implications, and accessible language. Lead with the “so what?” rather than the methodology.
• Data journalism demands narrative structure, visual evidence, and accountability framing. The question is not just “what did you find?” but “what should the public know?”

In all cases, the visualizations you create with matplotlib and seaborn are not decorations — they are arguments. Every axis label, every color choice, every title should serve your analytical purpose.

15.9.1 Visualization as Argument

Consider the difference between a default plot and one designed to communicate a finding. A default plt.plot() call with no labels, no title, and no context forces the reader to guess what they are looking at. A well-designed figure makes its argument legible at a glance: the title states a claim, the annotations connect data patterns to real-world causes, and the axis labels include units so the reader can evaluate the evidence independently.

import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import pandas as pd

# Suppose you have monthly Wikipedia pageviews for "Affordable Care Act"
# with columns: 'month' (datetime) and 'pageviews' (int)
df = pd.read_csv('aca_pageviews.csv', parse_dates=['month'])

# Scale raw counts to thousands so the data matches the axis label below
pageviews_k = df['pageviews'] / 1000

fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(df['month'], pageviews_k, color='#2c3e50', linewidth=1.5)

# Annotate key events with vertical lines and text
ax.axvline(pd.Timestamp('2017-07-28'), color='#e74c3c', linestyle='--', alpha=0.7)
ax.text(pd.Timestamp('2017-08-15'), pageviews_k.max() * 0.92,
        '"Skinny repeal"\nvote fails',
        fontsize=9, color='#e74c3c')

ax.axvline(pd.Timestamp('2020-03-11'), color='#2980b9', linestyle='--', alpha=0.7)
ax.text(pd.Timestamp('2020-04-01'), pageviews_k.max() * 0.78,
        'WHO declares\nCOVID-19 pandemic',
        fontsize=9, color='#2980b9')

# Labels that include units and a title that states a finding
ax.set_title('Public attention to the ACA spikes around legislative and health crises',
             fontsize=12, fontweight='bold')
ax.set_xlabel('Month')
ax.set_ylabel('Monthly pageviews (thousands)')
ax.xaxis.set_major_locator(mdates.YearLocator())
ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y'))
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)

plt.tight_layout()
plt.savefig('aca_pageviews_annotated.png', dpi=150)
# plt.show()

Notice that the title does not say “Wikipedia Pageviews for Affordable Care Act Over Time” — that merely describes the axes. Instead, it states a finding: public attention spikes around legislative and health crises. The annotations connect specific peaks to specific events, transforming a line chart into an argument about the relationship between political events and public information-seeking behavior. The clean axis formatting and removal of chart junk (top and right spines) keep the reader’s attention on the data rather than on visual noise. When you build a figure, ask yourself: if a reader saw only this image and its title, would they understand the claim you are making?

NotePublic Interest Connection

Research design is where the values of Chapter 3 stop being abstractions and become decisions. Openness is a design choice: a pipeline built for reproducibility — pinned environments, documented collection parameters, archived raw data — stays verifiable even as the platforms beneath it erode. Oversight is a question choice: studies designed around accountability questions serve publics, not just literatures. Ownership is an infrastructure choice: collecting into datasets and tools that others can reuse means your project outlives your paper.

15.10 Social History and Public Interest

Every data source in this book carries a social history — someone fought to make it public, keep it public, or archive it before it vanished. As you design your own research, return to the values introduced in Chapter 3:

• Openness: Will you publish your code, data (to the extent ethically permissible), and methods so others can evaluate and build on your work?
• Oversight: Does your research contribute to independent accountability — helping citizens, journalists, or policymakers understand systems that affect them?
• Ownership: Are you building infrastructure (datasets, tools, pipelines) that others can use, or are you extracting value from public resources without contributing back?

These questions do not have formulaic answers. But asking them is itself a practice of responsible web data science.

15.10.1 Data Journalism as a Model

Some of the most compelling examples of web data science in practice come not from academia but from data journalism. Consider ProPublica’s “Dollars for Docs” investigation: it began with a straightforward question — which doctors receive payments from pharmaceutical companies? — and pursued it by scraping publicly available but practically inaccessible disclosure databases maintained by drug manufacturers. The team cleaned, standardized, and linked records across companies, then published a searchable database that allowed local journalists across the country to investigate payment patterns in their own communities. The project moved from question to data collection to analysis to public accountability, following the same arc that a strong research proposal describes.

Academic researchers can learn several things from this model. Data journalists are accountable to an audience that will scrutinize their methods and challenge their conclusions. This pressure produces methodological transparency: the best data journalism outlets publish their code, describe their scraping and cleaning procedures, and explain the limitations of their data. They also structure their findings as narratives rather than tables of regression coefficients, making complex analyses accessible to non-specialist readers without sacrificing rigor. The journalistic commitment to showing your work — publishing the underlying data, linking to source documents, and explaining analytical choices — aligns directly with the reproducibility standards discussed earlier in this chapter.

The practitioners who do this work well — computational social scientists, civic technologists, data journalists — share a common profile: they combine data fluency with deep domain expertise. They understand not just how to scrape a website or query an API but why the data matters, who it affects, and what questions it can and cannot answer. This integration of technical and substantive knowledge is precisely what this book aims to develop. The tools you have learned are necessary but not sufficient; pairing them with domain knowledge, ethical reflection, and a commitment to public accountability is what transforms data collection into genuine contribution.

15.11 Exercises

1. Research proposal. Write a 1,500-word research proposal for a web data science project. Include all six components: motivation, data source, methods, expected results, implications, and limitations. Be specific about which tools from this book you would use and why.

2. Validity threats. Identify three potential threats to validity in a hypothetical study that uses scraped social media data to measure public opinion. For each threat, propose a mitigation strategy.

3. Data documentation. Take a dataset you collected in an earlier chapter and write a “datasheet” following Gebru et al. (2021): what does each variable measure, how was it collected, what are the known quality issues, and how should it be cited?

4. Data limitations assessment. Choose a dataset you collected in an earlier chapter. Write a 500-word assessment of its limitations, covering: sampling bias (what is included and excluded?), measurement validity (does your variable actually measure what you claim?), temporal scope (what period does the data cover, and what might have changed?), and platform constraints (how does the platform’s design shape the data you can collect?).

5. Multi-method design. Design a research study that combines at least three techniques from this book (e.g., API data collection, web scraping, PDF extraction, text analysis, or automated collection). Write a one-page methods section that describes your data sources, collection procedures, analysis plan, and anticipated limitations. This should read like a methods section in an academic paper.

6. Graduate extension (INFO 5617). Write a complete pre-registration-style research design of two to three pages for a study you could actually execute. Specify the research question, data sources, sampling frame, and measurement strategy, making an explicit construct validity argument for each measure (Salganik 2018). Then confront the threats: enumerate the validity risks (including algorithm dynamics (Lazer et al. 2014)), sketch the IRB determination you would request and why, and write a contingency plan for the platform-erosion scenarios of Chapter 3. The discipline of specifying everything before collecting anything is the point — a design you cannot write down is a design you have not finished.

15.12 Common Pitfalls

Research design failures are harder to debug than code failures — there is no traceback, just a dataset that cannot answer your question. Watch for these:

• Collecting before defining the unit of analysis. Is a row of your dataset a post, a user, a day, or a community? Deciding after collection usually means collecting again.
• Sampling on the dependent variable. Studying only viral posts to learn what makes posts go viral tells you nothing without the posts that never took off.
• Ignoring bot and spam contamination. On many platforms, automated accounts generate a substantial share of raw activity; skipping this filter means measuring bots while making claims about people.
• Survivorship bias in longitudinal scrapes. The accounts, pages, and posts still visible at collection time are the survivors; deleted and suspended content is invisible precisely because something happened to it.
• Changing instrumentation mid-collection. Editing your scraper’s selectors, query parameters, or sampling logic partway through a collection creates artificial discontinuities that look like real change — the algorithm dynamics problem, self-inflicted.
• Validating only at analysis time. If your first quality check happens months after collection began, so does your discovery that a site redesign in June silently emptied half your columns.

15.13 Key Takeaways

Technical skill without research design produces data, not knowledge. Every project needs a clear question, a defensible data collection strategy, honest assessment of limitations, and responsible communication of findings. The web is the richest data source humanity has ever created; using it well is both a technical and an ethical challenge. The skills in this book — scraping, API access, PDF extraction, NLP, LLM analysis, automation — are means, not ends. The end is producing knowledge that serves openness, oversight, and ownership.

15.14 Further Reading

• Salganik (2018) — comprehensive treatment of digital-age research design
• Gebru et al. (2021) — a framework for documenting datasets
• Lazer et al. (2020) — obstacles and opportunities in computational social science
• Atteveldt et al. (2022) — methods for computational analysis of communication
• Rogers (2019) — qualitative and critical approaches to digital methods

Atteveldt, Wouter van, Damian Trilling, and Carlos Arcila Calderón. 2022. Computational Analysis of Communication. Wiley.

Fiesler, Casey, Nathan Beard, and Brian C. Keegan. 2020. “No Robots, Spiders, or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social Media Terms of Service.” Proceedings of the International AAAI Conference on Web and Social Media 14: 187–96.

Gebru, Timnit, Jamie Morgenstern, Brenda Vecchione, et al. 2021. “Datasheets for Datasets.” Communications of the ACM 64 (12): 86–92. https://doi.org/10.1145/3458723.

Lazer, David M. J., Alex Pentland, Duncan J. Watts, et al. 2020. “Computational Social Science: Obstacles and Opportunities.” Science 369 (6507): 1060–62. https://doi.org/10.1126/science.aaz8170.

Lazer, David, Ryan Kennedy, Gary King, and Alessandro Vespignani. 2014. “The Parable of Google Flu: Traps in Big Data Analysis.” Science 343 (6176): 1203–5. https://doi.org/10.1126/science.1248506.

Rogers, Richard. 2019. Doing Digital Methods. SAGE Publications.

Salganik, Matthew J. 2018. Bit by Bit: Social Research in the Digital Age. Princeton University Press. https://www.bitbybitbook.com/.