5 Web Architecture and Protocols

Learning Objectives

Describe the client-server model and trace the sequence of operations from URL to rendered page
Use browser developer tools to examine page structure, HTTP requests, and response headers
Explain the roles of TCP/IP, DNS, and HTTP in web communication
Construct HTTP requests with custom headers using requests and parse URL components with urllib.parse
Make an API call and convert the response to a time series visualization

Companion Notebook

Run this chapter’s code as you read: download the companion notebook (see Appendix A for all of them).

5.1 What Happens When You Load a Web Page?

When you type a URL into your browser and press Enter, a remarkable sequence of operations unfolds in milliseconds. Your browser resolves the domain name to an IP address, opens a connection to a remote server, sends a request formatted in a specific protocol, receives a response, and renders the result on your screen. Understanding this sequence transforms web data collection from magic into engineering.

This chapter walks through each layer of the stack: the client-server model that structures the conversation, TCP/IP that transports the data, DNS that resolves names to addresses, HTTP that structures requests and responses, and URLs that identify what you are asking for. You will use browser developer tools to observe this machinery in action, then use Python libraries to replicate what your browser does programmatically.

5.2 The Client-Server Model

The web follows a client-server architecture. A client (your browser, your Python script) sends a request to a server (a computer hosting a website or API). The server processes the request and sends back a response. This request-response cycle is the fundamental unit of web communication, and every scraping and API operation you perform in this book is an instance of it.

Your browser is a sophisticated client that handles many things automatically — resolving domain names, managing cookies, rendering HTML and CSS, executing JavaScript. When you use requests.get() in Python, you are acting as a much simpler client that only handles the HTTP layer. Understanding what your browser does behind the scenes will help you diagnose problems when your Python client gets different results than your browser.

5.3 Browser Developer Tools

Before writing code to scrape a page, you should always inspect it manually using your browser’s developer tools. Every modern browser includes these tools:

Windows/Linux: Ctrl + Shift + I or F12
Mac: ⌘ + ⌥ + I

5.3.1 The Inspector Tab

The Inspector shows the HTML structure of the page as a tree. You can hover over elements to highlight them on the page, expand nested tags to explore the structure, and identify the CSS selectors or element attributes you will need for scraping. Start with a simple page like a Wikipedia article — most commercial websites are full of tracking scripts and dynamic content that obscure the actual structure.

5.3.2 The Network Tab

The Network tab reveals all the HTTP requests your browser makes to load a page. When you reload a page with the Network tab open, you will see dozens or even hundreds of requests — not just for the HTML, but for CSS stylesheets, JavaScript files, images, fonts, and tracking pixels. Each request shows:

The URL being requested
The HTTP method (GET, POST, etc.)
The status code of the response (200 OK, 404 Not Found, etc.)
The response headers, including Content-Type, Set-Cookie, and User-Agent

Click on any individual request to inspect its headers. Pay particular attention to the User-Agent header (how your browser identifies itself) and any Cookie headers (session information that the server uses to recognize you).

Missing Manual Reference

For a broader introduction to how HTTP and web APIs work, see Missing Manual Chapter 24: HTTP and Web APIs.

5.4 TCP/IP: The Transport Layer

Every computer on a public network has an IP address — a numerical identifier that functions like a postal address. IPv4 addresses are formatted as four numbers separated by dots (e.g., 151.101.130.133), allowing for about 4.3 billion unique addresses. IPv6 uses a longer hexadecimal format to support a vastly larger address space.

You can find your own computer’s IP address programmatically:

import socket

# Your computer's hostname on the local network
hostname = socket.gethostname()
print(f"Hostname: {hostname}")

# Your local IP address
local_ip = socket.gethostbyname(hostname)
print(f"Local IP: {local_ip}")

Be aware that on many Linux and macOS systems this lookup returns a loopback address like 127.0.0.1 or 127.0.1.1 — an address that simply means “this machine” — rather than your network-assigned IP, so the public-IP query below is the more reliable way to see how the internet identifies you.

To see your public IP address (as seen by servers on the internet), you can query an external service:

import requests

response = requests.get("https://api.ipify.org?format=json")
print(response.json())
# {'ip': '128.138.xxx.xxx'} — your public IP

5.4.1 Traceroute

When you send a request to a remote server, the data does not travel in a straight line. It hops through a series of intermediate routers. You can trace this path using the scapy library:

from scapy.all import traceroute

# Trace the path to Wikipedia
# Note: scapy requires root/administrator access on some systems
result, unans = traceroute("en.wikipedia.org", maxttl=20)

Each hop represents a router or server along the path. You can use an IP geolocation service to map these hops geographically:

import requests

def geolocate_ip(ip):
    """Look up the geographic location of an IP address."""
    response = requests.get(f"http://ip-api.com/json/{ip}")
    return response.json()

# Note: ip-api.com limits you to 45 requests per minute
location = geolocate_ip("151.101.130.133")
print(f"{location['city']}, {location['country']}")

scapy requires root/administrator privileges to send and receive raw network packets. On macOS or Linux, you may need to run sudo jupyter notebook from the terminal. On Windows, install the Npcap packet-capture driver and launch Anaconda Prompt as Administrator. If you cannot get packet capture working on your machine, read along instead — nothing later in this book depends on it.

5.5 DNS: The Address Book

The Domain Name System translates human-readable domain names (like en.wikipedia.org) into IP addresses that computers can route to. You can perform DNS lookups programmatically with the dnspython library:

import dns.resolver

# Look up the IP address for Wikipedia
answers = dns.resolver.resolve("en.wikipedia.org", "A")
for answer in answers:
    print(f"IP address: {answer}")

Domain names have a hierarchical structure: subdomain.domain.top-level-domain. Each top-level domain (.com, .org, .edu) is operated by a registry organization — Verisign runs .com, for example — under the coordination of ICANN, the nonprofit that oversees the internet’s naming system. The domain (wikipedia) is registered by the organization. Subdomains (en, es, m) are controlled by the domain owner and do not require separate registration.

5.5.1 The Structure of Domain Names

Understanding this hierarchy helps you navigate the web programmatically. Consider two examples:

en.wikipedia.org — The TLD is .org (a generic TLD for organizations). The domain is wikipedia (registered by the Wikimedia Foundation). The subdomain en indicates the English-language edition. Other subdomains like es, de, or m (mobile) are all controlled by Wikipedia and do not require separate registration. The pageviews API you will use later in this chapter lives at wikimedia.org — not a subdomain of wikipedia.org but a separate registered domain that the Wikimedia Foundation also operates.
api.census.gov — The TLD is .gov (restricted to U.S. government entities). The domain is census. The subdomain api indicates this is the programmatic access point rather than the public-facing website at www.census.gov.

When you construct URLs for scraping or API calls, knowing which part of the domain identifies the service (the domain) versus which part routes you to a specific feature (the subdomain and path) helps you anticipate URL patterns. Government APIs often live at api.agency.gov, data portals at data.agency.gov, and documentation at developers.agency.gov. Recognizing these conventions saves you time when exploring new data sources.

5.6 HTTP: The Application Layer

HTTP (Hypertext Transfer Protocol) is the language that clients and servers use to communicate. A client sends a request with a method (GET, POST, PUT, DELETE), headers (metadata), and optionally a body. The server sends a response with a status code, headers, and a body.

The status codes you will encounter most often:

200 OK: The request succeeded and the response body contains the data
301/302 Redirect: The resource has moved; follow the Location header
403 Forbidden: The server understood your request but refused to fulfill it (often a User-Agent issue)
404 Not Found: The resource does not exist at this URL
429 Too Many Requests: You are being rate-limited; slow down
500 Internal Server Error: Something went wrong on the server side

5.6.1 Diagnosing HTTP Errors

When a request fails, the status code tells you what went wrong, but diagnosing why requires looking at the full response. A diagnostic function can help:

import requests

def diagnose_request(url, headers=None):
    """Print diagnostic information about an HTTP request."""
    try:
        response = requests.get(url, headers=headers, timeout=10)
        print(f"URL: {url}")
        print(f"Status: {response.status_code}")
        print(f"Content-Type: {response.headers.get('Content-Type', 'not specified')}")
        print(f"Server: {response.headers.get('Server', 'not specified')}")
        print(f"Response size: {len(response.content):,} bytes")
        if response.history:
            print(f"Redirected from: {response.history[0].url}")
        return response
    except requests.exceptions.ConnectionError:
        print(f"Connection failed: {url}")
    except requests.exceptions.Timeout:
        print(f"Request timed out: {url}")

# Test on a few URLs
diagnose_request("https://en.wikipedia.org/wiki/Python_(programming_language)")
diagnose_request("https://en.wikipedia.org/wiki/Nonexistent_Page_12345")

The most common errors you will encounter in practice are 403 (Forbidden) and 429 (Too Many Requests). A 403 often means the server detected your request as automated — either because you are missing a User-Agent header or because the site blocks scraping entirely. A 429 means you are hitting the server too frequently. For 429 errors, a retry-with-backoff strategy helps:

import time

def get_with_backoff(url, headers=None, max_retries=3):
    """Retry a request with exponential backoff on 429 errors."""
    for attempt in range(max_retries):
        response = requests.get(url, headers=headers)
        if response.status_code == 429:
            wait_time = 2 ** attempt  # 1s, 2s, 4s
            print(f"Rate limited. Waiting {wait_time}s before retry...")
            time.sleep(wait_time)
        else:
            return response
    return response  # Return the last response even if still 429

Exponential backoff doubles the wait time with each retry, giving the server progressively more breathing room. Many APIs include a Retry-After header in their 429 responses that tells you exactly how long to wait — check for it with response.headers.get("Retry-After") before falling back to your own backoff schedule.

Understanding status codes in context makes debugging much faster. A 403 from the Wikimedia API almost always means you forgot the User-Agent header — the fix is a single line of code. A 403 from a news site might mean the site blocks all automated access, and no header change will help. A 404 could mean the page was deleted, the URL has a typo, or the site restructured its paths. A 500 is the server’s problem, not yours — retry after a delay. Building this diagnostic intuition takes practice, but the pattern is always the same: check the status code, inspect the response headers, read the response body for error messages, and adjust your request accordingly. This systematic approach to debugging carries through every chapter that follows.

5.6.2 Making API Requests with Custom Headers

Let us retrieve data from the Wikimedia pageviews API. This API requires a meaningful User-Agent header — if you send the default requests User-Agent, you will get a 403 error:

import requests

article = "University_of_Colorado_Boulder"
api_url = (
    f"https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article"
    f"/en.wikipedia/all-access/all-agents/{article}/daily/20240101/20240131"
)

# This will likely fail with a 403
response = requests.get(api_url)
print(response.status_code)  # 403

# Add a custom User-Agent header
headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
response = requests.get(api_url, headers=headers)
print(response.status_code)  # 200

# Parse the JSON response
data = response.json()

Now convert the response to a DataFrame and plot it:

import pandas as pd
import matplotlib.pyplot as plt

# The pageview data is nested inside data['items']
df = pd.DataFrame(data["items"])
df["timestamp"] = pd.to_datetime(df["timestamp"], format="%Y%m%d00")

plt.figure(figsize=(10, 4))
plt.plot(df["timestamp"], df["views"])
plt.title(f"Daily Pageviews: {article}")
plt.xlabel("Date")
plt.ylabel("Views")
plt.tight_layout()
plt.show()

5.6.3 Connections, Sessions, and Retries

Every bare requests.get() call does more work than it appears to: it opens a fresh TCP connection to the server, performs the TLS handshake, sends the request, and tears the connection back down. For a single request, that overhead is negligible. But the chapters ahead will have you making dozens or hundreds of requests to the same host — one per Wikipedia article, one per bill, one per archived page — and paying the connection setup cost every time is wasteful for both you and the server.

A requests.Session object fixes this. A Session keeps the underlying connection open and reuses it for subsequent requests to the same host, and it remembers headers, so you set your User-Agent once instead of passing it to every call. A Session can also carry a retry policy: rather than hand-rolling backoff logic like the get_with_backoff() function you wrote earlier, you can mount a Retry configuration that automatically retries rate limits and transient server errors with exponentially increasing delays — and it honors any Retry-After header the server sends:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Create a session with persistent headers
session = requests.Session()
session.headers.update(
    {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"}
)

# Retry up to 3 times on rate limits and server errors,
# waiting 1s, 2s, then 4s between attempts
retries = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
)
session.mount("https://", HTTPAdapter(max_retries=retries))

# Use the session exactly like the requests module itself
articles = ["Python_(programming_language)", "Web_scraping", "Data_science"]
for article in articles:
    url = f"https://en.wikipedia.org/wiki/{article}"
    response = session.get(url)  # Connection reused; headers already set
    print(article, response.status_code, len(response.text))
# Python_(programming_language) 200 812394
# Web_scraping 200 145207
# Data_science 200 291846

The setup costs six lines, and everything after that behaves exactly like the requests.get() calls you already know — session.get() returns the same Response object with the same .json(), .text, and .status_code.

A for loop around requests.get() is your cue to create a Session first. The scraping and API chapters ahead (Chapter 6, Chapter 10, Chapter 11) all involve exactly this kind of loop over many URLs at the same host, and this pattern carries over unchanged.

5.7 URLs: Anatomy and Parsing

URLs (Uniform Resource Locators) have a well-defined structure:

scheme://userinfo@host:port/path?query#fragment

Never write regular expressions to parse URLs. Use the urllib.parse module instead:

from urllib.parse import urlparse, parse_qs

url = "https://api.census.gov/data/2022/acs/acs5?get=NAME,B01001_001E&for=place:07850&in=state:08"

parsed = urlparse(url)
print(f"Scheme: {parsed.scheme}")      # https
print(f"Host: {parsed.netloc}")        # api.census.gov
print(f"Path: {parsed.path}")          # /data/2022/acs/acs5
print(f"Query: {parsed.query}")        # get=NAME,...

# Parse query parameters into a dictionary
params = parse_qs(parsed.query)
print(params)
# {'get': ['NAME,B01001_001E'], 'for': ['place:07850'], 'in': ['state:08']}

Understanding URL structure is essential for constructing API requests. In Chapter 10 through Chapter 13, you will build URLs programmatically by assembling scheme, host, path, and query parameters.

5.7.1 Building URLs Programmatically

Web APIs expect the values you supply in one of two places, and you need to recognize which pattern you are dealing with before you can build the URL correctly.

Path-segment APIs embed each value at a fixed position in the URL’s path. The Wikimedia pageviews endpoint you used earlier works this way: project, access method, agent, article, granularity, and date range are all path segments in a required order. Build these URLs with an f-string over the components — but URL-encode any component that might contain special characters using urllib.parse.quote, because a stray slash or space inside a path segment produces a URL the server cannot route:

from urllib.parse import quote

# Each value occupies a fixed position in the path
project = "en.wikipedia"
access = "all-access"
agent = "all-agents"
article = "University of Colorado Boulder"  # Note the spaces

# Wikipedia titles replace spaces with underscores; quote() percent-encodes
# anything else that would break a path segment (slashes, question marks)
encoded_article = quote(article.replace(" ", "_"), safe="")
print(encoded_article)
# University_of_Colorado_Boulder

url = (
    "https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article"
    f"/{project}/{access}/{agent}/{encoded_article}/daily/20240101/20240131"
)
print(url)
# https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/all-agents/University_of_Colorado_Boulder/daily/20240101/20240131

The encoding matters more than this example suggests: a band like “AC/DC” has a slash in its article title, and quote("AC/DC", safe="") correctly produces AC%2FDC. Paste the raw title into your f-string instead, and the API would interpret the slash as a path separator and return a 404.

Query-parameter APIs instead accept key=value pairs after a ?, where order does not matter and many parameters are optional. The MediaWiki Action API — Wikipedia’s general-purpose API, which you will explore in Chapter 10 — works this way. The urlencode function converts a dictionary into a properly escaped query string:

from urllib.parse import urlencode

base_url = "https://en.wikipedia.org/w/api.php"
params = {
    "action": "query",
    "format": "json",
    "titles": "University of Colorado Boulder",
    "prop": "info",
}

query_string = urlencode(params)
print(query_string)
# action=query&format=json&titles=University+of+Colorado+Boulder&prop=info

For query-parameter APIs, the requests library can do this encoding for you: pass the dictionary as the params argument. This is the preferred approach for API calls:

response = requests.get(base_url, params=params, headers=headers)
print(response.url)  # The full URL that requests actually built
# https://en.wikipedia.org/w/api.php?action=query&format=json&titles=University+of+Colorado+Boulder&prop=info
print(response.status_code)  # 200

How do you know which pattern an API expects? Read its documentation — but the URL itself usually tells you. Values between slashes are path segments: build the URL with an f-string and quote() each component. Values after a ? are query parameters: pass a dictionary via params= and let requests handle the escaping. REST-style endpoints that return one specific resource (like pageviews for one article) tend to use path segments; APIs with many optional settings (like the Action API’s dozens of parameters) tend to use query strings. Mixing them up is a classic source of confusing 404s — query-encoding values that an API expects in its path sends the server a URL it cannot route. You will appreciate this distinction especially in Chapter 10 and Chapter 11, where article titles and variable names frequently contain characters that require encoding.

5.8 Exercises

Browser developer tools. Open a Wikipedia article and a commercial news site side by side, each with the Network tab open. Reload both pages and compare: how many requests does each make? What types of resources (HTML, CSS, JS, images, fonts, tracking) does each load? What does this tell you about modern web architecture?
DNS exploration. Use dns.resolver to look up the IP addresses for five related domains (e.g., google.com, youtube.com, gmail.com, drive.google.com, cloud.google.com). Are any of them hosted at the same IP address? What does this tell you about Google’s infrastructure?
Wikimedia pageviews. Using the API pattern from this chapter, retrieve and plot daily pageview data for three related Wikipedia articles over the same time period. Identify any spikes and hypothesize about what real-world events caused them.
Network tab analysis. Pick a website you visit frequently. Reload it with the browser’s Network tab open and recording. Count and categorize all requests into types: HTML documents, CSS stylesheets, JavaScript files, images, fonts, and tracking/analytics scripts. What fraction of requests serve actual content vs. tracking and advertising? Write a brief analysis of what this reveals about the site’s business model.
URL diagnostic function. Write a function diagnose_url(url) that prints the status code, Content-Type, Server header, response size in bytes, and whether the URL redirected (and if so, to where). Test it on at least five URLs: a working page, a 404, a redirecting URL, an API endpoint, and a site that blocks automated access. Summarize the patterns you observe.
Graduate extension (INFO 5617). Choose a website you might plausibly study in your own research and profile its delivery infrastructure. Use the browser’s Network tab to record the time to first byte, the full redirect chain, and evidence of CDN involvement (inspect the Server, Via, and similar response headers), then run traceroute (or scapy’s traceroute) to the same host and geolocate a few of the hops. Write a 500-word memo connecting these protocol-level observations to research-infrastructure concerns: where does “the” site actually live, how stable are measurements of it likely to be across time and vantage points, and what would an automated collection pipeline need to handle — redirects, rate limits, geographically varying responses — to gather data from it reliably?

5.10 Common Issues to Debug

scapy import errors: Requires pcap drivers. On Windows, install Npcap. On macOS, the built-in libpcap usually works.
403 from Wikimedia API: You forgot the custom User-Agent header.
DNS results varying by location: Normal — content delivery networks return different IPs based on your geographic location.
URL encoding issues: Use urllib.parse.quote() for article titles with spaces or special characters.

5.11 Key Takeaways

The web is built on a layered stack of protocols: TCP/IP transports data, DNS resolves names to addresses, HTTP structures the conversation, and URLs identify the target. Understanding this stack transforms debugging from guesswork into systematic diagnosis. When a request fails, you can ask: can I reach the server at all? Am I resolving the right address? What status code did I get? Are my headers correct? Is my URL well-formed? This systematic approach carries through every chapter that follows.

5.12 Further Reading

Mozilla Developer Network — How the Web Works: https://developer.mozilla.org/en-US/docs/Learn/Getting_started_with_the_web/How_the_Web_works
Mozilla Developer Network — HTTP Overview: https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview
scapy documentation: https://scapy.readthedocs.io/
dnspython documentation: https://dnspython.readthedocs.io/

# Web Architecture and Protocols {#sec-protocols} ::: {.callout-tip} ## Learning Objectives - Describe the client-server model and trace the sequence of operations from URL to rendered page - Use browser developer tools to examine page structure, HTTP requests, and response headers - Explain the roles of TCP/IP, DNS, and HTTP in web communication - Construct HTTP requests with custom headers using `requests` and parse URL components with `urllib.parse` - Make an API call and convert the response to a time series visualization ::: ::: {.callout-tip} ## Companion Notebook Run this chapter's code as you read: download the [companion notebook](notebooks/ch-05-protocols.ipynb) (see @sec-appendix-notebooks for all of them). ::: ## What Happens When You Load a Web Page? When you type a URL into your browser and press Enter, a remarkable sequence of operations unfolds in milliseconds. Your browser resolves the domain name to an IP address, opens a connection to a remote server, sends a request formatted in a specific protocol, receives a response, and renders the result on your screen. Understanding this sequence transforms web data collection from magic into engineering. This chapter walks through each layer of the stack: the **client-server model** that structures the conversation, **TCP/IP** that transports the data, **DNS** that resolves names to addresses, **HTTP** that structures requests and responses, and **URLs** that identify what you are asking for. You will use browser developer tools to observe this machinery in action, then use Python libraries to replicate what your browser does programmatically. ## The Client-Server Model The web follows a client-server architecture. A **client** (your browser, your Python script) sends a **request** to a **server** (a computer hosting a website or API). The server processes the request and sends back a **response**. This request-response cycle is the fundamental unit of web communication, and every scraping and API operation you perform in this book is an instance of it. Your browser is a sophisticated client that handles many things automatically — resolving domain names, managing cookies, rendering HTML and CSS, executing JavaScript. When you use `requests.get()` in Python, you are acting as a much simpler client that only handles the HTTP layer. Understanding what your browser does behind the scenes will help you diagnose problems when your Python client gets different results than your browser. ## Browser Developer Tools Before writing code to scrape a page, you should always inspect it manually using your browser's developer tools. Every modern browser includes these tools: - **Windows/Linux**: Ctrl + Shift + I or F12 - **Mac**: ⌘ + ⌥ + I ### The Inspector Tab The Inspector shows the HTML structure of the page as a tree. You can hover over elements to highlight them on the page, expand nested tags to explore the structure, and identify the CSS selectors or element attributes you will need for scraping. Start with a simple page like a Wikipedia article — most commercial websites are full of tracking scripts and dynamic content that obscure the actual structure. ### The Network Tab The Network tab reveals all the HTTP requests your browser makes to load a page. When you reload a page with the Network tab open, you will see dozens or even hundreds of requests — not just for the HTML, but for CSS stylesheets, JavaScript files, images, fonts, and tracking pixels. Each request shows: - The URL being requested - The HTTP method (GET, POST, etc.) - The status code of the response (200 OK, 404 Not Found, etc.) - The response headers, including `Content-Type`, `Set-Cookie`, and `User-Agent` Click on any individual request to inspect its headers. Pay particular attention to the `User-Agent` header (how your browser identifies itself) and any `Cookie` headers (session information that the server uses to recognize you). ::: {.callout-tip} ## Missing Manual Reference For a broader introduction to how HTTP and web APIs work, see *Missing Manual* Chapter 24: HTTP and Web APIs. ::: ## TCP/IP: The Transport Layer Every computer on a public network has an IP address — a numerical identifier that functions like a postal address. IPv4 addresses are formatted as four numbers separated by dots (e.g., `151.101.130.133`), allowing for about 4.3 billion unique addresses. IPv6 uses a longer hexadecimal format to support a vastly larger address space. You can find your own computer's IP address programmatically: ```{python} import socket # Your computer's hostname on the local network hostname = socket.gethostname() print(f"Hostname: {hostname}") # Your local IP address local_ip = socket.gethostbyname(hostname) print(f"Local IP: {local_ip}") ``` Be aware that on many Linux and macOS systems this lookup returns a loopback address like `127.0.0.1` or `127.0.1.1` — an address that simply means "this machine" — rather than your network-assigned IP, so the public-IP query below is the more reliable way to see how the internet identifies you. To see your *public* IP address (as seen by servers on the internet), you can query an external service: ```{python} import requests response = requests.get("https://api.ipify.org?format=json") print(response.json()) # {'ip': '128.138.xxx.xxx'} — your public IP ``` ### Traceroute When you send a request to a remote server, the data does not travel in a straight line. It hops through a series of intermediate routers. You can trace this path using the `scapy` library: ```{python} from scapy.all import traceroute # Trace the path to Wikipedia # Note: scapy requires root/administrator access on some systems result, unans = traceroute("en.wikipedia.org", maxttl=20) ``` Each hop represents a router or server along the path. You can use an IP geolocation service to map these hops geographically: ```{python} import requests def geolocate_ip(ip): """Look up the geographic location of an IP address.""" response = requests.get(f"http://ip-api.com/json/{ip}") return response.json() # Note: ip-api.com limits you to 45 requests per minute location = geolocate_ip("151.101.130.133") print(f"{location['city']}, {location['country']}") ``` ::: {.callout-important} `scapy` requires root/administrator privileges to send and receive raw network packets. On macOS or Linux, you may need to run `sudo jupyter notebook` from the terminal. On Windows, install the [Npcap](https://npcap.com/) packet-capture driver and launch Anaconda Prompt as Administrator. If you cannot get packet capture working on your machine, read along instead — nothing later in this book depends on it. ::: ## DNS: The Address Book The Domain Name System translates human-readable domain names (like `en.wikipedia.org`) into IP addresses that computers can route to. You can perform DNS lookups programmatically with the `dnspython` library: ```{python} import dns.resolver # Look up the IP address for Wikipedia answers = dns.resolver.resolve("en.wikipedia.org", "A") for answer in answers: print(f"IP address: {answer}") ``` Domain names have a hierarchical structure: `subdomain.domain.top-level-domain`. Each top-level domain (`.com`, `.org`, `.edu`) is operated by a registry organization — Verisign runs `.com`, for example — under the coordination of ICANN, the nonprofit that oversees the internet's naming system. The domain (`wikipedia`) is registered by the organization. Subdomains (`en`, `es`, `m`) are controlled by the domain owner and do not require separate registration. ### The Structure of Domain Names Understanding this hierarchy helps you navigate the web programmatically. Consider two examples: - `en.wikipedia.org` — The TLD is `.org` (a generic TLD for organizations). The domain is `wikipedia` (registered by the Wikimedia Foundation). The subdomain `en` indicates the English-language edition. Other subdomains like `es`, `de`, or `m` (mobile) are all controlled by Wikipedia and do not require separate registration. The pageviews API you will use later in this chapter lives at `wikimedia.org` — not a subdomain of `wikipedia.org` but a separate registered domain that the Wikimedia Foundation also operates. - `api.census.gov` — The TLD is `.gov` (restricted to U.S. government entities). The domain is `census`. The subdomain `api` indicates this is the programmatic access point rather than the public-facing website at `www.census.gov`. When you construct URLs for scraping or API calls, knowing which part of the domain identifies the service (the domain) versus which part routes you to a specific feature (the subdomain and path) helps you anticipate URL patterns. Government APIs often live at `api.agency.gov`, data portals at `data.agency.gov`, and documentation at `developers.agency.gov`. Recognizing these conventions saves you time when exploring new data sources. ## HTTP: The Application Layer HTTP (Hypertext Transfer Protocol) is the language that clients and servers use to communicate. A client sends a **request** with a method (GET, POST, PUT, DELETE), headers (metadata), and optionally a body. The server sends a **response** with a status code, headers, and a body. The status codes you will encounter most often: - **200 OK**: The request succeeded and the response body contains the data - **301/302 Redirect**: The resource has moved; follow the `Location` header - **403 Forbidden**: The server understood your request but refused to fulfill it (often a User-Agent issue) - **404 Not Found**: The resource does not exist at this URL - **429 Too Many Requests**: You are being rate-limited; slow down - **500 Internal Server Error**: Something went wrong on the server side ### Diagnosing HTTP Errors When a request fails, the status code tells you *what* went wrong, but diagnosing *why* requires looking at the full response. A diagnostic function can help: ```{python} import requests def diagnose_request(url, headers=None): """Print diagnostic information about an HTTP request.""" try: response = requests.get(url, headers=headers, timeout=10) print(f"URL: {url}") print(f"Status: {response.status_code}") print(f"Content-Type: {response.headers.get('Content-Type', 'not specified')}") print(f"Server: {response.headers.get('Server', 'not specified')}") print(f"Response size: {len(response.content):,} bytes") if response.history: print(f"Redirected from: {response.history[0].url}") return response except requests.exceptions.ConnectionError: print(f"Connection failed: {url}") except requests.exceptions.Timeout: print(f"Request timed out: {url}") # Test on a few URLs diagnose_request("https://en.wikipedia.org/wiki/Python_(programming_language)") diagnose_request("https://en.wikipedia.org/wiki/Nonexistent_Page_12345") ``` The most common errors you will encounter in practice are 403 (Forbidden) and 429 (Too Many Requests). A 403 often means the server detected your request as automated — either because you are missing a `User-Agent` header or because the site blocks scraping entirely. A 429 means you are hitting the server too frequently. For 429 errors, a retry-with-backoff strategy helps: ```{python} import time def get_with_backoff(url, headers=None, max_retries=3): """Retry a request with exponential backoff on 429 errors.""" for attempt in range(max_retries): response = requests.get(url, headers=headers) if response.status_code == 429: wait_time = 2 ** attempt # 1s, 2s, 4s print(f"Rate limited. Waiting {wait_time}s before retry...") time.sleep(wait_time) else: return response return response # Return the last response even if still 429 ``` Exponential backoff doubles the wait time with each retry, giving the server progressively more breathing room. Many APIs include a `Retry-After` header in their 429 responses that tells you exactly how long to wait — check for it with `response.headers.get("Retry-After")` before falling back to your own backoff schedule. Understanding status codes in context makes debugging much faster. A 403 from the Wikimedia API almost always means you forgot the `User-Agent` header — the fix is a single line of code. A 403 from a news site might mean the site blocks all automated access, and no header change will help. A 404 could mean the page was deleted, the URL has a typo, or the site restructured its paths. A 500 is the server's problem, not yours — retry after a delay. Building this diagnostic intuition takes practice, but the pattern is always the same: check the status code, inspect the response headers, read the response body for error messages, and adjust your request accordingly. This systematic approach to debugging carries through every chapter that follows. ### Making API Requests with Custom Headers Let us retrieve data from the Wikimedia pageviews API. This API requires a meaningful `User-Agent` header — if you send the default `requests` User-Agent, you will get a 403 error: ```{python} import requests article = "University_of_Colorado_Boulder" api_url = ( f"https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article" f"/en.wikipedia/all-access/all-agents/{article}/daily/20240101/20240131" ) # This will likely fail with a 403 response = requests.get(api_url) print(response.status_code) # 403 # Add a custom User-Agent header headers = {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"} response = requests.get(api_url, headers=headers) print(response.status_code) # 200 # Parse the JSON response data = response.json() ``` Now convert the response to a DataFrame and plot it: ```{python} import pandas as pd import matplotlib.pyplot as plt # The pageview data is nested inside data['items'] df = pd.DataFrame(data["items"]) df["timestamp"] = pd.to_datetime(df["timestamp"], format="%Y%m%d00") plt.figure(figsize=(10, 4)) plt.plot(df["timestamp"], df["views"]) plt.title(f"Daily Pageviews: {article}") plt.xlabel("Date") plt.ylabel("Views") plt.tight_layout() plt.show() ``` ### Connections, Sessions, and Retries Every bare `requests.get()` call does more work than it appears to: it opens a fresh TCP connection to the server, performs the TLS handshake, sends the request, and tears the connection back down. For a single request, that overhead is negligible. But the chapters ahead will have you making dozens or hundreds of requests to the same host — one per Wikipedia article, one per bill, one per archived page — and paying the connection setup cost every time is wasteful for both you and the server. A `requests.Session` object fixes this. A Session keeps the underlying connection open and reuses it for subsequent requests to the same host, and it remembers headers, so you set your `User-Agent` once instead of passing it to every call. A Session can also carry a retry policy: rather than hand-rolling backoff logic like the `get_with_backoff()` function you wrote earlier, you can mount a `Retry` configuration that automatically retries rate limits and transient server errors with exponentially increasing delays — and it honors any `Retry-After` header the server sends: ```{python} import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry # Create a session with persistent headers session = requests.Session() session.headers.update( {"User-Agent": "WebDataScience/1.0 (your-email@colorado.edu)"} ) # Retry up to 3 times on rate limits and server errors, # waiting 1s, 2s, then 4s between attempts retries = Retry( total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504], ) session.mount("https://", HTTPAdapter(max_retries=retries)) # Use the session exactly like the requests module itself articles = ["Python_(programming_language)", "Web_scraping", "Data_science"] for article in articles: url = f"https://en.wikipedia.org/wiki/{article}" response = session.get(url) # Connection reused; headers already set print(article, response.status_code, len(response.text)) # Python_(programming_language) 200 812394 # Web_scraping 200 145207 # Data_science 200 291846 ``` The setup costs six lines, and everything after that behaves exactly like the `requests.get()` calls you already know — `session.get()` returns the same `Response` object with the same `.json()`, `.text`, and `.status_code`. ::: {.callout-tip} A `for` loop around `requests.get()` is your cue to create a `Session` first. The scraping and API chapters ahead (@sec-static-pages, @sec-wikipedia, @sec-government) all involve exactly this kind of loop over many URLs at the same host, and this pattern carries over unchanged. ::: ## URLs: Anatomy and Parsing URLs (Uniform Resource Locators) have a well-defined structure: ``` scheme://userinfo@host:port/path?query#fragment ``` Never write regular expressions to parse URLs. Use the `urllib.parse` module instead: ```{python} from urllib.parse import urlparse, parse_qs url = "https://api.census.gov/data/2022/acs/acs5?get=NAME,B01001_001E&for=place:07850&in=state:08" parsed = urlparse(url) print(f"Scheme: {parsed.scheme}") # https print(f"Host: {parsed.netloc}") # api.census.gov print(f"Path: {parsed.path}") # /data/2022/acs/acs5 print(f"Query: {parsed.query}") # get=NAME,... # Parse query parameters into a dictionary params = parse_qs(parsed.query) print(params) # {'get': ['NAME,B01001_001E'], 'for': ['place:07850'], 'in': ['state:08']} ``` Understanding URL structure is essential for constructing API requests. In @sec-wikipedia through @sec-ai-apis, you will build URLs programmatically by assembling scheme, host, path, and query parameters. ### Building URLs Programmatically Web APIs expect the values you supply in one of two places, and you need to recognize which pattern you are dealing with before you can build the URL correctly. **Path-segment APIs** embed each value at a fixed position in the URL's path. The Wikimedia pageviews endpoint you used earlier works this way: project, access method, agent, article, granularity, and date range are all path segments in a required order. Build these URLs with an f-string over the components — but URL-encode any component that might contain special characters using `urllib.parse.quote`, because a stray slash or space inside a path segment produces a URL the server cannot route: ```{python} from urllib.parse import quote # Each value occupies a fixed position in the path project = "en.wikipedia" access = "all-access" agent = "all-agents" article = "University of Colorado Boulder" # Note the spaces # Wikipedia titles replace spaces with underscores; quote() percent-encodes # anything else that would break a path segment (slashes, question marks) encoded_article = quote(article.replace(" ", "_"), safe="") print(encoded_article) # University_of_Colorado_Boulder url = ( "https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article" f"/{project}/{access}/{agent}/{encoded_article}/daily/20240101/20240131" ) print(url) # https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/all-agents/University_of_Colorado_Boulder/daily/20240101/20240131 ``` The encoding matters more than this example suggests: a band like "AC/DC" has a slash in its article title, and `quote("AC/DC", safe="")` correctly produces `AC%2FDC`. Paste the raw title into your f-string instead, and the API would interpret the slash as a path separator and return a 404. **Query-parameter APIs** instead accept `key=value` pairs after a `?`, where order does not matter and many parameters are optional. The MediaWiki Action API — Wikipedia's general-purpose API, which you will explore in @sec-wikipedia — works this way. The `urlencode` function converts a dictionary into a properly escaped query string: ```{python} from urllib.parse import urlencode base_url = "https://en.wikipedia.org/w/api.php" params = { "action": "query", "format": "json", "titles": "University of Colorado Boulder", "prop": "info", } query_string = urlencode(params) print(query_string) # action=query&format=json&titles=University+of+Colorado+Boulder&prop=info ``` For query-parameter APIs, the `requests` library can do this encoding for you: pass the dictionary as the `params` argument. This is the preferred approach for API calls: ```{python} response = requests.get(base_url, params=params, headers=headers) print(response.url) # The full URL that requests actually built # https://en.wikipedia.org/w/api.php?action=query&format=json&titles=University+of+Colorado+Boulder&prop=info print(response.status_code) # 200 ``` How do you know which pattern an API expects? Read its documentation — but the URL itself usually tells you. Values between slashes are path segments: build the URL with an f-string and `quote()` each component. Values after a `?` are query parameters: pass a dictionary via `params=` and let `requests` handle the escaping. REST-style endpoints that return one specific resource (like pageviews for one article) tend to use path segments; APIs with many optional settings (like the Action API's dozens of parameters) tend to use query strings. Mixing them up is a classic source of confusing 404s — query-encoding values that an API expects in its path sends the server a URL it cannot route. You will appreciate this distinction especially in @sec-wikipedia and @sec-government, where article titles and variable names frequently contain characters that require encoding. ## Exercises 1. **Browser developer tools.** Open a Wikipedia article and a commercial news site side by side, each with the Network tab open. Reload both pages and compare: how many requests does each make? What types of resources (HTML, CSS, JS, images, fonts, tracking) does each load? What does this tell you about modern web architecture? 2. **DNS exploration.** Use `dns.resolver` to look up the IP addresses for five related domains (e.g., `google.com`, `youtube.com`, `gmail.com`, `drive.google.com`, `cloud.google.com`). Are any of them hosted at the same IP address? What does this tell you about Google's infrastructure? 3. **Wikimedia pageviews.** Using the API pattern from this chapter, retrieve and plot daily pageview data for three related Wikipedia articles over the same time period. Identify any spikes and hypothesize about what real-world events caused them. 4. **Network tab analysis.** Pick a website you visit frequently. Reload it with the browser's Network tab open and recording. Count and categorize all requests into types: HTML documents, CSS stylesheets, JavaScript files, images, fonts, and tracking/analytics scripts. What fraction of requests serve actual content vs. tracking and advertising? Write a brief analysis of what this reveals about the site's business model. 5. **URL diagnostic function.** Write a function `diagnose_url(url)` that prints the status code, Content-Type, Server header, response size in bytes, and whether the URL redirected (and if so, to where). Test it on at least five URLs: a working page, a 404, a redirecting URL, an API endpoint, and a site that blocks automated access. Summarize the patterns you observe. 6. **Graduate extension (INFO 5617).** Choose a website you might plausibly study in your own research and profile its delivery infrastructure. Use the browser's Network tab to record the time to first byte, the full redirect chain, and evidence of CDN involvement (inspect the `Server`, `Via`, and similar response headers), then run `traceroute` (or `scapy`'s traceroute) to the same host and geolocate a few of the hops. Write a 500-word memo connecting these protocol-level observations to research-infrastructure concerns: where does "the" site actually live, how stable are measurements of it likely to be across time and vantage points, and what would an automated collection pipeline need to handle — redirects, rate limits, geographically varying responses — to gather data from it reliably? ## Social History and Public Interest The protocols described in this chapter were designed as public infrastructure. Vint Cerf and Bob Kahn developed TCP/IP in the 1970s for DARPA, creating a transport layer engineered for resilience under attack. Tim Berners-Lee invented HTTP and URLs at CERN in 1989–1991, and made the deliberate decision to release them into the public domain. The web was born open. Browser developer tools embody this original spirit of openness. The ability to "view source" — to inspect exactly what a web server sends your browser — was a foundational feature of the web. It is what makes web data science possible. As you will see in @sec-post-api, the tension between this transparency and platforms' desire to control their data is one of the defining conflicts of the contemporary web. ::: {.callout-note} ## Public Interest Connection The web's protocol stack was designed as public infrastructure — open standards documented in public RFCs, not proprietary products controlled by corporations. HTTP, HTML, DNS, and TCP/IP are all open protocols that anyone can implement. The "view source" capability and network inspection tools you learned in this chapter are features that embody **openness** (@sec-post-api). They exist because the web's architects believed that transparency was a feature, not a vulnerability. As platforms increasingly obfuscate their front-end code and move toward closed APIs, these tools become more important, not less. ::: ## Common Issues to Debug - **`scapy` import errors**: Requires `pcap` drivers. On Windows, install Npcap. On macOS, the built-in `libpcap` usually works. - **403 from Wikimedia API**: You forgot the custom `User-Agent` header. - **DNS results varying by location**: Normal — content delivery networks return different IPs based on your geographic location. - **URL encoding issues**: Use `urllib.parse.quote()` for article titles with spaces or special characters. ## Key Takeaways The web is built on a layered stack of protocols: TCP/IP transports data, DNS resolves names to addresses, HTTP structures the conversation, and URLs identify the target. Understanding this stack transforms debugging from guesswork into systematic diagnosis. When a request fails, you can ask: can I reach the server at all? Am I resolving the right address? What status code did I get? Are my headers correct? Is my URL well-formed? This systematic approach carries through every chapter that follows. ## Further Reading - Mozilla Developer Network — How the Web Works: <https://developer.mozilla.org/en-US/docs/Learn/Getting_started_with_the_web/How_the_Web_works> - Mozilla Developer Network — HTTP Overview: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview> - `scapy` documentation: <https://scapy.readthedocs.io/> - `dnspython` documentation: <https://dnspython.readthedocs.io/>