10 Time Series Visualization and Patterns

In Chapter 9 you learned to parse, index, and slice temporal data. This chapter puts those skills to work on a core challenge of data analysis: making sense of noisy temporal data. Raw time series are often too volatile to interpret — daily temperatures swing wildly, stock prices jitter, and taxi pickups spike and plummet. The signal is buried in the noise.

Temporal data is rarely random and uncorrelated. There are often consistent patterns in human and natural behavior that recur at various frequencies: hourly circadian rhythms, weekly work-weekend cycles, monthly billing cycles, and annual seasonality. Identifying and understanding these underlying temporal patterns is crucial for describing behavior, characterizing anomalies, and forecasting future behavior.

This chapter introduces techniques for revealing these patterns: resampling, rolling averages, boxplot summaries, temporal pattern decomposition across frequencies, and formal time series decomposition.

10.1 Setup

import numpy as np
np.set_printoptions(suppress=True)

import pandas as pd
idx = pd.IndexSlice
pd.options.display.max_columns = 100

%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import ticker

import seaborn as sb

# Define a weekday order — otherwise many libraries will list them alphabetically
weekday_order = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']

10.2 Loading Multiple Datasets

We will work with several datasets at different temporal frequencies to illustrate how the same techniques apply across scales.

Dataset	Frequency
NYC taxi pickups	minute
S&P 500	daily
Wikipedia pageviews	daily
Berkeley Earth temperature	daily
DIA passenger traffic	monthly

10.2.1 NYC taxi pickups

New York City’s Taxi and Limousine Commission publishes high-resolution anonymized data about every trip starting or ending in the city.

nyc_df = pd.read_csv('nyc_pickups.csv', parse_dates=['timestamp'])
nyc_df.set_index('timestamp', inplace=True)

# Extract temporal components for grouping
nyc_df['min'] = nyc_df.index.minute
nyc_df['hour'] = nyc_df.index.hour
nyc_df['weekday'] = nyc_df.index.day_name()

nyc_df.head()

10.2.2 S&P 500

sp500_df = pd.read_csv('sp500.csv', parse_dates=['Date'])
sp500_df.set_index('Date', inplace=True)

sp500_df['weekday'] = sp500_df.index.day_name()
sp500_df['day'] = sp500_df.index.day
sp500_df['month'] = sp500_df.index.month

sp500_df.head()

10.2.3 Berkeley Earth Surface Temperature

Daily land surface temperature estimates from 1880 to the present, from Berkeley Earth.

temp_df = pd.read_csv('avg_temp_1880_2024.csv', parse_dates=['Date'])
temp_df.set_index('Date', inplace=True)

temp_df['weekday'] = temp_df.index.day_name()
temp_df['day'] = temp_df.index.day
temp_df['month'] = temp_df.index.month
temp_df['Year'] = temp_df.index.year

temp_df.head()

10.2.4 DIA passenger traffic

passengers_df = pd.read_csv('passengers.csv')
passengers_df['Month'] = pd.to_datetime(passengers_df['Month'])
passengers_df.set_index('Month', inplace=True)

passengers_df['Total'] = passengers_df.sum(axis=1)
passengers_df['month'] = passengers_df.index.month

passengers_df.tail()

10.3 Visualizing Raw Time Series

Start by plotting the raw temperature data. With over 50,000 daily observations spanning 140+ years, the result is dense and hard to interpret.

f, ax = plt.subplots(figsize=(18, 6))

temp_df['AvgTemp'].plot(ax=ax)
ax.set_ylim((0, 16))
ax.set_ylabel('Temperature (°C)')
ax.set_title('Average daily temperature 1880–2024')

Is there a warming trend? Hard to say with this much daily variation. We need to smooth the data.

10.4 Resampling

Resampling changes the frequency of your time series. You can use pd.Grouper inside .groupby() or the .resample() method to aggregate from daily to monthly, annual, or any other frequency.

10.4.1 Sampling specific dates

Try resampling to just July 1 of each year. The temperature on a single day is not a reliable indicator of a warming pattern.

july_firsts = pd.date_range(
    start='1880-07-01',
    end='2022-07-01',
    freq=pd.DateOffset(years=1)
)

july_first_temps = temp_df.loc[july_firsts]

f, ax = plt.subplots(figsize=(18, 6))
july_first_temps['AvgTemp'].plot(ax=ax)
ax.set_ylim((6, 12))
ax.set_ylabel('Temperature (°C)')
ax.set_title('July 1 temperatures 1880–2022')
ax.axhline(8.59, c='k', ls='--', lw=.5)

10.4.2 Monthly averages

Averaging all days in each month is more robust than sampling a single day.

monthly_avg_temps = temp_df.groupby(pd.Grouper(freq='MS')).agg({'AvgTemp': 'mean'})

f, ax = plt.subplots(figsize=(18, 6))
monthly_avg_temps['AvgTemp'].plot(ax=ax)
ax.set_ylabel('Temperature (°C)')
ax.set_title('Average monthly temperature 1880–2024')
ax.axhline(8.59, c='k', ls='--', lw=.5)
ax.set_ylim((6, 12))

10.4.3 Annual averages

Aggregating to annual averages removes the seasonal cycle entirely and reveals the long-term trend.

annual_avg_temps = temp_df.groupby(pd.Grouper(freq='YS')).agg({'AvgTemp': 'mean'})

f, ax = plt.subplots(figsize=(18, 6))
annual_avg_temps['AvgTemp'].plot(ax=ax)
ax.set_ylabel('Temperature (°C)')
ax.set_title('Average annual temperature 1880–2024')
ax.axhline(8.59, c='k', ls='--', lw=.5)
ax.set_ylim((6, 12))

The warming trend that was invisible in the daily data is now unmistakable. This illustrates a fundamental trade-off: smoothing reveals trends but removes detail.

10.5 Rolling Averages

Unlike resampling, which bins data into discrete periods, rolling averages produce a value for every data point, making them useful for overlaying on the original data.

quarterly_rollmean = temp_df['AvgTemp'].rolling(90).mean()
annual_rollmean = temp_df['AvgTemp'].rolling(365).mean()
decadal_rollmean = temp_df['AvgTemp'].rolling(3650).mean()

f, ax = plt.subplots(figsize=(18, 6))

quarterly_rollmean.plot(ax=ax, color='tab:blue', label='Quarterly')
annual_rollmean.plot(ax=ax, color='tab:orange', lw=2, label='Annual')
decadal_rollmean.plot(ax=ax, color='tab:green', lw=5, label='Decadal')

ax.set_ylabel('Temperature (°C)')
ax.set_title('Average daily temperature 1880–2024')
ax.legend(loc='upper left')
ax.axhline(8.59, c='k', ls='--', lw=.5)
ax.set_ylim((6, 12))

Exercise 1: Visualize the NYC data

Use nyc_df to visualize the “yellowcab” and “rideshare” columns. Try hourly aggregation, daily aggregation, and overlaying both series.

Solution:

nyc_df.groupby(pd.Grouper(freq='h')).agg({'yellowcab': 'sum'}).plot()

nyc_df[['yellowcab', 'rideshare']].plot(alpha=.5)

10.6 Box Plots for Temporal Distributions

Box plots, introduced in Chapter 7, are excellent for summarizing the distribution of values within each time period. We can create one box per year to show both central tendency and variability.

First, a histogram for a single year (1984):

temps_1984 = temp_df.loc['1984', 'AvgTemp']

f, ax = plt.subplots(figsize=(6, 6))
temps_1984.hist(bins=np.linspace(5, 12, 30), ax=ax, grid=False)
ax.set_xlabel('Temperature (°C)')
ax.set_ylabel('Count (days)')
ax.set_xlim((5, 12))
ax.axvline(temps_1984.mean(), color='tab:red')
ax.axvline(temps_1984.median(), color='tab:orange')

The same data as a box plot:

f, ax = plt.subplots(figsize=(2, 6))
sb.boxplot(y=temps_1984, ax=ax, color='tab:blue', whis=2)
ax.set_ylabel('Temperature (°C)')
ax.set_ylim((5, 12))

Now a box plot for every year since 1970. The gradual upward shift of the boxes makes the warming trend visible while preserving information about annual variability.

temps_since_1970 = temp_df.loc['1970':, :]

g = sb.catplot(
    data=temps_since_1970,
    x='Year',
    y='AvgTemp',
    kind='box',
    color='tab:blue',
    aspect=3
)

ax = g.axes.flat[0]
ax.axhline(8.59, c='tab:red', ls='--', lw=2, zorder=1)

Exercise 2: DIA traffic boxplot

Use passengers_df and seaborn to make a boxplot of the distribution of monthly passenger totals (y-axis) by year (x-axis).

Solution:

passengers_df['Year'] = passengers_df.index.year

g = sb.catplot(
    data=passengers_df,
    x='Year',
    y='Total',
    kind='box',
    aspect=2.5,
    color='steelblue'
)

g.set_xticklabels(rotation=45)

10.7 Temporal Patterns at Different Frequencies

One of the most useful applications of temporal data analysis is identifying recurring patterns at different time scales. Let’s systematically examine patterns at the hourly, daily, weekly, monthly, and annual level.

10.7.1 Hourly patterns: Circadian rhythms

Most people in a given location sleep at roughly the same time. When people are asleep, activity drops, and this should be visible in the data.

sb.catplot(
    x='hour',
    y='yellowcab',
    data=nyc_df,
    kind='point',
    aspect=2
)

Taxi pickups are lowest around 4–5 AM and peak during the evening rush.

10.7.2 Weekly patterns: Workdays vs. weekends

We can make a heatmap to show the interaction between hour of day and day of week.

rideshare_pivot = pd.pivot_table(
    nyc_df,
    index='hour',
    columns='weekday',
    values='yellowcab',
    aggfunc='sum'
)

rideshare_pivot = rideshare_pivot.loc[:, weekday_order]

f, ax = plt.subplots(figsize=(4, 6))
sb.heatmap(rideshare_pivot, ax=ax, square=True)
f.tight_layout()

The heatmap reveals that weekday patterns (morning and evening peaks) differ from weekend patterns (later starts, more nightlife activity). Different datasets show different weekly patterns.

f, ax = plt.subplots(figsize=(12, 4))
sb.pointplot(x='weekday', y='Volume', data=sp500_df, order=weekday_order)

f, ax = plt.subplots(figsize=(12, 4))
sb.pointplot(x='weekday', y='AvgTemp', data=temp_df, order=weekday_order)

The weekday temperature “pattern”

If you see a tiny weekly pattern in global temperature data, pause before concluding that the weather follows a weekly cycle. Any apparent pattern is likely an artifact of the data processing, measurement timing, or station distribution — not a real physical phenomenon. Critical thinking about spurious patterns is essential.

10.7.3 Day-of-month patterns

f, ax = plt.subplots(figsize=(12, 4))
sb.pointplot(x='day', y='Volume', data=sp500_df)

10.7.4 Annual patterns: Seasonality

Seasonality is one of the most common temporal patterns. Temperature has an obvious annual cycle, but many human activities do as well.

f, ax = plt.subplots(figsize=(12, 4))
sb.pointplot(x='month', y='AvgTemp', data=temp_df)

f, ax = plt.subplots(figsize=(12, 4))
sb.pointplot(x='month', y='Total', data=passengers_df)

DIA passenger traffic shows a strong seasonal pattern: summer months (June–August) are busiest, with a secondary peak in March (spring break).

10.8 Decomposing Time Series

A more formal approach to separating trend, seasonal, and residual components uses statistical decomposition. The statsmodels library provides seasonal_decompose, which breaks a time series into three additive (or multiplicative) components: trend, seasonal pattern, and residual noise.

import statsmodels.api as sm

f, ax = plt.subplots(figsize=(8, 4))
passengers_df['Total'].plot(ax=ax)
ax.set_ylabel('Total passengers')
ax.set_title('Monthly passenger traffic through DIA')

A 12-month rolling average removes the seasonal signal to reveal the trend.

f, ax = plt.subplots(figsize=(12, 6))

passengers_df['Total'].plot(ax=ax)
passengers_df['Total'].rolling(12).mean().plot(ax=ax, c='tab:green', lw=2)

ax.set_ylabel('Total passengers')
ax.set_title('Monthly DIA traffic with 12-month rolling average')

10.8.1 Formal decomposition

The seasonal_decompose function produces four panels: the observed data, the estimated trend, the seasonal component, and the residual.

decomposition = sm.tsa.seasonal_decompose(passengers_df['Total'], model='additive')

plt.rcParams["figure.figsize"] = (8, 8)
f = decomposition.plot()

for _ax in f.axes:
    _ax.axvline(pd.Timestamp('2001-09-11'), color='r', ls='--', lw=1, zorder=0)
    _ax.axvline(pd.Timestamp('2020-03-11'), color='r', ls='--', lw=1, zorder=0)

The trend component shows steady growth interrupted by 9/11 and COVID-19. The seasonal component shows the consistent summer peak and winter trough. The residual captures everything the trend and seasonal components do not explain — including the massive COVID shock.

10.8.2 Linear modeling of trend

You can also model the long-term trend with a linear regression, as introduced in Chapter 13. First, create a numeric feature for months since the airport opened.

passengers_df['months_since_opening'] = (
    passengers_df.index - pd.Timestamp('1995-01-01')
) / pd.Timedelta(30, 'D')

g = sb.lmplot(
    x='months_since_opening',
    y='Total',
    data=passengers_df,
    aspect=2,
    ci=0,
    scatter_kws={'color': 'k', 's': 25},
    line_kws={'lw': 5, 'color': 'r', 'zorder': 0}
)

The raw data has enormous variance because of seasonality. Smoothing first with a 12-month rolling average and then fitting a linear model produces a much cleaner trend estimate.

passengers_df['Total_smoothed'] = passengers_df['Total'].rolling(12).mean()

g = sb.lmplot(
    x='months_since_opening',
    y='Total_smoothed',
    data=passengers_df,
    aspect=2,
    ci=0,
    scatter_kws={'color': 'k', 's': 25},
    line_kws={'lw': 5, 'color': 'r', 'zorder': 0}
)

Exercise 3: Rolling averages on NYC data

Compute 7-day and 30-day rolling averages of daily yellow cab pickups. Plot them together. Which window size better reveals the weekly cycle? Which better reveals the monthly trend?

Solution:

daily_cabs = nyc_df.groupby(pd.Grouper(freq='D')).agg({'yellowcab': 'sum'})

f, ax = plt.subplots(figsize=(14, 5))
daily_cabs['yellowcab'].rolling(7).mean().plot(ax=ax, label='7-day', alpha=0.7)
daily_cabs['yellowcab'].rolling(30).mean().plot(ax=ax, label='30-day', alpha=0.9)
ax.legend()
ax.set_ylabel('Daily Pickups')
ax.set_title('Rolling Average Yellow Cab Pickups')

The 7-day average smooths daily variation while preserving weekly structure. The 30-day average smooths weeks and reveals monthly-scale trends.

Exercise 4: Year-over-year DIA comparison

Compute the annual sum of DIA passengers. Then compute the year-over-year percentage change. Which year had the biggest decline? The biggest recovery?

Solution:

annual = passengers_df.groupby(passengers_df.index.year).agg({'Total': 'sum'})
annual['pct_change'] = annual['Total'].pct_change() * 100
annual

2020 will show the largest decline (COVID-19); 2021 will show the largest recovery.

Exercise 5: Seasonal overlay

Create a line plot with monthly total passengers for each year overlaid on the same axes (one line per year). Do you see a consistent seasonal pattern?

Solution:

passengers_df['Year'] = passengers_df.index.year
passengers_df['MonthNum'] = passengers_df.index.month

f, ax = plt.subplots(figsize=(12, 6))

for year, group in passengers_df.groupby('Year'):
    ax.plot(group['MonthNum'], group['Total'], alpha=0.4)

ax.set_xlabel('Month')
ax.set_ylabel('Total Passengers')
ax.set_title('DIA Monthly Passengers by Year (overlaid)')
ax.set_xticks(range(1, 13))

The seasonal pattern is highly consistent: summer peaks, winter troughs, March secondary peak. The 2020 outlier year is clearly visible as the line that collapses in March.

Exercise 6: Heatmap of Wikipedia pageviews

If you have access to Wikipedia pageview data, create a heatmap showing pageviews by hour and day of week. What patterns do you see? How do they differ from the taxi data?

Solution:

Wikipedia pageviews would likely show higher traffic during work hours (people researching at their desks) and a weekly pattern with lower traffic on weekends. Unlike taxi pickups, which peak during commuting hours, Wikipedia traffic may peak during the mid-afternoon “procrastination window.”

10.9 Looking Ahead

You have developed a toolkit for visualizing and analyzing temporal data: resampling at different frequencies, smoothing with rolling averages, decomposing into trend and seasonal components, and identifying patterns across time scales. In Chapter 11, you will address one of the most important and most neglected aspects of data communication: how to honestly represent what you do not know.

# Time Series Visualization and Patterns {#sec-time-series} In @sec-temporal-data you learned to parse, index, and slice temporal data. This chapter puts those skills to work on a core challenge of data analysis: making sense of noisy temporal data. Raw time series are often too volatile to interpret — daily temperatures swing wildly, stock prices jitter, and taxi pickups spike and plummet. The signal is buried in the noise. Temporal data is rarely random and uncorrelated. There are often consistent patterns in human and natural behavior that recur at various frequencies: hourly circadian rhythms, weekly work-weekend cycles, monthly billing cycles, and annual seasonality. Identifying and understanding these underlying temporal patterns is crucial for describing behavior, characterizing anomalies, and forecasting future behavior. This chapter introduces techniques for revealing these patterns: resampling, rolling averages, boxplot summaries, temporal pattern decomposition across frequencies, and formal time series decomposition. ## Setup ```{python} import numpy as np np.set_printoptions(suppress=True) import pandas as pd idx = pd.IndexSlice pd.options.display.max_columns = 100 %matplotlib inline import matplotlib.pyplot as plt from matplotlib import ticker import seaborn as sb ``` ```{python} # Define a weekday order — otherwise many libraries will list them alphabetically weekday_order = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'] ``` ## Loading Multiple Datasets We will work with several datasets at different temporal frequencies to illustrate how the same techniques apply across scales. | Dataset | Frequency | | --- | --- | | NYC taxi pickups | minute | | S&P 500 | daily | | Wikipedia pageviews | daily | | Berkeley Earth temperature | daily | | DIA passenger traffic | monthly | ### NYC taxi pickups New York City's Taxi and Limousine Commission publishes [high-resolution anonymized data](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page) about every trip starting or ending in the city. ```{python} nyc_df = pd.read_csv('nyc_pickups.csv', parse_dates=['timestamp']) nyc_df.set_index('timestamp', inplace=True) # Extract temporal components for grouping nyc_df['min'] = nyc_df.index.minute nyc_df['hour'] = nyc_df.index.hour nyc_df['weekday'] = nyc_df.index.day_name() nyc_df.head() ``` ### S&P 500 ```{python} sp500_df = pd.read_csv('sp500.csv', parse_dates=['Date']) sp500_df.set_index('Date', inplace=True) sp500_df['weekday'] = sp500_df.index.day_name() sp500_df['day'] = sp500_df.index.day sp500_df['month'] = sp500_df.index.month sp500_df.head() ``` ### Berkeley Earth Surface Temperature Daily land surface temperature estimates from 1880 to the present, from [Berkeley Earth](http://berkeleyearth.org/data/). ```{python} temp_df = pd.read_csv('avg_temp_1880_2024.csv', parse_dates=['Date']) temp_df.set_index('Date', inplace=True) temp_df['weekday'] = temp_df.index.day_name() temp_df['day'] = temp_df.index.day temp_df['month'] = temp_df.index.month temp_df['Year'] = temp_df.index.year temp_df.head() ``` ### DIA passenger traffic ```{python} passengers_df = pd.read_csv('passengers.csv') passengers_df['Month'] = pd.to_datetime(passengers_df['Month']) passengers_df.set_index('Month', inplace=True) passengers_df['Total'] = passengers_df.sum(axis=1) passengers_df['month'] = passengers_df.index.month passengers_df.tail() ``` ## Visualizing Raw Time Series Start by plotting the raw temperature data. With over 50,000 daily observations spanning 140+ years, the result is dense and hard to interpret. ```{python} f, ax = plt.subplots(figsize=(18, 6)) temp_df['AvgTemp'].plot(ax=ax) ax.set_ylim((0, 16)) ax.set_ylabel('Temperature (°C)') ax.set_title('Average daily temperature 1880–2024') ``` Is there a warming trend? Hard to say with this much daily variation. We need to smooth the data. ## Resampling Resampling changes the frequency of your time series. You can use `pd.Grouper` inside `.groupby()` or the `.resample()` method to aggregate from daily to monthly, annual, or any other frequency. ### Sampling specific dates Try resampling to just July 1 of each year. The temperature on a single day is not a reliable indicator of a warming pattern. ```{python} july_firsts = pd.date_range( start='1880-07-01', end='2022-07-01', freq=pd.DateOffset(years=1) ) july_first_temps = temp_df.loc[july_firsts] f, ax = plt.subplots(figsize=(18, 6)) july_first_temps['AvgTemp'].plot(ax=ax) ax.set_ylim((6, 12)) ax.set_ylabel('Temperature (°C)') ax.set_title('July 1 temperatures 1880–2022') ax.axhline(8.59, c='k', ls='--', lw=.5) ``` ### Monthly averages Averaging all days in each month is more robust than sampling a single day. ```{python} monthly_avg_temps = temp_df.groupby(pd.Grouper(freq='MS')).agg({'AvgTemp': 'mean'}) f, ax = plt.subplots(figsize=(18, 6)) monthly_avg_temps['AvgTemp'].plot(ax=ax) ax.set_ylabel('Temperature (°C)') ax.set_title('Average monthly temperature 1880–2024') ax.axhline(8.59, c='k', ls='--', lw=.5) ax.set_ylim((6, 12)) ``` ### Annual averages Aggregating to annual averages removes the seasonal cycle entirely and reveals the long-term trend. ```{python} annual_avg_temps = temp_df.groupby(pd.Grouper(freq='YS')).agg({'AvgTemp': 'mean'}) f, ax = plt.subplots(figsize=(18, 6)) annual_avg_temps['AvgTemp'].plot(ax=ax) ax.set_ylabel('Temperature (°C)') ax.set_title('Average annual temperature 1880–2024') ax.axhline(8.59, c='k', ls='--', lw=.5) ax.set_ylim((6, 12)) ``` The warming trend that was invisible in the daily data is now unmistakable. This illustrates a fundamental trade-off: smoothing reveals trends but removes detail. ## Rolling Averages Unlike resampling, which bins data into discrete periods, rolling averages produce a value for every data point, making them useful for overlaying on the original data. ```{python} quarterly_rollmean = temp_df['AvgTemp'].rolling(90).mean() annual_rollmean = temp_df['AvgTemp'].rolling(365).mean() decadal_rollmean = temp_df['AvgTemp'].rolling(3650).mean() ``` ```{python} f, ax = plt.subplots(figsize=(18, 6)) quarterly_rollmean.plot(ax=ax, color='tab:blue', label='Quarterly') annual_rollmean.plot(ax=ax, color='tab:orange', lw=2, label='Annual') decadal_rollmean.plot(ax=ax, color='tab:green', lw=5, label='Decadal') ax.set_ylabel('Temperature (°C)') ax.set_title('Average daily temperature 1880–2024') ax.legend(loc='upper left') ax.axhline(8.59, c='k', ls='--', lw=.5) ax.set_ylim((6, 12)) ``` ::: {.callout-note collapse="true"} ## Exercise 1: Visualize the NYC data Use `nyc_df` to visualize the "yellowcab" and "rideshare" columns. Try hourly aggregation, daily aggregation, and overlaying both series. **Solution:** ```{python} nyc_df.groupby(pd.Grouper(freq='h')).agg({'yellowcab': 'sum'}).plot() ``` ```{python} nyc_df[['yellowcab', 'rideshare']].plot(alpha=.5) ``` ::: ## Box Plots for Temporal Distributions Box plots, introduced in @sec-catplots, are excellent for summarizing the distribution of values within each time period. We can create one box per year to show both central tendency and variability. First, a histogram for a single year (1984): ```{python} temps_1984 = temp_df.loc['1984', 'AvgTemp'] f, ax = plt.subplots(figsize=(6, 6)) temps_1984.hist(bins=np.linspace(5, 12, 30), ax=ax, grid=False) ax.set_xlabel('Temperature (°C)') ax.set_ylabel('Count (days)') ax.set_xlim((5, 12)) ax.axvline(temps_1984.mean(), color='tab:red') ax.axvline(temps_1984.median(), color='tab:orange') ``` The same data as a box plot: ```{python} f, ax = plt.subplots(figsize=(2, 6)) sb.boxplot(y=temps_1984, ax=ax, color='tab:blue', whis=2) ax.set_ylabel('Temperature (°C)') ax.set_ylim((5, 12)) ``` Now a box plot for every year since 1970. The gradual upward shift of the boxes makes the warming trend visible while preserving information about annual variability. ```{python} temps_since_1970 = temp_df.loc['1970':, :] g = sb.catplot( data=temps_since_1970, x='Year', y='AvgTemp', kind='box', color='tab:blue', aspect=3 ) ax = g.axes.flat[0] ax.axhline(8.59, c='tab:red', ls='--', lw=2, zorder=1) ``` ::: {.callout-note collapse="true"} ## Exercise 2: DIA traffic boxplot Use `passengers_df` and seaborn to make a boxplot of the distribution of monthly passenger totals (y-axis) by year (x-axis). **Solution:** ```{python} passengers_df['Year'] = passengers_df.index.year g = sb.catplot( data=passengers_df, x='Year', y='Total', kind='box', aspect=2.5, color='steelblue' ) g.set_xticklabels(rotation=45) ``` ::: ## Temporal Patterns at Different Frequencies One of the most useful applications of temporal data analysis is identifying recurring patterns at different time scales. Let's systematically examine patterns at the hourly, daily, weekly, monthly, and annual level. ### Hourly patterns: Circadian rhythms Most people in a given location sleep at roughly the same time. When people are asleep, activity drops, and this should be visible in the data. ```{python} sb.catplot( x='hour', y='yellowcab', data=nyc_df, kind='point', aspect=2 ) ``` Taxi pickups are lowest around 4–5 AM and peak during the evening rush. ### Weekly patterns: Workdays vs. weekends We can make a heatmap to show the interaction between hour of day and day of week. ```{python} rideshare_pivot = pd.pivot_table( nyc_df, index='hour', columns='weekday', values='yellowcab', aggfunc='sum' ) rideshare_pivot = rideshare_pivot.loc[:, weekday_order] f, ax = plt.subplots(figsize=(4, 6)) sb.heatmap(rideshare_pivot, ax=ax, square=True) f.tight_layout() ``` The heatmap reveals that weekday patterns (morning and evening peaks) differ from weekend patterns (later starts, more nightlife activity). Different datasets show different weekly patterns. ```{python} f, ax = plt.subplots(figsize=(12, 4)) sb.pointplot(x='weekday', y='Volume', data=sp500_df, order=weekday_order) ``` ```{python} f, ax = plt.subplots(figsize=(12, 4)) sb.pointplot(x='weekday', y='AvgTemp', data=temp_df, order=weekday_order) ``` ::: {.callout-tip} ## The weekday temperature "pattern" If you see a tiny weekly pattern in global temperature data, pause before concluding that the weather follows a weekly cycle. Any apparent pattern is likely an artifact of the data processing, measurement timing, or station distribution — not a real physical phenomenon. Critical thinking about spurious patterns is essential. ::: ### Day-of-month patterns ```{python} f, ax = plt.subplots(figsize=(12, 4)) sb.pointplot(x='day', y='Volume', data=sp500_df) ``` ### Annual patterns: Seasonality Seasonality is one of the most common temporal patterns. Temperature has an obvious annual cycle, but many human activities do as well. ```{python} f, ax = plt.subplots(figsize=(12, 4)) sb.pointplot(x='month', y='AvgTemp', data=temp_df) ``` ```{python} f, ax = plt.subplots(figsize=(12, 4)) sb.pointplot(x='month', y='Total', data=passengers_df) ``` DIA passenger traffic shows a strong seasonal pattern: summer months (June–August) are busiest, with a secondary peak in March (spring break). ## Decomposing Time Series A more formal approach to separating trend, seasonal, and residual components uses statistical decomposition. The `statsmodels` library provides `seasonal_decompose`, which breaks a time series into three additive (or multiplicative) components: trend, seasonal pattern, and residual noise. ```{python} import statsmodels.api as sm ``` ```{python} f, ax = plt.subplots(figsize=(8, 4)) passengers_df['Total'].plot(ax=ax) ax.set_ylabel('Total passengers') ax.set_title('Monthly passenger traffic through DIA') ``` A 12-month rolling average removes the seasonal signal to reveal the trend. ```{python} f, ax = plt.subplots(figsize=(12, 6)) passengers_df['Total'].plot(ax=ax) passengers_df['Total'].rolling(12).mean().plot(ax=ax, c='tab:green', lw=2) ax.set_ylabel('Total passengers') ax.set_title('Monthly DIA traffic with 12-month rolling average') ``` ### Formal decomposition The `seasonal_decompose` function produces four panels: the observed data, the estimated trend, the seasonal component, and the residual. ```{python} decomposition = sm.tsa.seasonal_decompose(passengers_df['Total'], model='additive') plt.rcParams["figure.figsize"] = (8, 8) f = decomposition.plot() for _ax in f.axes: _ax.axvline(pd.Timestamp('2001-09-11'), color='r', ls='--', lw=1, zorder=0) _ax.axvline(pd.Timestamp('2020-03-11'), color='r', ls='--', lw=1, zorder=0) ``` The trend component shows steady growth interrupted by 9/11 and COVID-19. The seasonal component shows the consistent summer peak and winter trough. The residual captures everything the trend and seasonal components do not explain — including the massive COVID shock. ### Linear modeling of trend You can also model the long-term trend with a linear regression, as introduced in @sec-linear-models. First, create a numeric feature for months since the airport opened. ```{python} passengers_df['months_since_opening'] = ( passengers_df.index - pd.Timestamp('1995-01-01') ) / pd.Timedelta(30, 'D') g = sb.lmplot( x='months_since_opening', y='Total', data=passengers_df, aspect=2, ci=0, scatter_kws={'color': 'k', 's': 25}, line_kws={'lw': 5, 'color': 'r', 'zorder': 0} ) ``` The raw data has enormous variance because of seasonality. Smoothing first with a 12-month rolling average and then fitting a linear model produces a much cleaner trend estimate. ```{python} passengers_df['Total_smoothed'] = passengers_df['Total'].rolling(12).mean() g = sb.lmplot( x='months_since_opening', y='Total_smoothed', data=passengers_df, aspect=2, ci=0, scatter_kws={'color': 'k', 's': 25}, line_kws={'lw': 5, 'color': 'r', 'zorder': 0} ) ``` ::: {.callout-note collapse="true"} ## Exercise 3: Rolling averages on NYC data Compute 7-day and 30-day rolling averages of daily yellow cab pickups. Plot them together. Which window size better reveals the weekly cycle? Which better reveals the monthly trend? **Solution:** ```{python} daily_cabs = nyc_df.groupby(pd.Grouper(freq='D')).agg({'yellowcab': 'sum'}) f, ax = plt.subplots(figsize=(14, 5)) daily_cabs['yellowcab'].rolling(7).mean().plot(ax=ax, label='7-day', alpha=0.7) daily_cabs['yellowcab'].rolling(30).mean().plot(ax=ax, label='30-day', alpha=0.9) ax.legend() ax.set_ylabel('Daily Pickups') ax.set_title('Rolling Average Yellow Cab Pickups') ``` The 7-day average smooths daily variation while preserving weekly structure. The 30-day average smooths weeks and reveals monthly-scale trends. ::: ::: {.callout-note collapse="true"} ## Exercise 4: Year-over-year DIA comparison Compute the annual sum of DIA passengers. Then compute the year-over-year percentage change. Which year had the biggest decline? The biggest recovery? **Solution:** ```{python} annual = passengers_df.groupby(passengers_df.index.year).agg({'Total': 'sum'}) annual['pct_change'] = annual['Total'].pct_change() * 100 annual ``` 2020 will show the largest decline (COVID-19); 2021 will show the largest recovery. ::: ::: {.callout-note collapse="true"} ## Exercise 5: Seasonal overlay Create a line plot with monthly total passengers for each year overlaid on the same axes (one line per year). Do you see a consistent seasonal pattern? **Solution:** ```{python} passengers_df['Year'] = passengers_df.index.year passengers_df['MonthNum'] = passengers_df.index.month f, ax = plt.subplots(figsize=(12, 6)) for year, group in passengers_df.groupby('Year'): ax.plot(group['MonthNum'], group['Total'], alpha=0.4) ax.set_xlabel('Month') ax.set_ylabel('Total Passengers') ax.set_title('DIA Monthly Passengers by Year (overlaid)') ax.set_xticks(range(1, 13)) ``` The seasonal pattern is highly consistent: summer peaks, winter troughs, March secondary peak. The 2020 outlier year is clearly visible as the line that collapses in March. ::: ::: {.callout-note collapse="true"} ## Exercise 6: Heatmap of Wikipedia pageviews If you have access to Wikipedia pageview data, create a heatmap showing pageviews by hour and day of week. What patterns do you see? How do they differ from the taxi data? **Solution:** Wikipedia pageviews would likely show higher traffic during work hours (people researching at their desks) and a weekly pattern with lower traffic on weekends. Unlike taxi pickups, which peak during commuting hours, Wikipedia traffic may peak during the mid-afternoon "procrastination window." ::: ## Looking Ahead You have developed a toolkit for visualizing and analyzing temporal data: resampling at different frequencies, smoothing with rolling averages, decomposing into trend and seasonal components, and identifying patterns across time scales. In @sec-uncertainty, you will address one of the most important and most neglected aspects of data communication: how to honestly represent what you do not know.