By the end of today you can

• explain dependence over time, and why forecasts are ranges;
• name the parts of a series: trend, seasonality, noise;
• store a series in pandas with a datetime index, and plot it;
• read time plots, seasonal plots, and autocorrelation;
• decompose a series with STL;
• forecast with benchmarks, ETS, and ARIMA;
• evaluate forecasts on a test period, and report prediction intervals.

A different way of thinking

A forecast of the future is honestly a range, never a single number.

The counselling service’s question

Five years of weekly visit counts: overwhelmed before exams, quiet in summer, staffing planned by guesswork.

How many visits should the service expect next year, week by week?

An internal analysis, done for the service rather than for publication.

A pandas series knows which column is time

clinic = pd.DataFrame({
    'day': range(1, 15),
    'visits': [12, 10, 9, 11, 7, 3, 2,  14, 11, 10, 12, 8, 4, 2]
}).set_index('day')

Two weeks of daily visits to a small clinic, from a Monday.

The index keeps observations in chronological order. Busy early in the week, quiet at the weekend.

Three parts of a pattern

weeks = np.arange(1, 157)
trend = 30 + 0.05 * weeks
seasonal = 12 * np.sin(2 * np.pi * weeks / 52)
noise = np.random.normal(0, 3, 156)
series = trend + seasonal + noise

An artificial weekly series for three years, built from its parts.

Only the sum is observed

Forecasting extends trend and season. Noise cannot be forecast: the main reason every forecast has a range.

The counselling data in pandas

counselling_visits = dt.load_counselling_visits()
counselling_visits['week_start'] = pd.to_datetime(counselling_visits['week_start'])
visits = counselling_visits.set_index('week_start').sort_index()

Five academic years, September 2020 to August 2025: 260 weeks.

week_start as DatetimeIndex aligns each observation to regular weekly Monday intervals.

Plot the series first

A seasonal pattern each academic year, a gentle upward trend, noise, and one unusual week in autumn 2023.

A seasonal plot

Every year has the same shape; later years lie slightly higher. The period is 52 weeks.

Busy weeks follow busy weeks

Knowing last week’s visits says a good deal about this week’s.

Autocorrelation

The correlation between the series and itself \(k\) steps earlier.

visits_acf = acf(visits['visits'], nlags=60)

A week resembles the same week a year earlier. Half a year apart, a busy term week often meets a quiet summer week.

The autocorrelation function

A peak at the seasonal lag is the signature of seasonality. No autocorrelation would mean nothing to forecast beyond the average.

Decomposition with STL

stl = STL(visits['visits'], period=52, robust=True)
visits_stl = stl.fit()

robust=True stops unusual weeks from distorting trend and season.

The recovered parts

The trend rises from about 35 to 41 visits a week, levelling off in the last year.

The unusual week

decomp_df = pd.DataFrame({
    'week': visits.index, 'visits': visits['visits'],
    'trend': visits_stl.trend, 'seasonal': visits_stl.seasonal,
    'remainder': visits_stl.resid
})
top_odd = decomp_df.iloc[decomp_df['remainder'].abs().argsort()[::-1][:1]]

The week of 23 October 2023: 71 visits, about 30 more than trend and season predict.

It was the mental health awareness week. A real week, not an error: kept, and the robust fit stops it distorting forecasts.

Growing seasonal swings

The peaks get higher as the trend rises, while the summer lows stay low.

When the season grows with the level, model the logarithm of the series:

\[ \log(\text{visits}) = \text{trend} + \text{seasonal} + \text{noise} \]

Forecast on the log scale, then exponentiate back with np.exp().

Benchmarks first

Every forecasting method must beat these, as every model in Chapters 11–13 had to beat a baseline.

Benchmarks on the clinic

fc_mean   = np.repeat(np.mean(clinic['visits']), 7)
fc_naive  = np.repeat(clinic['visits'].iloc[-1], 7)
fc_snaive = clinic['visits'].iloc[-7:].values

Two families of models

Both work best with short seasons: 4 quarters, 7 days, 12 months. A 52-week season is long.

Two strategies for weekly data

flowchart LR
    A["Weekly series"] --> B["STL: remove<br>the season"] --> C["ETS on the<br>adjusted series"] --> D["Add the<br>season back"]
    A --> E["ARIMA with<br>Fourier waves"]

Strategy 1: decompose with STL, fit ETS on adjusted data, add season back. Strategy 2: ARIMA with six pairs of harmonic waves.

Training and test periods

train = visits[visits.index < '2024-09-01']
test  = visits[visits.index >= '2024-09-01']

The test period must come after the training period: a forecast can only use the past.

Train on four academic years, test on the fifth.

Five models compared

# STL + ETS
stl_tr = STL(np.log(y_tr), period=52, robust=True).fit()
sa_tr = np.log(y_tr) - stl_tr.seasonal
ets_fit = Holt(sa_tr, initialization_method='estimated').fit()
fc_stl_ets = np.exp(ets_fit.forecast(52) + stl_tr.seasonal[-52:])

Fit benchmarks, STL + ETS, and Fourier ARIMA.

Accuracy on the test year

STL with ETS is best. The Fourier model struggles with sharp steps, such as the sudden drop into summer.

The two best, against reality

The seasonal naive copies last year’s noise, even the awareness-week spike. STL with ETS is smoother and closer.

Forecasting next year

stl_all = STL(np.log(y_all), period=52, robust=True).fit()
sa_all = np.log(y_all) - stl_all.seasonal
ets_all = Holt(sa_all, initialization_method='estimated').fit()
fc_log_next = ets_all.forecast(52) + stl_all.seasonal[-52:]
fc_next = np.exp(fc_log_next)

Refitted on all five years, so the forecasts use the most recent information.

Prediction intervals

First week: about 24 visits, anywhere from 17 to 33 unsurprising.

A total for the year needs simulation

np.random.seed(2026)
resid = stl_all.resid
futures = []
for _ in range(1000):
    sim_noise = np.random.choice(resid, size=52, replace=True)
    sim_path = np.exp(fc_log_next + sim_noise)
    futures.append(np.sum(sim_path))
q = np.quantile(futures, [0.025, 0.5, 0.975]).round()

Weeks do not vary independently, so weekly intervals cannot simply be added.

Residual bootstrap simulation produces 1,000 possible futures; their totals give the range.

The report to the service

• about 2,177 visits expected in 2025/26 (95% interval 2,050 to 2,464), against 2,056 in 2024/25;
• the busiest weeks will again be just before and during the exam periods;
• an awareness campaign can raise demand sharply for a week: plan extra capacity.

Forecasts assume the future resembles the past

A new booking system, a changed exam calendar, or a crisis like COVID-19 can make any forecast wrong.

• report forecasts with their intervals;
• say what they assume;
• update them as new data arrives.

In your field: environmental science

co2_m = co2_raw['co2'].resample('MS').mean().ffill()['1959-01-01':'1997-12-01']
# Train through 1990, forecast 1991-1997 (7 years)

Mauna Loa CO₂: a strong trend and a 12-month cycle that ETS and ARIMA model directly.

Practical lab: the Chapter 16 playground

Work through the playground exercises in your browser, with hints and solutions.

Run interactive Python code with Pyodide; download the complete offline project.

Practical exercises 1–4: parts and models

1. Which benchmark suits a trend without seasonality? Test it on the clinic.
2. Plot the seasonally adjusted series: what does it reveal?
3. Fourier terms with \(K = 2\) and \(K = 12\).
4. Train on three years, test on the fourth.

Practical exercises 5–7: forecasts and communication

5. Expected visits in the four weeks before the first exams, with an interval.
6. Explain a 95% prediction interval to the service.
7. Double the noise in the artificial series: what happens to forecast intervals?

Try this yourself

Find a series in your own field.

• plot it, and name its trend, season, and noise;
• choose the seasonal period;
• pick a benchmark it must beat;
• decide what event could make a forecast wrong.

Troubleshooting guide (Part 1)

Troubleshooting guide (Part 2)

Completion checklist

• the series has a clean datetime index;
• time, seasonal, and ACF plots were inspected;
• the series was decomposed, and unusual weeks explained;
• benchmarks are reported;
• models are tested on a later period;
• forecasts include prediction intervals;
• totals come from simulation;
• assumptions are stated.

Misconceptions to leave behind (Part 1)

Misconceptions to leave behind (Part 2)

The chapter in one sentence

Separate trend, season, and noise, beat the simple benchmarks on a later test period, and always report a forecast as a range with its assumptions.

Next: Chapter 17

The next chapter makes research reproducible:

• reproducibility and credibility;
• documents that contain their own analysis;
• a results chapter in Quarto;
• fixed package versions;
• interactive dashboards and open science.

Questions

What regular measurement in your field has a seasonal pattern?

What event could break its forecast?