What Are We Predicting?

Market prediction tests whether factors contain information about future returns. If value/momentum are risk factors or mispricing patterns, they might predict market movements.

Prediction task: Forecast next-month market return using lagged factor data

Setup:

• Target (Y): Next month’s market return (MKT_{t+1})
• Predictors (X): Current month’s factor returns (MKT_t, HML_t, MOM_t, etc.)
• Challenge: Returns are noisy: signal-to-noise ratio is low
• Goal: Build model that outperforms naive benchmark (historical mean)

Why this matters: If R² OOS > 0, the model beats the naive benchmark (historical mean): market timing may add value. If R² OOS ≤ 0 (yes, it can be negative!), the model is no better than: or worse than: simply predicting the mean. Stick to strategic asset allocation.

Prediction Setup: Data Structure

Understanding the data structure is essential before discussing methodology. Here’s what factor prediction data looks like.

import pandas as pd
import numpy as np

# Simulate JKP-style factor data for illustration
np.random.seed(42)
n_months = 360  # 30 years monthly

# Create date index
dates = pd.date_range(start='1994-01-01', periods=n_months, freq='ME')

# Simulate factor returns (correlated, as in reality)
# Market excess return (target, but lagged for prediction)
mkt = np.random.normal(0.008, 0.045, n_months)

# Other factors (correlated with each other)
hml = 0.2 * mkt + np.random.normal(0.003, 0.03, n_months)  # Value
mom = -0.1 * mkt + np.random.normal(0.006, 0.04, n_months)  # Momentum
smb = 0.3 * mkt + np.random.normal(0.002, 0.03, n_months)  # Size
rmw = 0.1 * hml + np.random.normal(0.003, 0.02, n_months)  # Quality

# Create DataFrame
data = pd.DataFrame({
    'date': dates,
    'MKT': mkt,
    'HML': hml,
    'MOM': mom,
    'SMB': smb,
    'RMW': rmw
})

# Create prediction target: NEXT month's market return
data['MKT_next'] = data['MKT'].shift(-1)

# Show structure
print("=== Prediction Data Structure ===\n")
print("Columns: Date, Factor returns (current month), Target (next month's MKT)")
print(f"Sample period: {dates[0].strftime('%Y-%m')} to {dates[-1].strftime('%Y-%m')}")
print(f"Total observations: {len(data):,}\n")

print("First 5 rows (note: MKT_next is shifted forward):")
print(data.head().round(4).to_string(index=False))

print("\n📊 Key insight:")
print("   At time t, we use factors (HML_t, MOM_t, ...) to predict MKT_{t+1}")
print("   The shift operation creates proper temporal alignment")
print("   Last row will have NaN for MKT_next (no future data available)")

# Show correlations among predictors
print("\n=== Predictor Correlations (Multicollinearity Check) ===\n")
predictors = ['MKT', 'HML', 'MOM', 'SMB', 'RMW']
corr_matrix = data[predictors].corr()
print(corr_matrix.round(3))
print("\n⚠️  Non-zero correlations cause multicollinearity")

=== Prediction Data Structure ===

Columns: Date, Factor returns (current month), Target (next month's MKT)
Sample period: 1994-01 to 2023-12
Total observations: 360

First 5 rows (note: MKT_next is shifted forward):
      date     MKT    HML     MOM     SMB     RMW  MKT_next
1994-01-31  0.0304 0.0247  0.0153  0.0006  0.0187    0.0018
1994-02-28  0.0018 0.0493 -0.0626 -0.0079  0.0314    0.0371
1994-03-31  0.0371 0.0072 -0.0516  0.0035  0.0073    0.0765
1994-04-30  0.0765 0.0304  0.0281  0.0873 -0.0199   -0.0025
1994-05-31 -0.0025 0.0232  0.0131  0.0127  0.0133   -0.0025

📊 Key insight:
   At time t, we use factors (HML_t, MOM_t, ...) to predict MKT_{t+1}
   The shift operation creates proper temporal alignment
   Last row will have NaN for MKT_next (no future data available)

=== Predictor Correlations (Multicollinearity Check) ===

       MKT    HML    MOM    SMB    RMW
MKT  1.000  0.193 -0.169  0.349  0.014
HML  0.193  1.000 -0.050  0.124  0.250
MOM -0.169 -0.050  1.000 -0.012  0.051
SMB  0.349  0.124 -0.012  1.000  0.002
RMW  0.014  0.250  0.051  0.002  1.000

⚠️  Non-zero correlations cause multicollinearity

Data Structure for Prediction

At each date t, we observe this month’s factor returns. We want to predict next month’s market return. The key: predictors must be lagged: only past/current information, never future.

In-Sample vs Out-of-Sample Performance

The cardinal sin of prediction modelling: testing on the same data used to train the model. This guarantees overfitting and spurious significance.

In-sample (training data):

• Model fits the data used to estimate parameters
• Performance always looks good (model memorizes patterns, including noise)
• Overstates true predictive power

Out-of-sample (test data):

• Model predicts data it has never seen
• Performance reveals true forecasting ability
• Often much worse than in-sample (overfitting is exposed)

Why In-Sample R² Is Misleading

With enough parameters, any model can achieve R² = 100% in-sample: even on pure noise. This is the fundamental problem of overfitting.

The polynomial example:

Key insight: The 10th-degree polynomial fits 10 data points perfectly: but it memorised noise. Out-of-sample, it performs worse than predicting the mean (negative R²).

The Prediction Challenge: Signal vs Noise

Market returns are dominated by noise, making prediction extremely difficult. But how do we measure signal vs noise rigorously?

Naive approach (intuitive but imprecise):

• Mean return / Std dev = 0.8% / 4.0% = 0.2
• Problem: treats unconditional mean as “signal,” but mean is constant: not predictable variation

Rigorous approach (from Week 10):

• Signal = \(\text{Var}(E[r_t | \mathcal{I}_{t-1}])\) : variance of conditional expectation
• Noise = \(\text{Var}(r_t - E[r_t | \mathcal{I}_{t-1}])\) : variance of residuals
  
• Signal Fraction = R² of the prediction model

Implication: R² OOS directly measures what fraction of return variance is predictable. If R² OOS = 3%, only 3% is signal; 97% is noise.

Walk-Forward Validation: Visual Demonstration

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle

# Walk-forward illustration: Rolling 10-year window, predict 1 year ahead
fig, ax = plt.subplots(figsize=(14, 6))

# Show consecutive rolling windows (the actual process)
window_size = 10
examples = [
    (2011, 2001, 2010),  # Step 1: Train 2001-2010, predict 2011
    (2012, 2002, 2011),  # Step 2: Train 2002-2011, predict 2012
    (2013, 2003, 2012),  # Step 3: Train 2003-2012, predict 2013
    (2014, 2004, 2013),  # Step 4: Train 2004-2013, predict 2014
    (2015, 2005, 2014),  # Step 5: Train 2005-2014, predict 2015
]

colors = plt.cm.Blues(np.linspace(0.3, 0.7, len(examples)))
for idx, (forecast_year, train_start, train_end) in enumerate(examples):
    y_pos = (len(examples) - 1 - idx) * 1.2
    # Training window
    train_width = train_end - train_start + 1
    ax.add_patch(Rectangle((train_start, y_pos), train_width - 0.1, 0.8, 
                            facecolor=colors[idx], edgecolor='black', linewidth=1.5))
    ax.text(train_start + train_width/2 - 0.5, y_pos + 0.4, f'Train: {train_start}-{train_end}',
            ha='center', va='center', fontsize=9, fontweight='bold')
    
    # Forecast point (one year ahead)
    ax.scatter([forecast_year], [y_pos + 0.4], s=150, c='red', marker='*', 
               edgecolors='darkred', linewidths=1, zorder=10)
    ax.annotate(f'{forecast_year}', (forecast_year, y_pos + 0.4),
                xytext=(forecast_year + 0.8, y_pos + 0.4),
                fontsize=8, fontweight='bold', color='red',
                va='center')

ax.set_xlim(1999, 2017)
ax.set_ylim(-0.5, 6.5)
ax.set_xlabel('Year', fontsize=11, fontweight='bold')
ax.set_title('Walk-Forward Validation: Rolling 10-Year Window (Predict 1 Year Ahead)', 
             fontsize=12, fontweight='bold', pad=15)
ax.set_yticks([])
ax.grid(axis='x', alpha=0.3, linestyle='--')

# Add timeline
for year in range(2000, 2017, 2):
    ax.axvline(x=year, color='gray', alpha=0.2, linestyle=':')
    ax.text(year, -0.3, str(year), ha='center', fontsize=8)

# Add annotation showing the "roll"
ax.annotate('Window rolls forward\none year at a time', 
            xy=(2006, 3), fontsize=9, style='italic',
            bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))

plt.tight_layout()
# Note: plt.show() not needed - Quarto captures figures automatically

What this shows:

• Each forecast uses only past data (training window before forecast date)
• Window “walks forward” through time as new data arrives
• No overlap between training and forecast: prevents look-ahead bias
• Process repeats hundreds of times to generate out-of-sample predictions

The Multicollinearity Problem

When predictors are correlated (which factors always are), OLS parameter estimates become unstable. Small data changes cause large parameter swings, leading to poor out-of-sample forecasting.

Multicollinearity in factor prediction:

• Value (HML) and profitability (RMW) correlate ~0.5
• Momentum (MOM) and market (MKT) correlate ~0.3
• When predictors correlate, OLS can’t reliably separate their individual effects

Consequences:

• Parameter estimates have high variance (large standard errors)
• Small changes in training data → large parameter changes
• Model overfits training noise → poor out-of-sample performance

Ridge Regression: Penalizing Complexity

Ridge regression adds penalty for large coefficients, forcing model to distribute weight across predictors rather than overfitting to training noise. This reduces variance at cost of small bias.

Ridge objective function:

\[ \min_{\beta} \sum_{i=1}^{n} (y_i - X_i \beta)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \]

• First term: OLS objective (minimize prediction error)
• Second term: Penalty for large coefficients (shrinks toward zero)
• λ (lambda): Regularization strength (controls bias-variance tradeoff)

How ridge helps:

• Forces coefficients toward zero → simpler model → less overfitting
• Stabilizes estimates when predictors correlate
• λ = 0 → OLS (no regularization); λ → ∞ → all coefficients = 0

OLS vs Ridge: What to Expect

Understanding when ridge improves over OLS helps interpret results and avoid overinterpreting random performance differences.

When ridge helps:

• High multicollinearity among predictors (factor correlations >0.3)
• Limited training data relative to number of predictors
• Strong overfitting tendency (large in-sample vs out-of-sample gap)

When OLS is competitive:

• Low multicollinearity (orthogonal factors)
• Large training sample relative to predictors
• True relationships are strong (high signal-to-noise)

Realistic expectations:

• Ridge typically improves R² OOS by 0.5-2 percentage points over OLS
• Don’t expect dramatic differences: both models use same weak signal
• Sometimes OLS wins (if λ choice was poor or relationships are stable)

Out-of-Sample R² (R² OOS)

R² OOS measures whether model forecasts better than naive benchmark (historical mean). This is the primary metric for prediction evaluation.

Definition:

\[ R^2_{OOS} = 1 - \frac{\sum_{t} (y_t - \hat{y}_t)^2}{\sum_{t} (y_t - \bar{y})^2} \]

• Numerator: Model’s prediction errors squared
• Denominator: Benchmark (mean) prediction errors squared
• R² OOS > 0: Model beats benchmark
• R² OOS < 0: Benchmark beats model (model is useless)

Interpretation:

• R² OOS = 5%: Model reduces prediction error by 5% vs naive mean
• R² OOS = 2-3% is meaningful in return prediction (signal is weak)
• R² OOS > 10% is suspiciously high (likely overfit)

Directional Accuracy

Directional accuracy measures how often model correctly predicts direction (positive or negative return), not magnitude. This matters for market timing strategies.

Definition:

\[ \text{Directional Accuracy} = \frac{\text{No. of correct sign predictions}}{\text{Total predictions}} \times 100\% \]

• Benchmark: 50% (random guessing, coin flip)
• Directional accuracy > 50%: Model has timing skill
• Target: 55-60% is meaningful; 70%+ is suspiciously high

Why directional accuracy matters:

• Market timing requires knowing direction (go long when forecast positive, short when negative)
• Magnitude matters less than sign for binary asset allocation decisions
• Easier to achieve than high R² (predicting sign is easier than magnitude)

Economic Value and Certainty Equivalent Returns

Statistical metrics (R² OOS, directional accuracy) don’t directly measure profitability. Certainty Equivalent Return (CER) quantifies portfolio value in economic terms.

CER framework:

• Allocate between risky asset (market) and risk-free asset
• Use model forecasts to adjust allocation dynamically
• CER measures how much risk-free return you’d accept instead of strategy
• CER difference between strategy and benchmark quantifies economic value

Typical CER gains from prediction:

• Naive benchmark (historical mean): CER ~4% annualized
• Ridge model: CER ~4.5-5% annualized
• CER gain of 0.5-1% is meaningful (compounds significantly over years)

Signs of Overfitting

Overfitting is the cardinal sin of prediction modelling. Recognizing it is essential for critical analysis.

Red flags for overfitting:

• Large in-sample vs OOS gap: In-sample R² = 10%, OOS R² = 1% → 9pp gap indicates overfitting
• Suspiciously high R² OOS: R² OOS > 10% is unlikely given weak signal in returns
• Unstable coefficients: Coefficients flip signs across training windows
• Deteriorating performance over time: R² OOS declines in later test periods (arbitrage?)
• Many predictors relative to sample size: 20 factors, 120 training months → overfitting risk

How to guard against overfitting:

• Use regularization (ridge) to penalize complexity
• Minimize number of predictors (≤ 10 factors)
• Honest out-of-sample testing with walk-forward validation
• Report both in-sample and OOS performance (transparency)

Critical Analysis: Interpretation Principles

Strong interpretation asks questions and acknowledges limitations. Weak interpretation just reports numbers. This is where 35% of marks come from.

Questions to ask when interpreting prediction results:

• Is R² OOS positive? If yes, model beats benchmark. If no, model is useless.
• Is R² OOS realistic? 2-3% is meaningful; 10%+ is suspicious.
• How does ridge compare to OLS? Is improvement meaningful or within sampling error?
• Is directional accuracy >50%? Is it statistically significant?
• Are results stable over time? Does R² OOS decline in later periods?
• What about transaction costs? Would net-of-cost returns be profitable?

Coursework 2 Option B: Prediction Pathway

The scaffold notebook implements walk-forward validation, OLS vs ridge comparison, and evaluation metrics. Your report provides understanding and critical analysis.

Scaffold provides:

• Pre-written code for walk-forward validation (rolling 120-month window)
• OLS and ridge regression with cross-validated λ selection
• R² OOS, directional accuracy, rolling performance metrics
• Publication-quality tables and figures

You must provide:

• Understanding: Why walk-forward prevents look-ahead, when ridge helps
• Interpretation: What does R² OOS = 2% mean economically?
• Model comparison: Why did ridge beat/lose to OLS?
• Limitations: Sample size, cost omissions, instability concerns
• Investment implications: Would you recommend using this model?

Note: The scaffold handles technical execution; your report focuses on critical analysis and interpretation.

Key Principles for Coursework 2 Option B

Today’s principles guide your prediction analysis. Focus on understanding that enables critical interpretation, not mechanical execution.

Methodological principles:

• Walk-forward validation prevents look-ahead bias (essential for honest testing)
• Out-of-sample evaluation reveals true forecasting ability (in-sample R² is misleading)
• Honest testing often shows weaker performance than expected (overfitting is pervasive)

Statistical principles:

• Ridge reduces overfitting when predictors correlate (typical with factors)
• R² OOS = 2-3% is meaningful for monthly returns (signal is weak)
• Directional accuracy >55% indicates timing skill (50% is random guessing)

Critical thinking principles:

• Large in-sample vs OOS gap reveals overfitting (be skeptical of high in-sample R²)
• Transaction costs can eliminate apparent profitability (discuss net-of-cost returns)
• Temporal instability suggests declining predictability (arbitrage or regime change)