7.1 Point-in-Time Correctness and Feature Leakage: Use Information Available at Prediction Time

A regression coefficient can change with units, omitted variables and the sampled population. Feature engineering adds another question: was each input actually available when the prediction was needed? A precisely encoded future value is still a future value.

The first step in feature engineering is not creating new columns; it’s defining the prediction context. Only when time, entity, and label are properly aligned can subsequent encoding and modeling make sense.

Learning Objectives

• Define the prediction time, observation window, and label window;
• Identify result leakage, future leakage, split leakage, and proxy leakage;
• Use point-in-time joins to construct reproducible historical features;
• Assign timestamp, refresh interval, and default behavior to feature records.

1. Write the Prediction Contract First

A supervised learning sample must answer at least four questions:

• Who is the entity? Is it a service, deployment, host, or job?
• What is the prediction time? For example, 30 minutes before a planned deployment.
• Up to what historical point is visibility available? Which records have been generated and arrived in the system by that time?
• When does the label occur? For example, did a service incident occur within 24 hours after deployment?

The $i$th sample can be expressed as:

$$X_i=f(\text{history at or before }{t}_i),y_i=g(\text{events in }(u_i,u_i+h]).$$

Here $t_i$ is the prediction cutoff, $u_i$ is the start of the outcome window, and h is its length. For a prediction 30 minutes before deployment with outcomes measured for 24 hours afterward, $t_i=u_i-30\text{ min}$ and $h=24\text{ h}$. The label may legitimately use future outcomes; the features may not. A negative label also requires that its full outcome window and reporting delay have elapsed.

2. Event Time Does Not Equal Available Time

Events are often misunderstood due to two distinct clocks:

• event_time: When the event actually occurs in reality;
• available_time: When the event is recorded and becomes accessible within the prediction system.

A field report might occur at 9 a.m. but only sync at 2 p.m. Any predictions made at noon cannot use that data. Filtering by event_time <= cutoff_time alone still risks backfill leakage.

Availability means readable by the prediction path, not merely written to an upstream raw table. Queueing, feature computation and publication can add delay. Corrections need versioned availability timestamps too; overwriting history with the latest corrected value makes an as-of replay impossible.

A reliable feature definition must explicitly specify:

Entity key: team_id
Prediction timestamp: departure_time - 30min
Event condition: event_time <= cutoff_time
Availability condition: available_time <= cutoff_time
Observation window: [cutoff_time - 30d, cutoff_time]
Aggregation: median resource consumption of completed tasks
Default value: missing if no historical data exists; do not use global future median

3. Five Common Types of Leakage

Result Leakage

Putting the outcome or a direct derivative of it into a feature, such as using "final consumption volume" to predict whether resupply will be insufficient.

Future Leakage

Using full-month aggregates to predict events that occur mid-month, or relying on post-task retrospection fields to predict risks before the task begins.

Preprocessing Leakage

Fitting imputation values, scaling parameters, category vocabularies, or feature selectors on the entire dataset before splitting the training and validation sets.

Proxy Leakage

A field may not look like the label but can still be generated by the outcome. For example, a ticket number assigned only after escalation nearly confirms that a severe incident has already occurred.

Entity Leakage

Multiple rows from the same task appear across both the training and validation sets, causing the model to memorize the task rather than learn generalizable patterns.

Leakage cannot be detected solely by correlation analysis. It requires a thorough review of field origins, timing of generation, and the underlying business workflow.

4. Point-in-time join

For this snapshot feature, select the most recent event-time state among versions that were already available. For equal event times, select the latest visible correction, then use a unique snapshot ID to break remaining ties. Sorting only by arrival time would let a late older snapshot replace a newer state. This standalone DuckDB 1.5 query uses UTC timestamp values:

WITH prediction_samples(sample_id, warehouse_id, cutoff_time) AS (
  VALUES ('p1','W1',TIMESTAMP '2026-08-01 12:00:00'),
         ('p2','W1',TIMESTAMP '2026-08-01 09:05:00'),
         ('p3','W2',TIMESTAMP '2026-08-01 12:00:00')
),
inventory_snapshots(snapshot_id, warehouse_id, event_time, available_time, inventory_level) AS (
  VALUES
    ('s1','W1',TIMESTAMP '2026-08-01 09:00:00',TIMESTAMP '2026-08-01 09:10:00',5),
    ('s2','W1',TIMESTAMP '2026-08-01 08:00:00',TIMESTAMP '2026-08-01 11:00:00',100),
    ('s3','W1',TIMESTAMP '2026-08-01 10:00:00',TIMESTAMP '2026-08-01 10:05:00',7),
    ('s4','W1',TIMESTAMP '2026-08-01 10:00:00',TIMESTAMP '2026-08-01 10:15:00',8),
    ('s5','W1',TIMESTAMP '2026-08-01 11:00:00',TIMESTAMP '2026-08-01 14:00:00',0)
),
candidates AS (
  SELECT p.sample_id, s.inventory_level,
    ROW_NUMBER() OVER (
      PARTITION BY p.sample_id
      ORDER BY s.event_time DESC, s.available_time DESC, s.snapshot_id DESC
    ) AS rn
  FROM prediction_samples AS p
  LEFT JOIN inventory_snapshots AS s
    ON s.warehouse_id = p.warehouse_id
   AND s.event_time <= p.cutoff_time
   AND s.available_time <= p.cutoff_time
)
SELECT sample_id, inventory_level
FROM candidates WHERE rn = 1
ORDER BY sample_id;

The actual system must also specify a deterministic ordering for records at the same timestamp, a strategy for handling late-arriving data, timezone handling, and whether a snapshot represents the state before or after an update.

The result is p1=8, p2=NULL and p3=NULL. At noon, s5 is still unavailable; s2 arrived later than s4 but describes an older state. p2 has no available snapshot yet, and p3 has no matching warehouse. IDs and timestamps must be non-null and their ordering must reflect the declared revision policy. Add a maximum snapshot age if stale inventory is unacceptable.

The Feast point-in-time documentation distinguishes event-time retrieval from created-time filtering. Verify the installed version, store support and meaning of the availability field before relying on a feature-store API to enforce both conditions.

5. Window Features Must Close the Right Boundary

"Number of alerts in the last 7 days" is not a complete definition. It requires specification of:

• Whether the left and right boundaries are inclusive;
• Whether the period is based on natural days or continuous 168-hour intervals;
• Whether event occurrence time or availability time is used;
• How revisions and cancellations of the same alert are handled;
• Whether to return zero or indicate missing data when no historical records exist.

For “days since the last confirmed supply,” find the latest event time among records already available at the cutoff, then subtract it from the cutoff. A shift over arrival-sorted rows only finds the previous arrival; an older late event may be next in that order. Neither groupby().shift() nor sorting alone replaces the two-clock condition.

6. Splitting for Simulation of Deployment

If the deployment goal is to predict future tasks, training must precede validation; if the goal is to generalize to new teams, hold out complete teams from training for validation. Both temporal and entity-based conditions may coexist, requiring a combined splitting strategy.

In time-series cross-validation, gap separates the training and validation windows, but it does not automatically correct for flawed features. Feature queries must still be bounded by each sample's own cutoff point.

7. Feature Specification

Every feature entering the model must be documented with at least the following:

Name and business meaning
Entity key, unit, and data type
Source table and owner
Event time, availability time, and observation window
Computation logic and version
Policies for missing, late, and unknown values
Refresh frequency and maximum acceptable latency
Allowed prediction scenarios

The value of a specification lies not in the volume of documentation, but in enabling training, replay, and online prediction to answer the same question.

Common Misconceptions

• "The field exists in the training table, so it's safe to use": The training table might be derived from a post-hoc wide table, introducing hidden dependencies.
• "Removing only the label column eliminates leakage": Proxy fields for labels or post-processing fields can still introduce leakage.
• "Filtering by event time is sufficient": Late data must also be checked against availability time windows.
• "Computing global aggregates first and then splitting is more efficient": Global statistics may have already seen the validation period, leading to data leakage.

Exercise

1. Define the entity, cutoff time, observation window, and label window for “predicting a service incident 24 hours in advance.”
2. Review a post-hoc wide table and identify five fields that are unavailable at prediction time, explaining why each is problematic.
3. Write a point-in-time join for the most recent inventory snapshot and verify that late-arriving records do not enter the historical sample.
4. Compare validation scores obtained from random splitting versus time- and team-based joint splitting.

Summary

Features begin as a contract about time and visibility, rather than as a list of numbers alone. To determine which historical data is eligible for inclusion in the sample, and which high-correlation fields merely peeked at the answer, we must first clearly define the prediction context.

The next lesson covers how to represent valid historical data: scaling, categorical encoding, and time transformations are all model parameters that must be learned from the training data.