Delete the future, diff the features: “no leakage” as a test, not a slogan

The most instructive number in RealPath's history is one I can't use. On a public relational benchmark, the driver-dnf task of RelBench's rel-f1 dataset, the experimental graph neural network backend scored a ROC-AUC of 0.76. That's above the roughly 0.72 the project's benchmark notes cite for RelBench's own reference model.

It was also leaky. The GNN's time-aware neighbour sampling needs pyg-lib, which has no Windows build. So on my machine it sampled neighbours without looking at their timestamps, and a prediction made as of one date could draw on rows from after it. The handover notes file the number under "LEAKY, adil değil": leaky, not fair.¹

That's what leakage looks like from the outside. Not an error. A number you're happy to see.

No test flagged it. The GNN path has none, and the number was marked only because the fallback was known to be leaky (ADR-011). RealPath's default pipeline does have one. It turns "no leakage" into a pytest: delete every row after the prediction date, rebuild the features, diff. This post walks through that test, what it caught when I broke the code on purpose, and the two leaks that still get past it.

RealPath is a local-first prediction engine for relational databases. Point it at DuckDB (or Postgres and MySQL through optional connectors) and ask something like "which customers won't buy anything in the next 30 days?". It compiles the question into a predictive task, builds features across the foreign keys with Featuretools, trains LightGBM and explains each score by its join path. Its decision log calls leakage the most insidious error in relational prediction. ADR-005 adds, in my translation from the Turkish, that "no leakage" shouldn't be a slogan but an auditable guarantee.

In a single flat table, leakage is usually a column you forgot to drop. In a relational database it's a join. To predict whether a customer goes quiet in the next 30 days, you want their history: spend, frequency, returns. Each of those is an aggregation over a related table, like SUM(transactions.amount) or MEAN(returns.transactions.amount). Compute one over all transactions and it quietly includes the next 30 days, the same window the label is measured on. The model trains on the answer key and looks brilliant until it meets the future it was supposed to predict.

Nothing about that is exotic. It's the default: SELECT SUM(amount) FROM transactions GROUP BY customer_id has no idea what date you're predicting from. So I didn't want "no leakage" to be a convention every query has to remember. It had to be structural, and it had to be checkable.

A RealPath question, written in its predictive query language (PQL), looks like this:

PREDICT COUNT(transactions.*, 0, 30, days) == 0
FOR EACH customers.customer_id

For each customer, count their transactions from 0 to 30 days after some moment, and predict whether that count is zero. The query deliberately doesn't say which moment. That moment is the anchor, and the compiler owns it. pql/compile.py turns the query plus an anchor into one SQL statement. For the churn question at the anchor 2025-03-01, it sends this (parameters inlined, quoting and ordering trimmed, comments mine):

WITH universe AS (         -- customers that exist at the anchor
  SELECT customers.customer_id AS _eid
  FROM customers
  WHERE customers.signup_date <= '2025-03-01'
),
events AS (                -- their purchases in (anchor, anchor + 30 days]
  SELECT customers.customer_id AS _eid, COUNT(*) AS _val
  FROM customers
  JOIN transactions ON customers.customer_id = transactions.customer_id
  WHERE transactions.tx_time >  '2025-03-01'
    AND transactions.tx_time <= '2025-03-31'
  GROUP BY customers.customer_id
)
SELECT u._eid AS entity_id, COALESCE(e._val, 0) AS agg_value
FROM universe u
LEFT JOIN events e ON u._eid = e._eid

Two filters do the work. The universe keeps customers whose own timestamp, signup_date, is at or before the anchor, so next month's sign-ups can't be part of this month's question. The events are target rows in (anchor + start, anchor + end]: here, after the anchor and at most 30 days later. Customers with no events get a zero through the LEFT JOIN and COALESCE, and for churn that zero is the positive class.

The features come from the other side of the same timestamp. build_split also returns a cutoff table, one row per customer stamped with the anchor. features.py hands it to Featuretools' Deep Feature Synthesis (DFS) as cutoff_time, and DFS aggregates only rows at or before it. Features see t ≤ anchor, labels see anchor < t ≤ anchor + 30 days, and the two windows never overlap. A transaction stamped exactly at the anchor counts as history.

The compiler also picks the anchors. It reads the target table's time range and puts the test anchor one horizon (30 days here) before the last timestamp, then the train anchor one horizon before that. The model learns from one 30-day window and is graded on the next. On the sample database that's 2025-04-29 and 2025-05-29, and realpath predict prints both.

ADR-005 says each entity gets an anchor. In practice a split stamps every row with the same one. Featuretools would accept a time per row, and so far only RealPath's RelBench adapter uses that, with each task row's own timestamp.

The idea behind tests/test_leakage.py fits in a sentence. If the features at an anchor really depend only on rows at or before the anchor, deleting every row after the anchor can't change them. So the test deletes them.

def test_no_future_leakage(sample_db, tmp_path):
    anchor = pd.Timestamp("2025-03-01")
 
    X_full, _ = _build_split_and_features(sample_db, PQL, anchor)
 
    trunc_path = tmp_path / "trunc.duckdb"
    _truncated_db(sample_db, anchor, trunc_path)
    X_trunc, _ = _build_split_and_features(str(trunc_path), PQL, anchor)
 
    common_idx = X_full.index.intersection(X_trunc.index)
    common_cols = X_full.columns.intersection(X_trunc.columns)
    assert len(common_idx) > 100 and len(common_cols) > 5
 
    a = X_full.loc[common_idx, common_cols]
    b = X_trunc.loc[common_idx, common_cols]
    # identical => no feature depended on post-anchor rows
    mismatches = []
    for col in common_cols:
        if not _series_equal(a[col], b[col]):
            mismatches.append(col)
    assert not mismatches, f"Future data leaked into features: {mismatches}"

It builds the churn features on the synthetic sample database at 2025-03-01. Then it copies the database without any row whose time index is later than the anchor and builds the same features on the copy. It compares the two matrices on the rows and columns they share. Numeric columns must agree within float noise (np.allclose with a relative tolerance of 1e-6). Everything else must match as strings.

The docstring promises "byte-for-byte identical", which is stronger than the assertion. I think the assertion is right: floating-point aggregations owe you no bit-stability. The > 100 and > 5 guard makes sure a comparison of nothing can't pass.

What I like most is what the test doesn't know: what DFS is, how cutoff_time works, which primitives exist. If a new primitive, a deeper join path or a library upgrade starts reading tomorrow's rows, the mismatch list names the columns anyway. The project's design spec (v2) promised something grander, an auditor tracing every feature back to its source rows. It was never built. What got built is dumber, and I think better: a black-box check on behaviour.

A test that can't fail proves nothing, so I tried to make this one fail. In a throwaway copy of the repository, I made one-line edits to the code under test and ran pytest tests/test_leakage.py after each. None of them is committed, and each is small enough to redo by hand:

Removing the cutoff let DFS aggregate the whole database. Every aggregate over transactions and returns changed. Only the customer's own columns still matched: country, segment, and the month and weekday of the signup date. Moving the cutoff to the end of the label window, the classic off-by-a-window bug, failed with the same list. Then the smallest edit, one day late:

-        cutoff_time=cutoff,
+        cutoff_time=cutoff.assign(time=cutoff["time"] + pd.Timedelta(days=1)),

The failure, wrapped and trimmed:

E       AssertionError: Future data leaked into features:
        ['MAX(transactions.amount)', 'MAX(transactions.quantity)',
         'MEAN(transactions.amount)', …, 'MAX(returns.transactions.amount)',
         'MEAN(returns.transactions.amount)', …]
FAILED tests/test_leakage.py::test_no_future_leakage - AssertionError: Future...
1 failed, 1 passed

One day of future rows was enough to change the matrix. That's the result that made me trust the test.

A leakage test you've never seen fail isn't evidence. It's decoration.

The fourth edit passed. I switched off the universe filter, the signup_date <= anchor condition that keeps later sign-ups out of the question, and both tests stayed green.

Customers who signed up after the anchor now appeared in the full database's feature matrix. They don't exist in the truncated copy, and the comparison runs over X_full.index.intersection(X_trunc.index). The intersection quietly dropped exactly the rows that were wrong. The > 100 guard stops a comparison of nothing, but not a comparison of less.

That's a population leak: training on customers who don't exist yet at the anchor. The fix is two assertions before any value is compared. They aren't in the repository yet:

assert X_full.index.equals(X_trunc.index), "entities differ: population leak"
assert X_full.columns.equals(X_trunc.columns), "feature columns differ"

Both hold on the shipped code. With the universe filter switched off, the first one fails.

The second gap has nothing to do with feature code. It's the question.

The grammar accepts negative window bounds (the parser's pattern for them is -?\d+), and nothing downstream rejects them. So I wrote a query by hand that looks backwards:

PREDICT COUNT(transactions.*, -30, 0, days) == 0
FOR EACH customers.customer_id

That reads as "customers with no purchases in the 30 days before the anchor". It's something you can look up at prediction time, not a forecast. It compiled, trained and reported a perfect score. This is what realpath predict printed for it, trimmed:

[realpath] anchors: train=2025-06-28 test=2025-06-28
metrics: roc_auc=1.0000  accuracy=1.0000

The perfect score comes from a second bug. A window that ends at the anchor gets a zero-day horizon (horizon_days takes abs(end)). So both anchors land on the same date, and the model is graded on the rows it trained on. Same symptom as the GNN in the opening: no error, just a number you're happy to see. At the churn query's two anchors, the backwards question is far from perfect. It's still a question about the past.

The truncation test stays green on this query, because the features really are clean. The file's second test, test_label_window_is_in_the_future, asserts the missing property and fails when I swap this query into its PQL constant. But it only ever runs against its own hard-coded question. The check lives in the test suite, not in the compiler that every real query passes through.

That matters because most questions won't arrive as PQL. RealPath also takes English and Turkish (ADR-007). The offline templates only write forward windows, but with an API key Claude writes the PQL and the parser checks it.

The model never writes a join or picks a timestamp. The anchor, the universe and the cutoff belong to the compiler, and I'd still defend that split. But the model does choose the window. Its prompt shows only forward-looking examples and never says a window can't start below zero, and re-parsing checks syntax, not tense. "The last 30 days" is how a backwards window would arrive, and nothing after the parser would stop it.

One check in the compiler would close both holes, the backwards window and the zero horizon behind the perfect score:

# pql/compile.py, CompiledTask._validate(): a proposal, not in the repo
w = t.target.window
if not 0 <= w.start < w.end:
    raise PQLCompileError(
        "The label window must start at or after the anchor and end after it."
    )

In a scratch copy, the full suite still passes with it, and the backwards query stops at compile time.

The feature names the test diffs are also what RealPath's explanations are built from (ADR-006). A DFS name is a join path with an aggregation on top. MEAN(returns.transactions.amount) means: over this customer's returns, take the amount of the transaction each return belongs to, and average it. explain.provenance() is two regular expressions that split the name back into tables and an aggregation, and format_card() prints that as one line of a per-customer card.

Take the last line of the card for the customer the quickstart notebook scores highest: returns -> transactions: MEAN = 148.19. In words, the purchases this customer had returned by the anchor averaged 148.19. That's the literal computation behind one of the model's inputs, and it's one of the columns the one-day edit broke. The explanation inherits whatever the test proves.

One caveat on that card: its contributions are SHAP values because the notebook's environment had shap installed (the optional explain extra). Without it, the card falls back to the model's global gain, and nothing on the card says which kind you're looking at.

The top line is the other lesson. The churn model's strongest driver by gain is MONTH(signup_date), yet nothing in the generator reads the calendar month. It draws each churner's last active day relative to their signup date, so tenure shifts the odds that someone has gone quiet by the anchor.

The primitive set has month and weekday but nothing like "days since signup". My best explanation is that the trees grabbed the closest handle on offer: a calendar field that can't tell January 2024 from January 2025. Gain rank isn't a causal ranking, though, and a 12-level categorical tends to collect gain.

A leakage test proves the model only saw the past. It can't prove the model learned anything real about it, and the notebook's ROC-AUC of 0.7492 on synthetic data would never have shown me that.

The two gaps above aside:

• It sees deleted rows, not updated ones. Truncation removes rows by time index, and a table without one (products in the sample) is copied whole. In a real database, a price or a segment overwritten in place is today's value dressed up as history. No amount of deleting will reveal it. That takes snapshots or history tables.
• It trusts the time index it tests. Truncation and cutoff both use the first timestamp column RealPath infers for each table (ADR-010). If that guess is wrong, both sides share the mistake and the diff can't see it.
• It doesn't look at the split. The test pins one anchor. Nothing checks that training and scoring use different ones, which is how the backwards query graded itself.
• It proves one case. One anchor, one query, one synthetic database. Parametrising it over the templates and a few anchors would be cheap.
• It runs where pytest runs. ADR-005 counts on CI to prove the property, but GitHub Actions has never run on this account (ADR-015). The gate is a local run.

And the GNN from the opening is still waiting for a fair run. ADR-011 labels every comparison temporal or leaky, so its 0.76 stays out of the benchmark table. A local Linux container filled the disk. The next plan was a CI workflow, written but never enabled, on an account where Actions doesn't run. The decision log and the handover now mark the fair number PENDING, while the benchmarks page still points at CI.

The pattern isn't specific to Featuretools or to RealPath. Any pipeline that builds features from timestamped rows can run the same audit.

RealPath is on GitHub under MIT: Ozgurisikdamar/RealPath. It isn't published on PyPI, so install it from a clone:

git clone https://github.com/Ozgurisikdamar/RealPath.git && cd RealPath
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pip install "setuptools<81"
python -m pytest tests/test_leakage.py -q

On Python 3.12+ or in a uv-built environment, the venv has no setuptools, and woodwork (a Featuretools dependency) still imports pkg_resources. Hence the pin, which is harmless on 3.10 and 3.11. The edits from the table are one-liners in realpath/features.py and realpath/pql/compile.py. The backwards query runs as-is:

realpath make-sample
realpath predict "PREDICT COUNT(transactions.*, -30, 0, days) == 0
  FOR EACH customers.customer_id" --db data/shop.duckdb

"No leakage" is a claim about the future, so test it by deleting the future. Then break your own cutoff until the test goes red. Compare the whole matrices, not their overlap. And put the direction check where every real question passes, not only in the one your test happens to ask. Until then, "no leakage" is still a slogan. It just comes with a green check mark.