Inside the MLB Edge Engine: how the pipeline works, stage by stage

1. What the system does

Once a day, one command does the whole loop: settle yesterday's picks against final scores, grade them against closing lines, pull fresh data, rebuild features, retrain through yesterday, score today's slate, scan for cross-book value, and write an HTML report. Every stage is also its own CLI command, so any of it can be run in isolation.

The architecture is shaped by three constraints. Everything must be free, which rules out paid odds and stats feeds and forces the ingestion layer to be polite toward public endpoints. Everything must be point-in-time correct, because the fastest way to build a "profitable" model is to let it see the future. Every claimed edge must be measurable after the fact, which is why the ledger exists.

I'll walk each stage in order, then trace a single real game through all of them.

Ten source clients, all subclassing a common SourceClient that supplies the HTTP client factory, retry policy, and cache contract.

client	provides
MlbApiSource	schedule, probable pitchers, lineups, officials
MlbPlayerStatsSource	season-to-date batting and pitching lines
MlbTransactionsSource	roster moves, IL placements and activations
MlbVenueClient	park metadata and coordinates
StatcastSource	pitch-level data via pybaseball
RetrosheetGameLogsSource	historical game logs for backfill
OpenMeteoSource	hourly weather at park coordinates
UmpireScorecardsSource	umpire run-impact tendencies
OddsApiSource	moneyline, run line, total snapshots
OddsMultiBookSource	per-book prices for shopping and steam

Raw pulls are written as immutable, timestamped parquet:

data/raw/mlb_player_stats/2026/07/31/2025_2026-07-24_2026-07-30.parquet
data/raw/statcast/2026/07/31/2026-07-24_2026-07-30.parquet
data/raw/odds_multibook/2026/07/31/...

Nothing is ever fetched twice, and any feature can be rebuilt from scratch without touching the network — which matters more than it sounds, because a feature bug becomes a re-run rather than a re-download of three seasons.

The unglamorous parts took the longest. A multi-hour backfill over public APIs will fail somewhere, so fetchers retry with backoff and resume via --skip-existing. Statcast in particular degrades from parallel to serial to day-by-day when the upstream CSV comes back malformed on long ranges — a fallback chain that exists because the naive version failed reliably around hour two.

Three sources will spell the same team three ways, and one of them will still say Indians. A registry maps every alias to a canonical key and abbreviation:

TeamInfo(114, "Cleveland Guardians", "CLE", "AL", "AL Central",
         "America/New_York", ("guardians", "cleveland indians", "indians"))

Normalization strips punctuation and case, so "NY Yankees", "New York Yankees" and "yankees" all resolve to the same key. Player identity reconciles MLBAM IDs against Retrosheet codes and free-text names. This is boring code that prevents a whole class of silent join failures where a team's features quietly become null for a season.

Feature tables are built in DuckDB, one table per family, joined into a per-game matrix of 264 columns (442 after categorical encoding).

table	rows	cols	grain
pitcher_features	84,210	50	pitcher × date
batter_features	—	45	batter × date
team_features	—	43	team × date
lineup_features	10,810	24	game
market_features	10,334	22	game
environment_features	10,890	18	game
catcher_features	10,810	15	game
bullpen_features	18,915	12	team × date
travel_features	21,780	11	team × date
injury_features	—	10	team × date
strength_features	4,410	10	team × week

Four families are worth describing in detail, because they're where the modelling thought actually went.

Pitcher form, by pitch family

A starter's ERA is nearly useless for prediction. What the model gets instead is arsenal-level: whiff rate, barrel rate, and called-strike-plus-whiff by pitch family, each in both a 30-day rolling window and an exponentially weighted version with a 7-day halflife, plus release speed, spin, extension, and perceived velocity. The EWM version matters because a pitcher who lost two ticks of velocity last week is a different pitcher than his 30-day average says.

The most informative single pitcher feature by SHAP is xfip_minus_fip_rolling_30d — the gap between a pitcher's home-run-normalized and actual fielding-independent numbers, which is a regression signal: a large gap says recent results are running ahead of or behind the underlying process.

Lineup versus arsenal

A batter's season xwOBA "against pitchers" is a weak signal. A batter's xwOBA against the specific pitch mix this starter actually throws is a better one. For each game, every batter's rates by pitch family are weighted by the opposing starter's observed arsenal distribution, then aggregated up the lineup — including slot-weighted versions, since the leadoff hitter gets ~4.6 plate appearances and the nine-hole ~3.8.

Lineups aren't always posted before first pitch, so lineup_features carries home_lineup_confirmed and home_confirmed_batters alongside the values. The model can tell "this lineup is weak" from "this lineup is a projection."

Opponent- and park-adjusted strength

Rolling run averages are badly confounded. Six runs a game against replacement-level arms in Coors is not six runs against contenders in Petco, but a rolling mean treats them identically. So instead of averaging runs, fit them:

Empirical park factors fall out of the same fit for free, regressed toward league average by the ridge penalty rather than a hand-tuned constant. They land where baseball says they should:

park	runs/game vs average
Coors Field	+1.11
Citizens Bank Park	+0.38
Fenway Park	+0.03
Petco Park	−0.30
Wrigley Field	−0.49
T-Mobile Park	−0.67

Bullpen, catcher, environment, travel

Bullpen availability is tiered from recent usage — high_leverage_ready, unavailable_arms, heavy_use_arms, pitches_last_3d_team — because a good bullpen that threw 60 pitches yesterday is not a good bullpen tonight.

Catcher framing is a real and underpriced effect: home_catcher_framing_rate_rolling_30 and its EWM twin both rank in the SHAP top 12. Environment carries umpire identity and run impact, temperature, humidity, and wind at game time. Travel carries miles, miles in the last 48 hours, timezone shift, crossings, and hours since the previous game.

Spring training, excluded by venue

The ingested schedule has no game-type column. It took me longer than I'd like to notice that 1,646 of 10,810 played games in the store were spring training — split-squad exhibitions where starters go three innings and half the roster is minor-league depth.

Detecting by date fails, because the opener moves and MLB now opens some seasons in Seoul and Tokyo. Detecting by venue works: every regular-season game is at one of the 30 clubs' home parks, while spring training runs out of the Florida and Arizona complexes. So the code learns the real park set from May–August — unambiguously regular season in every year — and flags anything played elsewhere. It correctly keeps Steinbrenner Field, which was the Rays' actual home in 2025.

Eleven tables at four different grains have to become one row per game containing only information that existed before first pitch. Every join is an as-of join: for a game on July 30, the pitcher's features are the most recent row strictly before July 30, not the row for July 30.

The leak-safety rules are mechanical:

• Training rows must have a final status and non-null scores; prediction rows must be unplayed.
• All joins are as-of joins against data timestamped before first pitch.
• Result-bearing fields — Retrosheet's columns, post-game ratings — are excluded from the model matrix entirely.
• A SHAP audit runs after every training pass and writes shap_alerts.json if anything ID-like or suspiciously dominant climbs the ranking.

The model does not predict a win probability from scratch. It predicts a correction to the market price:

Three layers produce that signal:

Base learners. XGBoost, LightGBM, and logistic regression train on the feature matrix with the raw price columns (market_home_prob_nv, market_home_ml, …) withheld — the line enters once, as the offset, rather than being double-counted. Derived market columns like spread lines and book counts stay, since they describe market context rather than restating the moneyline.

Stacking. A logistic meta-learner combines base predictions on time-series folds, so it only ever trains on out-of-fold predictions from the bases. Training on in-fold predictions is the classic stacking mistake — the meta-learner learns to trust whichever base overfit hardest.

The anchor. scikit-learn can't express a fixed offset, and the fixed offset is the entire design, so this is a direct two-parameter penalised solve:

def objective(params):
    bias, lam = float(params[0]), float(params[1])
    z = offset + bias + lam * signal
    loss = np.mean(np.logaddexp(0.0, z) - y * z)    # stable log loss
    return loss + l2 * (lam ** 2) / max(len(y), 1)  # prior toward the market

best = minimize(objective, x0=np.array([0.0, 0.0]), method="L-BFGS-B")

The critical property is what happens when the features are worthless: λ collapses toward zero and predictions degrade gracefully to the line rather than wandering below it. "At least as good as the market" becomes the floor of the design instead of something to hope for. The L2 term is a mild prior pulling λ toward zero — toward trusting the market.

Games with no usable price fall back to a league home prior of 0.535, with a per-row flag so the model can distinguish "the market says 50%" from "there is no market."

Calibration. Meta output goes through isotonic regression or Platt scaling, chosen by what the sample supports — isotonic overfits badly on small ones. Current selection is Platt. The resulting reliability:

bin	avg predicted	empirical	n	bias
0.35	0.372	0.337	392	+0.035
0.45	0.459	0.465	2,213	−0.006
0.55	0.546	0.540	3,724	+0.006
0.65	0.636	0.649	1,254	−0.013
0.75	0.712	0.824	51	−0.111

A separate April artifact is trained when there are enough rows and the predictor switches by month, since early-season baseball is a different regime — tiny samples, roster churn, cold weather. There are also DNN, LSTM, and PyMC Bayesian base models wired into the ensemble config but off by default; they haven't earned their compute.

Two simulation layers run independently of the classifier — deliberately, because they serve as a cross-check rather than as inputs. If two independent estimates of the same game disagree, that disagreement is information.

Monte Carlo. 50,000 games per matchup drawn from context-adjusted run distributions, with team run means and dispersions estimated separately.

The Markov chain is the more interesting one. It walks the 24 base-out states at plate-appearance granularity, vectorized across all sims at once. Each lineup gets an event-rate profile:

@dataclass(slots=True)
class TeamOffenseProfile:
    """Per-plate-appearance event rates. Everything not listed is an out."""
    bb_rate:     float = 0.085
    single_rate: float = 0.140
    double_rate: float = 0.043
    triple_rate: float = 0.003
    hr_rate:     float = 0.031

Those league-average baselines are then bent to match the feature-driven run expectation for that specific game via calibrate_profile_to_mu(), so park, umpire, bullpen fatigue and lineup context flow into the simulation automatically rather than being modelled twice. Runner advancement uses league-typical rates — P_SCORE_FROM_SECOND_ON_SINGLE = 0.60, P_SCORE_FROM_FIRST_ON_DOUBLE = 0.45.

Because it produces full event sequences rather than a single run total, one pass prices full-game totals, first-five-innings lines, team totals, and strikeout props (binomial over expected batters faced) — the markets where book attention, and therefore pricing quality, is thinnest.

A posted price is not a probability — it contains the book's margin. Both sides of a two-way market imply probabilities summing to more than 1, and that excess is the vig. Stripping it is the first step of every comparison.

Take a real game: Washington at Atlanta, July 30. Home −150, away +124.

Proportional is the simple method and the default for edge computation. The cross-book scanner uses Shin's method instead, which backs out an implied share of informed money z and removes it — a better model of how books actually set margin, because it puts proportionally more of the vig on the longshot, which is where the empirical bias lives. Power de-vigging is also available.

Edge is then simply:

This definition is the reason the anchor matters so much, and it's worth stating explicitly: if the model is less accurate than the market, then the games where it disagrees most are mechanically the games where it is most wrong. The edge filter doesn't select opportunities in that case — it selects model error, sorted descending.

Selection is a series of vetoes, not a score.

The agreement gate. If the classifier and the simulator disagree materially about the same game, the honest conclusion is "we don't know this game." Depending on the size of the gap the bet is either blocked outright or has its stake scaled down — the ledger records blocked_reason: disagreement_reduce for the latter.

Thresholds. edge_threshold_pct (3.0) gates staking; tracking_edge_threshold_pct (2.0) gates logging; tracking_min_model_prob (0.52) prevents logging picks the model doesn't actually favour.

Sizing starts from Kelly. For decimal odds b and win probability p, the optimal fraction is (bp − q)/b, taken at a quarter for variance reasons. But Kelly assumes p is exact, and it never is, so stakes are shrunk by the signal-to-noise of the edge itself:

Uncertainty about an edge is a reason to bet less, not to bet differently. On top of that, when a slate contains correlated bets — shared weather, same umpire crew, divisional dynamics — a portfolio Kelly optimizer maximizes expected log wealth over jointly sampled outcomes via a Gaussian copula, cutting total exposure rather than sizing each bet as though it were alone. Hard caps finish the job: 2% of bankroll per bet, 15% per day.

Here is an actual staked pick from the ledger — Washington at Atlanta, July 30, 2026 — with every intermediate value the pipeline produced.

game

Washington Nationals @ Atlanta Braves · game_id 824894

market prices

ATL −150  ·  WSH +124

no-vig market

p(WSH) = 0.426621  (overround 1.046429)

classifier + sim

blended p(ATL) = 0.466369

team bias adj

−0.029518 → p(ATL) = 0.436852

model

p(WSH) = 0.563148

edge

0.563148 − 0.426621 = +13.65 pts

full kelly

(1.24 × 0.563148 − 0.436852) / 1.24 = 0.210848

quarter kelly

0.052712 → $527.12 on a $10,000 bankroll

2% cap

→ $200.00

agreement gate

model_sim_gap 0.0492 → disagreement_reduce, ×0.6766

stake

$135.33

final score

ATL 5, WSH 4

result

loss — settled automatically the next morning

Every number there is reproducible from the repo, and the row lives in recommendations_tracked_2026-07-30.csv with all 49 columns intact.

It's also a fair representative of the problem. The model claimed 56.3% on a team the market priced at 42.7% — a 13.7-point disagreement on a game the market called correctly. Which brings us to how that gets measured.

Every pick clearing the tracking gate is logged at its entry price whether or not it was staked — grading only the bets you liked enough to stake is exactly how survivorship bias gets into a betting record. The next morning settle-tracked joins final scores and writes win/loss/push, and update-clv grades each entry against the closing line in no-vig probability points.

CLV matters because it converges to signal far faster than win-loss does. Beating the close consistently is the precondition for long-term profit, and it's measurable in weeks rather than seasons.

Model validation is walk-forward: train through date X, predict the next 500 chronological games, slide, repeat — with a 7-day purge gap between train and validation so nothing bleeds across the boundary. Eleven folds cover 2023 through early 2026.

metric	out-of-fold	walk-forward	market line, same rows
Log loss	0.67853	0.6784	0.67844
Brier	0.24284	0.2428	0.24282
AUC	0.5876	0.590	0.5877
Accuracy	56.5%	56.3%	56.4%

The system works as engineered. It does not beat the market, and the instrument that says so is λ.

λ is fitted out-of-fold, free to take any value, over 7,635 games. It came back at −0.0039, with beats_market_log_loss = False and a log-loss difference against the line of −0.000094. A coefficient free to choose chose to lean marginally against the residual signal. That is what "no information here" looks like when nothing forces the answer.

The adjusted-strength ratings — the feature I was most pleased with — carry real signal in isolation (AUC 0.531) and contribute λ ≈ −0.01 once the line is in the model. The market had already priced them, which in hindsight is obvious: park factors and team quality are the first two things anyone models.

Both values are written to train_metrics_oof.csv on every retrain, so this can't quietly stop being true. It's now the acceptance test for any new feature: if λ doesn't move, the feature is already in the price. Building that instrument is the part of this project I'd keep if I threw away the rest.

The live record agrees. Across 437 settled picks the ledger is 205-225-7, −$5,056, −11.8% ROI. Full breakdown, filterable, with the running equity curve, is published here alongside the raw CSV.

What survives is structural rather than predictive — cross-book fair value, steam and line lag, same-game-parlay correlation, promo conversion. All four are implemented. The honest finding on the first: scanned live against a 13-game slate, the best available price beat consensus by 0.00%, because the free feed exposes only two to four bettable books quoting within 0.002–0.006 of each other. The scanner reports zero rather than loosening its threshold until picks appear.

git clone https://github.com/samarpreetxd/mlbmodel.git
cd mlbmodel && python -m venv .venv && pip install -e .

# one-time backfill from 2023
python -u -m mlb_edge_engine.cli backfill-history \
    --start-date 2023-03-01 --chunk-days 21 --skip-existing
python -u -m mlb_edge_engine.cli build-features
python -u -m mlb_edge_engine.cli train 2023-03-01 <yesterday>

# then, daily
python run.py

No API keys required. If the backfill is interrupted, rerun it — --skip-existing resumes where it left off. The -u matters on long jobs; without it Python buffers the progress logs and the backfill looks frozen when it isn't.