Starter Removal Methodology
Methods and sources for “When to Pull the Starter.” The study is MLB-02 in the Pascal research repository. It prices every keep-or-pull moment in 2026 regular-season starts, using only information available before each game.
- Design and preregistration
- Data and population
- Models
- Margins, bullpen cost and ties
- Gates, failures and amendments
- League totals
- Sensitivity runs
- Team table
- Where each number comes from
- Post-publication review
- Figures
- Replication bundle
- Limitations
Design and preregistration
The study was chartered and its methods were frozen in a preregistration on October 13, 2026, before any study data was read. The preregistration's SHA-256 is 117f4424717fc74121a19ee8b7e756f7dec40b42c8ff83fc5020f3999f615814. Every later change was written as a dated amendment that states whether results had been seen, and every owner decision is logged with what was known when it was made. Four amendments and ten owner decisions are on record, and the failures they address are described below rather than hidden. The last decision, on October 16, corrected the published article after the owner's review; the analyses behind those corrections are exploratory and are reported in the post-publication review below.
The headline was chosen to be league-level before the article was outlined. Team results appear only in the table on this page, with the assumptions that drive them stated beside each number.
Data and population
The data are MLB Stats API game feeds for the 2026 regular season, stored once and identified by SHA-256 in the research repository's receipts. The development window runs from March 25 through September 10, 2026, and a confirmation window from September 11 through 27 was held out for calibration checks. Talent estimates use only games dated before each game, and no estimate uses the result of the plate appearance being priced.
| Development window | Starts |
|---|---|
| Starts reconstructed from 2,196 of 2,197 eligible games; one suspended game excluded | 4,392 |
| Excluded: starter removed during a plate appearance | 22 |
| Excluded: injury delay before the removal | 110 |
| Excluded: starter ejected | 2 |
| Excluded: openers, by the pitcher's prior starts | 133 |
| Used only to build talent rates: starts before April 15 | 493 |
| Primary starts | 3,632 |
The primary starts yield 99,072 batter-level comparisons. Of those, 84,501 are observed keep-or-pull moments: 80,887 batters the starter faced and 3,614 removal points. The other 14,571 are extrapolated comparisons after each removal, up to nine per start, when the starter had already left. They are not decisions a manager could still make, and they are used only to price pulls that came too early. Every starter matched the boxscore's games-started credit, and pre-plate-appearance base states agreed with the processed Statcast pitch file 99.92 percent of the time.
Models
Talent. Plate appearances are grouped into seven outcomes: strikeout, walk or hit by pitch, single, double or triple, home run, ground-ball out and air out. Hitter rates are split by the pitcher's hand and pitcher rates by the batter's side, and each is shrunk toward the league rate with a separate pseudo-count for each outcome. The pseudo-counts were chosen by held-out likelihood on development data only. For hitters they run from 25 for strikeouts to 3,200 for doubles and triples, and for pitchers from 50 for strikeouts to 1,600 for doubles and triples. Hitter and pitcher rates are combined by the odds-ratio method.
Starter. A multinomial logit on 80,630 development plate appearances, with the matchup as an offset and terms for times through the order, a pitch-count spline, fastball velocity against the pitcher's own earlier baseline and days of rest. The third time through the order adds about 0.013 runs per plate appearance compared with the first, with a 95 percent bootstrap range of 0.006 to 0.020.
Reliever. The same matchup structure fitted on 61,754 development reliever plate appearances, with a term for a reliever's first batter. On the confirmation window, every outcome's average prediction was within 0.3 points of what happened.
Win probability. A half-inning boundary model by inning and half with a lead slope, combined with the distribution of runs to the end of the half-inning from each base-out state. Both were estimated on development games, and the boundary model used 39,065 half-inning starts. Its Brier score on confirmation states was 0.155, against 0.249 for a constant home-win rate.
Margins, bullpen cost and ties
At each decision point, the margin is the modeled win probability, for the team in the field, with the reliever facing the next batter minus the same with the starter facing him. A positive margin favors the reliever. The run-scale version uses expected runs allowed instead. Because only the next batter differs, margins add across decision points.
Using the bullpen has an unobserved cost, c, expressed in win probability per batter the bullpen faces. A reference rule pulls the starter at the first decision point where the margin is at least c. Late cost sums the margin above c at points where the starter stayed in after the rule would have pulled him, and early cost sums the shortfall at points the reliever faced before the rule would have pulled. For each start, the break-even interval is the range of c that makes the actual timing agree with the rule. That interval is empty when an earlier margin was larger than the margin at the actual pull.
Talent uncertainty comes from 200 draws of each player's rates per decision. A decision counts as too close to call at cost c when c lies inside the margin's 90 percent range or within 0.25 points of the margin, and such a decision adds nothing to early or late cost. The range reflects uncertainty in player talent rates only. It does not include uncertainty in the model coefficients, the form of the models, bullpen availability, the win-probability calibration failure or the extrapolated starter state.
Gates, failures and amendments
| Check | Result | Handling |
|---|---|---|
| Input integrity and population reconstruction | Passed | |
| Leakage guard: a same-game final pitch count must be rejected | Passed | |
| Outcome calibration on the confirmation window | Failed. Home-run, extra-base-hit and strikeout predictions were spread too widely or too narrowly | Amendment 003, approved after the failure: a separate shrinkage amount for each outcome. The rerun passed every outcome check and failed one cell of 13 plate appearances after pitch 100, which the owner accepted as a limitation |
| Times-through-the-order effect positive | Passed | |
| Win-probability calibration within 3 points in every decile, on 229 games | Failed. Deciles 5 through 9 under-predicted home wins by 3.1 to 6.8 points; every game-bootstrap interval included zero | Accepted with a limitation. Every win-probability result is reported with its run-scale equivalent, and a team result is used only when both scales agree |
| Talent-uncertainty intervals centered on the model | First run failed: only 60 percent of margins lay inside their own range | Amendment 004 corrected the draws. The rerun placed 99.998 percent of margins inside their own range. First-run team results were withheld and are kept on record |
| Two identical runs produce identical outputs | Passed | |
| Negative control: shuffled pull timing | Passed, weakly. Actual timing fit a common cost for a median 3.2 percent of a team's starts, against at most 2.5 percent for 20 shuffles |
Amendment 001 added a sensitivity run that restores injury-delay starts, because the feed does not say who was hurt. Amendment 002 read pitch velocity from the game feeds, which match the processed Statcast file for 99.998 percent of pitches, because that file ends before the confirmation window.
League totals
League totals of late minus early cost, in modeled wins for the development window. Rows marked "added" use cost values added after the first analysis run; they are descriptive and were never used to name a team. At 0.25 points per batter and above, nearly every margin falls below the assumed cost, so every team scores early by construction.
| Cost per batter, points of win probability | Early | Late | Late minus early | Run scale, runs |
|---|---|---|---|---|
| 0 | 0.60 | 16.19 | +15.58 | +146.3 |
| 0.05, added | 1.19 | 9.12 | +7.93 | +66.1 |
| 0.10, added | 2.36 | 3.32 | +0.96 | -12.6 |
| 0.15, added | 3.86 | -2.57 | -6.43 | -92.4 |
| 0.25 | 10.88 | -18.37 | -29.26 | -292.9 |
| 0.50 | 41.66 | -57.72 | -99.38 | -893.0 |
Negative late values at higher costs come from the first-crossing rule: an early spike in the margin can trigger it, after which the remaining margins fall short of the cost.
Sensitivity runs
League late minus early at zero bullpen cost, in modeled wins, changing one assumption at a time.
| Run | Change | Late minus early |
|---|---|---|
| Primary | Reliever who actually entered | +15.58 |
| 1 | Shrinkage halved or doubled | +15.72 or +18.25 |
| 2 | Fatigue and times-through-order penalty increased by half | +14.06 |
| 3 | No velocity term | +14.84 |
| 4 | Median rested reliever instead of the actual one | -1.35 |
| 5 | Run 4 with four alternative rest rules | -1.49 to -1.17 |
| 6 | Each pitcher finishes the inning instead of facing one batter | Same keep-or-pull call in 78.5 percent of 14,456 decisions |
| 7 | Openers included | +15.43 |
| 8 | No early-season cutoff | +15.63 |
| 10 | Injury-delay starts included | +15.86 |
Team table
At zero bullpen cost, ten teams met every preregistered condition for being named: a late-minus-early total of at least half a modeled win with a 90 percent game-bootstrap range excluding zero, the same direction under sensitivity runs 1 through 5, and the same direction on the run scale. All ten lean late. Both assumptions behind this table matter: the bullpen is treated as free, and the reliever who eventually entered is treated as available at every earlier batter. The last column replaces that reliever with the median rested arm, and the effect nearly disappears. Managers are not named.
| Team | Late minus early, wins | 90% range | Run scale, runs | With median rested reliever, wins |
|---|---|---|---|---|
| Cincinnati Reds | 1.10 | 0.75 to 1.48 | 10.7 | 0.04 |
| Houston Astros | 1.01 | 0.69 to 1.36 | 9.1 | 0.03 |
| Colorado Rockies | 0.95 | 0.59 to 1.36 | 8.7 | 0.11 |
| Chicago White Sox | 0.87 | 0.56 to 1.19 | 8.4 | 0.25 |
| San Francisco Giants | 0.79 | 0.50 to 1.10 | 8.7 | 0.02 |
| San Diego Padres | 0.75 | 0.53 to 1.01 | 6.6 | 0.08 |
| Los Angeles Angels | 0.70 | 0.47 to 0.94 | 7.7 | 0.05 |
| Athletics | 0.65 | 0.39 to 0.94 | 7.6 | 0.20 |
| Toronto Blue Jays | 0.64 | 0.32 to 0.98 | 6.2 | 0.11 |
| Detroit Tigers | 0.52 | 0.23 to 0.81 | 4.8 | 0.06 |
The other 20 teams were unresolved at zero cost: five were large enough but changed direction under a sensitivity run, two were too small, and thirteen were both.
Where each number comes from
All files are in research/mlb02/ in the pascal-abs-research repository. Row-level outputs are regenerated by the scripts named here and identified by SHA-256 in each stage's results file. CLAIM_LEDGER.md gives the approved wording and prohibited overstatements for each claim.
| Article statement | Value | Source |
|---|---|---|
| Starts, observed moments and comparisons | 3,632; 84,501; 99,072 | G0_G1_RESULTS.md, G6_RESULTS.md |
| Third time through the order | about 0.013 runs per PA, 0.006 to 0.020 | MODELS_RESULTS.md, G4b |
| Median margin and median 90% range, observed moments | 0.16 and 1.25 points | POST_PUBLICATION_REVIEW.md, A; figures/manifest.json |
| Too close to call at zero cost, observed moments | about 96 percent; talent range alone 95.6 percent | POST_PUBLICATION_REVIEW.md, B and C |
| Stability across draws | shares within half a point | POST_PUBLICATION_REVIEW.md, D |
| No single consistent cost | 92 percent of starts | G6_RESULTS.md, finding 2 |
| Example start | Astros, June 21, 2026 | figures/FIGURE_REVIEW.md, rule in FIGURE_SPEC.md |
| One-batter and rest-of-inning agreement | 78.5 percent | G6_RESULTS.md, run 6 |
| League balance point, actual-reliever assumption | 0.05 to 0.15 points per batter, by allowance | G6_RESULTS.md, finding 3; POST_PUBLICATION_REVIEW.md, C |
| Late total, actual and median reliever | about 16 and -1.3 wins | G6_RESULTS.md, finding 4 |
| Win-probability calibration | 229 games | G5_RESULTS.md |
Post-publication review
These analyses were run on October 16, 2026, after publication, at the owner's request. They are exploratory, they change no preregistered gate, and they are in POST_PUBLICATION_REVIEW.md with the script that produced them. All shares below are for the 84,501 observed moments at zero bullpen cost.
| Part of the tie rule | Share of observed moments |
|---|---|
| 90% talent range contains zero | 95.6% |
| Margin within the 0.25-point allowance | 65.0% |
| Both | 64.7% |
| Talent range only | 30.9% |
| Allowance only | 0.3% |
| Either, the published rule | 95.9% |
| Allowance, points | Tie share | League late minus early at zero cost, wins | League balance point between |
|---|---|---|---|
| 0 | 95.6% | +15.71 | 0.05 and 0.10 points |
| 0.05 | 95.7% | +15.71 | 0.05 and 0.10 points |
| 0.10 | 95.7% | +15.71 | 0.05 and 0.10 points |
| 0.25, published | 95.9% | +15.58 | 0.10 and 0.15 points |
| 0.50 | 96.6% | +14.50 | 0.10 and 0.15 points |
The tie share barely depends on the allowance, but the league balance point does. The article therefore reports it as 0.05 to 0.15 points per batter, under the assumption that the reliever who entered was available earlier.
| Talent draws | Range contains zero | Agreement with published classification |
|---|---|---|
| 200, published seed | 95.6% | |
| 200, second seed | 95.6% | 98.2% |
| 1,000, published seed | 96.0% | 98.6% |
Point margins are identical in every run. Aggregate shares move by less than half a point, while 1.4 to 1.8 percent of individual classifications change because they sit near an interval edge, so single borderline classifications should not be relied on. No variance decomposition was run on how much the hitter, fatigue, leverage, handedness, base-out state or the reliever matchup each contribute, and the article no longer claims that any one of them dominates.
Figures
The three figures are generated by research/mlb02/build_figures.py from the decision-valuation outputs; no figure value is typed by hand. Since the October 16 correction, Figure 1 uses only the 84,501 observed moments. The example start in Figure 2 was chosen by a rule committed before any start was viewed: the median team by policy consistency, then that team's median eligible start by batters faced. scripts/insights/starter-pull-figures/build.py in the pascal-site repository copies the figures after checking each file's SHA-256 against the research manifest and renders the share card.
Replication bundle
The research repository is private, so the material behind this page is published as a replication bundle (118 KB). It contains the charter, the frozen preregistration and its SHA-256, the four amendments and implementation notes, the owner decisions log, the claim ledger, the results file for every gate, the post-publication review, the figure spec, the figure-generation code with its data, and aggregate outputs: model manifests, team totals and sensitivity totals. MANIFEST.txt inside the bundle lists the SHA-256 of every file, and README.txt names the source commit. Raw MLB Stats API feeds and row-level outputs are not redistributed.
Limitations
The comparison looks one batter ahead and is a named reference, not an optimal policy, and margins after each removal are extrapolated from the starter's state at that point. The win-probability model failed its calibration tolerance on 229 held-out games, and decisions after pitch 100 rest on thin calibration evidence. Warm-up status, pitch limits set before the game, injuries the feed does not flag and game planning are not observed. The talent ranges cover player talent only, and individual classifications near an interval edge can change with the random draws. The bullpen cost is unknown, and the league balance point depends on the tie allowance and uses cost values added after the first analysis run. The study is observational and makes no causal or psychological claims.