Putting Machine Learning in the Checkout Path Without Making Checkout Depend on It

There’s a moment that changes how you think about machine learning in a purchase funnel. It’s the moment you pull up real production traces for your model-serving endpoint and look at the tail. Ours was multiple seconds at the 99th percentile, with only a fraction of calls landing under a second. The funnel those models were supposed to improve renders in a few hundred milliseconds. The math doesn’t work. You cannot put a multi-second dependency on the critical render path of the page that makes the money.

That single fact reshaped the entire design. This post is about how we run several production ML predictors inside a high-conversion checkout flow: a dropoff predictor, a max-cart-value predictor, a ranked-upsell scorer, and a purchase-propensity model that influences which package options are surfaced. And how we did it without ever letting inference become a single point of failure or a drag on page speed.

Three forces that won’t sit still

Adding prediction to a money-making, latency-sensitive funnel means fighting a three-way tension, and you don’t get to ignore any corner of it.

The first is model quality. Our hardest signals are rare. The purchase-propensity model is a binary classifier where positives are a very small fraction of traffic, well under one percent. Training something honest on a tiny positive class, with features that are only partially available at the moment you need a prediction, is its own discipline.

The second is runtime integration. Real measured serving latency was far outside any inline checkout budget. A multi-second tail does not fit in a 100ms window, no matter how clean the architecture diagram looks.

The third is operational trust. Product teams will not, and should not, put an ML dependency on the critical path unless failure modes are explicit, ownership is unambiguous, and every decision is traceable and reversible. “The model said so” is not an acceptable explanation when conversion dips.

Scores are signals, not decisions

The first principle is that clients never call model endpoints. Browsers don’t even know the models exist. A server-side proxy, a backend-for-frontend layer, gathers and normalizes feature inputs, then calls the model services. Credentials, schemas, and failure handling live in one place instead of being smeared across front-end code.

Each predictor ships as an independent Python/FastAPI microservice on Kubernetes, deployed through a GitOps pipeline (Kustomize plus Argo CD). We deliberately ship discrete service images rather than one shared monolith, so a change to the upsell scorer can’t take down the propensity model. The trained models are served from a central internal ML platform; the services wrap them with feature normalization and failure handling.

Where models compose, we keep them close. The ranked-upsell service calls the max-cart-value service internally, and the two are deliberately co-located so that hop stays cluster-local. A composed prediction that crosses the network twice is twice as fragile and twice as slow. Keeping it in-cluster removes a whole class of avoidable failure.

The second principle matters more: model scores feed a centralized decisioning layer, not the experience directly. That layer ingests context, features, and model scores, applies governed and versioned rules, and returns a declarative JSON “experience manifest.” The client decides how to render; the service decides what to render. A rule pack expressed in something like CEL or JSONLogic can change the offered experience without shipping a client release. Roughly:

context + features + model scores
        │
        ▼
  decisioning layer
   ├─ versioned, code-reviewed rule packs
   ├─ model scores as inputs (never the final word)
   └─ deterministic fallback matrix
        │
        ▼
  experience manifest (JSON)  ──►  client renders

This indirection is what makes ML safe to ship here. The model is one input among several, and the layer that turns inputs into an experience is code-reviewed, versioned, and reversible.

Make the model’s absence a non-event

Because tail latency was multi-second, the funnel never waits on a model inline. Instead, there’s an explicit failure matrix with deterministic fallbacks. If the model is slow or down, we serve a rules-only manifest and flag the models as skipped. If the feature snapshot is stale, we serve last-good within a TTL, or fall back to a default. On a cold cache, we use single-flight so one request populates the cache instead of a thundering herd hammering the model platform.

The pattern we converged on is update-on-resolve. When new context arrives, we dispatch the model call. If it returns within the timeout, we generate and cache a fresh manifest. If it doesn’t, we apply fallback rules, cache that, and serve it immediately, then update the cached manifest once the model call eventually resolves, so the next request benefits. The user on the critical path always gets a fast, deterministic answer. Inference improves the experience asynchronously; it never gates it.

This is the whole game. ML becomes a quality enhancement layered on top of a system that is correct and fast without it, rather than a dependency the funnel can’t render without.

What the models taught us once we started training them

The first surprise had nothing to do with model architecture. Our single biggest model-quality improvement came from fixing how we labeled positives. Specifically, we started including “immature-but-already-converted” rows that our original query had excluded. That roughly quadrupled the positive count and moved our ranking metrics, PR-AUC and lift, far more than any architecture or feature change we tried before or since. With a tiny positive class, every real positive you can correctly recover is worth more than another hyperparameter sweep.

The model itself is deliberately boring: a LightGBM classifier on tabular behavioral and contextual features, with tuned class-imbalance weighting (scale_pos_weight) to cope with the rare positives. Boring is a feature. It’s fast, it’s interpretable, and it’s easy to reason about in production.

Reading feature importance turned out to need its own caution. We measured it three ways, LightGBM gain, SHAP, and permutation importance, and reasoned explicitly about each method’s bias. Tree gain systematically underrated binary features like entry-path and geo-match, while permutation importance showed they were among the most impactful signals we had. If we’d trusted gain alone, we’d have dropped genuinely predictive features. Behavioral signals (which package the customer chose, how they arrived, time-of-day and seasonality) dominated predictions; demographic features like industry, device, and browser barely mattered. One feature was nearly constant, with the overwhelming majority of values falling in a single “Other” bucket, and shuffling it had zero or even positive effect, which made it a clear candidate for removal.

The most counterintuitive find was that the offline upper bound is not the deployable model. Features that improve an offline “all-features” model often do not help, and can actively hurt, a lean deployable model, especially with few positives, because of feature redundancy and overlap. We could not close the gap between the offline ceiling and the deployed model by sprinkling in a few more features. The lean model is its own optimization problem.

And the constraint that bit hardest was timing. We had a genuinely useful signal we had to drop entirely because it depended on invoice and checkout attributes that simply don’t exist yet at the moment we make the prediction, early in the funnel. The most important question about any candidate feature isn’t whether it predicts. It’s whether it’s available, accurately, at the instant of inference. A feature you compute from data that arrives after the decision is a leak in training and a null in production.

Trust you can flip a switch on

Operational trust isn’t a vibe; we built mechanisms for it. Every service must register in our service catalog with owner and team metadata, validated in CI, so alerts route to a human who can act. We adopted a deliberately fail-loud deployment convention: a new service’s image placeholder forces an ImagePullBackOff rather than silently pulling some wrong shared image. A deploy that’s misconfigured screams instead of quietly serving garbage.

Rollout is flag-driven, with sticky assignments, exposure logging for clean lift measurement, decision tracing so we can answer “why did this customer see this,” and central kill switches for both experiments and rule packs. If a rule pack misbehaves, someone flips a switch. No redeploy, no scramble. Production access stays internal-only while the platform keeps hardening; we’d rather earn the right to broaden exposure than start wide.

The direction from here is to push more logic into the centralized decisioning service so that experience changes are configuration, not code releases, and so the boundary between “fast deterministic baseline” and “ML-enhanced experience” stays crisp and observable. The async update-on-resolve pattern gives us room to use heavier models over time without ever renegotiating the inline latency budget, because the inline budget no longer includes inference at all.

If there’s one idea worth taking from this, it’s the inversion at the center of the design. Don’t ask how to make checkout fast enough to wait for your model. Ask how to make your model’s absence a non-event. Once the funnel renders correctly and quickly with no model in the loop, you can add as much intelligence on top as you can train, safely, reversibly, and without ever betting the conversion rate on a prediction arriving in time.