Skip to the stories
AI & ML

Shopify’s GraphQL Specialist Shows How Open Models Can Cut AI Costs

  • John Johnson
  • 422 reads

Shopify post-trained an open model to match its GraphQL quality target, return answers faster and reduce serving costs by about 90%.

Shopify’s GraphQL Specialist Shows How Open Models Can Cut AI Costs

Shopify’s GraphQL subagent shows a practical way to divide work between a general purpose AI model and a task specific one: keep the larger model in charge of understanding merchants and choosing actions, then send a token heavy technical job to a specialist. Shopify says its post-trained open model now scores slightly higher than the larger model on GraphQL tasks, responds much faster and costs about 90% less to serve.

The design sits inside Sidekick, Shopify’s AI commerce assistant. Its main model talks with merchants, plans work and calls tools. Subagents handle focused jobs; some use open models that Shopify post-trains and serves on its own GPUs. That gives the company control over model weights and the timing of updates, while reducing reliance on per-token API charges for work that consumes far more input than output.

GraphQL is a revealing case. Asked which products are almost out of stock, the subagent must inspect Shopify’s API schema, find relevant documentation, write a query, run it and return the results. Shopify says this process reads about 40 tokens for every token it writes. A general model can do the job, but that input-heavy pattern makes speed and serving cost key design constraints.

Sidekick’s task experts are built through a recurring process Shopify calls its continual learning loop. Each round measures quality, improves the existing agent, selects an open model, checks serving performance and turns production failures into training examples. After post-training, the team scores the new model and begins another round.

The optimization process starts with setup work, then repeats as new production data becomes available.

Set a quality bar before training

Product experts define what a good GraphQL response means using a rubric that covers completeness, whether the query runs, and the quality of the response. Shopify measures how often two human experts agree on the same conversations, then tunes an LLM judge with the GEPA and ACE optimizers until its ratings match held-out human labels at a similar level.

That judge becomes the measuring tool for agent changes, model candidates and training runs. Establishing it first matters: without a consistent score, teams can’t tell whether a new prompt or checkpoint has improved the task or merely changed the output.

Shopify first tries to improve its existing agent, which uses a larger model. An autoresearch loop proposes small edits to prompts, tool descriptions or control flow. The team keeps a change only when the judge’s score rises. The resulting agent sets the target that an open model must meet.

Choose for quality and serving fit

Shopify says it doesn’t pretrain these specialists. Instead, it tests open models inside the actual subagent, using the same tools and judge, and considers initial quality, serving speed and support in NVIDIA Megatron Bridge, the training stack it uses. For GraphQL, it chose NVIDIA Nemotron 3.5 Lightning, a mixture-of-experts model with 30 billion total parameters and three billion active for each token.

Serving performance is tested early, since a model that misses production response needs won’t be useful even if training raises its score. Shopify uses NVIDIA AIPerf, an open-source load tester in NVIDIA Dynamo, to replay production traces with synthetic text rather than customer data. Shopify also contributed support for live-response replay, allowing multi-turn tests to exercise the server’s prefix cache as production traffic does.

The best serving setup depends on the workload. For Buyer Profile, a batch job, Shopify compared vLLM, SGLang and NVIDIA TensorRT-LLM with AIPerf; TensorRT-LLM did the most work per GPU. Separately, NVIDIA’s measurements found that SimGym, Shopify’s shopping simulator, ran 4.5 times faster on a Blackwell B200 than on an H200, at 1.48 times lower cost.

Convert failures into usable examples

Each night, the judge scores a sample of production GraphQL conversations. Anything short of a perfect result enters a self-healing process: a larger model acting as critic writes a hint, but not the answer. A simulated merchant retries the request with that hint against the same store. If the judge approves the retry, Shopify removes the hint, anonymizes the conversation and adds it to the training set. If it still fails, the critic can try again, up to three times.

Across 86 nightly runs, the process repaired 69% of the failures it attempted.

In 86 nightly runs, the process repaired 59,535 conversations, or 69% of the failures it tried. Human annotators corrected the remainder. Those corrections also feed the ground truth used to calibrate the judge, tying real production errors back into both evaluation and training.

Measure the gains across the loop

Shopify post-trains with NVIDIA Megatron Bridge, which connects standard Hugging Face checkpoints to NVIDIA’s Megatron Core training library without requiring a model rewrite. The company says it can scale training across nodes on Nebius by changing settings rather than code. On the same FP8 post-training job for its Analytics model, eight B300 GPUs trained 1.86 times faster than eight H200s and used 46% fewer GPU-hours.

The comparison covers an Analytics training job and NVIDIA’s SimGym serving measurements.

The GraphQL results show how much the repeated process contributed. Nemotron 3.5 Lightning started at 37.7 on Shopify’s judge, while the larger model scored 73. After one loop, the open model reached 61.5. Following six loops and the addition of 61,000 repaired examples, it scored 74.5.

The GraphQL score rose over six rounds as repaired examples were added.

Speed improved too. In an AIPerf replay of production GraphQL traffic, the post-trained model returned a full response in a median of 0.87 seconds, compared with 6.50 seconds for the larger model’s API. The slowest 1% of requests finished eight times sooner. Another post-trained open model of comparable size came close at the median but took more than twice as long at the 99th percentile.

The load test compares median and 99th-percentile response times, along with serving costs.

Shopify puts GraphQL serving costs at about 90% below the larger model’s API pricing. Across its post-trained models, the company reports reductions of 90% to 98% against that pricing. These results depend on the workload, model and serving setup, but the engineering case is clear: keep the larger model on merchant interaction and planning, while assigning repetitive, context-heavy work to a specialist whose quality can be measured and improved. The next test is whether that loop continues to deliver the same gains as Shopify applies it to more tasks.

Source: John Johnson · shopify.engineering

From the same set