Skip to the stories
AI & ML

Shopify’s Cross-Store Product Clustering Combines Matching Models and Graphs

  • William Rodriguez
  • 974 reads

Shopify connects listings across millions of stores with tiered matchers, bounded graph components and learned merge decisions.

Shopify’s Cross-Store Product Clustering Combines Matching Models and Graphs

Recognizing the same product across millions of stores is a data problem with a costly failure mode: miss a match and listings stay separate; accept a bad one and distinct products can be grouped together. Shopify’s cross-store clustering system tackles both by combining fast candidate search, three matching models and a learned process for deciding whether whole groups should merge.

The system supports Shopify Catalog, a shared product data layer exposed to developers and AI agents through the Catalog API. Its job is to map listings from different merchants to a common Universal Product Identifier, or UPI. That matters because stores can represent the same real-world item in different ways: one merchant might list several sizes and colors as variants of a product, while another creates separate listings.

Shopify says its platform contains billions of product listings across millions of stores. Matching them across merchants is harder than grouping listings within a single store. Merchants use different naming conventions, put product details in different fields and take their own photos. Barcodes can help, but they may be absent or unreliable. A model number in one listing’s title might appear only in another listing’s description.

Shopify Catalog clustering

Three stages, from candidates to clusters

The system separates the work into three stages: retrieve plausible pairs, score those pairs and use the evidence to form clusters. This structure avoids asking a model to assign every listing a globally consistent product label. Instead, a model answers a narrower question: do these two listings belong to the same UPI?

Pairwise decisions make it possible to gather matching evidence without first having a complete catalog of known product groups. But pairwise scores can disagree. A model might match A with B and B with C while deciding that A and C are different. That makes simple connected-component grouping unsafe: one mistaken link could join otherwise unrelated products.

Candidate retrieval starts with embeddings. The system represents products as vectors and uses approximate nearest-neighbor search to find listings that are close in that space. This reduces the search from every possible pair across billions of listings to a smaller candidate set. It also creates a recall limit: if a true match never makes the shortlist, none of the later models can evaluate it. The listing remains searchable under its own identifier, but won’t be grouped with the matching listing unless a later build finds the connection.

Even after retrieval, the candidate set is large enough that applying the most capable model to every pair would be too expensive. Shopify instead uses a cascade of three scorers, named L1, L2 and L3. Their scores and edge types feed the later cluster-building stage. The models have different costs and capabilities, so their scores aren’t treated as interchangeable.

Scoring pairs at different levels of cost

L1 is the inexpensive first pass. It’s a gradient-boosted decision tree classifier that uses features such as embedding distance, title and vendor similarity, and price relationships. Shopify says it can score trillions of candidate pairs without running a neural cross-encoder on each one.

L1 also helps decide which pairs deserve more processing. That decision is separate from whether the pair already has enough evidence to justify a merge. A candidate can receive a weak match score and still be worth sending to a more capable model. Keeping those decisions separate gives the system a way to spend additional compute where it may produce useful evidence.

L2 compares the two listings’ text together. It’s a fine-tuned multilingual E5 cross-encoder that reads titles, vendor names, predicted categories, URLs and shop names in a joint representation. Unlike retrieval embeddings, which represent items independently so they can be searched efficiently, joint encoding lets the model inspect differences between the two listings directly.

That helps when a small detail changes product identity despite substantial overlap elsewhere. Two phone cases might share a brand and product line, but one fits an iPhone 15 while the other fits an iPhone 15 Pro. L2 can weigh the compatibility wording against the shared text rather than treating the overall similarity as decisive.

There’s a limit: L2 doesn’t read product descriptions. If a distinguishing detail appears only there, the model may not see it. Selected comparisons can go to L3, which adds images and richer product information.

L3 brings visual evidence into the comparison. It’s a fine-tuned open-model reranker that considers images alongside text and metadata. A connector, button arrangement or cutout can distinguish products even when their text looks similar. At the same time, merchants can photograph the same product from different angles, so image differences need to be interpreted in context.

L3 returns a match score, not a written explanation. That fits the serving task: the cluster builder needs evidence it can use to make a decision. More expensive language-model reasoning is used earlier in the training pipeline, where it can produce labels for sampled comparisons. Shopify says it checks the language-model judge against human annotations, since errors in those labels could affect the models trained on them. Scores from stronger models can also supply training signals to less expensive ones.

The thresholds for each scorer are set separately, and Shopify evaluates how they work together at the cluster level. A higher-quality pair score has practical value only if it leads to better product groups.

Keeping the graph manageable

After scoring, listings become nodes in a graph and pairwise relationships become edges. Processing the entire graph as one unit would be difficult, while ordinary connected components can grow too large. The system first divides the graph into bounded components that can be handled on a single machine. These components are work units, not final product identities.

The partitioning problem has two competing needs: keep likely matches together, but limit the work required for each component. Randomly assigning products to machines would split many valid connections. Simply retaining every connected component could create groups too large to process efficiently.

Shopify starts with a bounded version of union-find, a data structure that tracks which component each product belongs to. Every product starts in its own component. The algorithm processes edges from highest score to lowest, skipping connections whose endpoints are already together. It merges two components only when the combined group fits within a set budget.

That budget uses the sum of product degrees—the number of graph connections attached to the products—rather than just the number of products. A densely connected component can take more work to process than a group with the same number of products and fewer edges.

At Shopify’s scale, a single machine still can’t efficiently process the full edge set. The system divides edges among parallel workers. Each worker uses ordinary union-find to remove redundant connections within its own partition. For example, in a triangle of three mutually connected products, one edge can be removed while preserving connectivity. Each worker outputs a spanning forest, a reduced set of edges that keeps its partition connected.

The workers’ forests are then combined, and a final bounded pass processes the remaining edges in score order. The original graph’s connection counts still determine the size budget. Because the workers simplify edges independently, the resulting bounded components aren’t guaranteed to match the components that would result from processing every original edge in one pass. The trade-off is that workers handle much of the edge processing in parallel, while a final shared pass enforces the component limit.

Cross-store clustering pipelineBounded components

Merge decisions use group-level evidence

Once graph components are small enough to process independently, the UPI builder uses hierarchical agglomerative clustering with a learned merge rule. It progressively considers joining smaller groups into larger ones. A gradient-boosted decision tree scores each proposed merge.

The model evaluates more than the score of the pair that prompted a proposed merge. It considers how much evidence connects the two clusters, how that evidence is distributed, how cohesive each cluster is, and whether weaker L2 or L3 scores contradict the merge. The system can distinguish evidence types and confidence levels instead of collapsing every edge into one number.

  • Support: How many connections run between the groups, and how dense are they relative to group size?
  • Coverage: Do many products on both sides have connections, or do only one or two products account for most of them?
  • Cohesion: How strongly connected are the products within each group compared with their links outside it?
  • Contradiction: Does lower-scoring L2 or L3 evidence argue against combining the groups?

This approach uses the pair scores already stored on graph edges. The builder can summarize relationships between groups without rereading all the product data for every proposed merge. A strong pair is useful evidence, but it doesn’t automatically establish that two entire groups represent the same product.

Shopify sets a very high precision target for accepting merges. A mistaken merge can combine two distinct product groups, while a rejected valid merge leaves related listings apart. The stated approach is to tune for precision first and then increase recall within that constraint.

A strong pair is evidence

The merge model’s training data must keep pace with its own decisions. A new version changes which merges it accepts, which changes the groups and the proposals it encounters later. Training only on proposals produced by the previous version would leave gaps in the situations the updated model faces.

Shopify describes an iterative process: run the builder, sample proposed merges, label them with the language-model judge and retrain. The examples include both acceptable and unacceptable merges, so the model learns from the evidence rather than simply copying its predecessor’s choices. Graph-derived features also need refreshing as scores arrive, listings change or products leave the catalog. The team says it now collects new proposals and retrains regularly.

Daily builds with continuous scoring

Product data changes over time: merchants add listings and update titles or images, while new candidate relationships emerge. The system needs to score new pairs and refresh scores when relevant product information changes.

L1 runs on CPUs and handles the full candidate set. L2 and L3 require GPUs, so Shopify places their work in queues rather than running them across every candidate at once. New comparisons join work already waiting for those models. The queues let the available GPU capacity keep processing comparisons continuously instead of waiting for a daily build to finish every expensive evaluation.

Queue priorities use earlier model scores, purchase and search signals, and the current graph structure. Those signals help identify likely matches, products buyers interact with and comparisons that could affect a cluster merge. New work can be processed ahead of older candidates when its priority is higher.

Completed scores are cached along with the model and relevant product features used to produce them. When those inputs haven’t changed, a later build can reuse the result. Each daily UPI build combines current L1 scores with the L2 and L3 results available at that point; it doesn’t wait for the queues to empty. Background processing continues to add scores for new, changed and previously unscored pairs.

The design makes expensive comparisons reusable: as training data for faster models and as cached evidence for future builds. It also sets clear limits on what to watch. Retrieval can miss matches before scoring begins, and the daily build uses whichever advanced-model scores are ready at the time. Shopify’s approach aims to balance those constraints with high-precision merges and ongoing updates, while merchants keep control of how their own listings are structured.

Source: William Rodriguez · shopify.engineering

From the same set