How LinkedIn Replaced Five Retrieval Systems with One LLM at 1.3 Billion User Scale

Imagine inheriting a codebase with five separate retrieval systems, each with its own infrastructure, its own index, its own optimization strategy — all wired together to produce one feature: the feed a user sees when they open an app. Now imagine maintaining that at 1.3 billion users. That was LinkedIn's reality.

Over the past year, LinkedIn's engineering team tore that apart. They replaced the entire heterogeneous retrieval architecture with a single, unified system powered by LLM-generated embeddings, and layered a new sequential ranking model on top that treats your professional history as a story rather than a bag of isolated clicks. The result, detailed in a new LinkedIn Engineering post and confirmed by VP of Engineering Tim Jurka in a VentureBeat interview, is a feed that's both more relevant and cheaper to run.

For developers building recommendation, retrieval, or ranking systems — even at a fraction of LinkedIn's scale — this case study is a goldmine. The engineering decisions they made, and the problems they ran into, apply anywhere you're trying to match users to content.

Background: Why Five Pipelines Became a Liability

LinkedIn's feed architecture had grown organically over 15 years. When a user opened their feed, content was retrieved from multiple specialized sources simultaneously:

• A chronological index of what people in your network had posted
• Geographic trending content based on your location
• Collaborative filtering drawing on what similar members engaged with
• Industry-specific trending posts from your professional sector
• Several embedding-based retrieval systems for semantic similarity

Each source had its own infrastructure, its own index structure, its own team doing optimization. The architecture worked — LinkedIn built a reputation for recommendation quality precisely because it had invested in those systems. But the maintenance cost was brutal, and holistic optimization was essentially impossible. Tuning one pipeline could break the balance with the others.

The deeper problem was what Erran Berger, VP of Product Engineering at LinkedIn, described in a recent podcast interview: traditional systems treated each impression independently, completely missing how professionals actually engage with content over time. Your interest in renewable energy infrastructure doesn't show up in a keyword search for "electrical engineering." The old system couldn't bridge that gap.

Unifying Retrieval with Fine-Tuned LLM Embeddings

The core question LinkedIn asked: What if a single unified retrieval system, backed by LLM-generated embeddings, could replace all five pipelines?

The answer turned out to be yes — but it required rethinking the entire stack.

LinkedIn built a dual encoder architecture using a fine-tuned LLM. The model takes two inputs — a member prompt (your profile, skills, work history, and a sequence of posts you've engaged with) and a post prompt (format, author info, engagement metadata, and text) — and maps them into a shared embedding space. Retrieval becomes a nearest-neighbor search: find the posts whose embeddings are closest to your member embedding.

The semantic power of LLM pretraining is what makes this work. The old embedding models could detect shallow correlations — "power," "energy," "electronics" — but couldn't connect "electrical engineering" to "small modular reactors." The fine-tuned LLM brings world knowledge from its pretraining corpus, understanding that electrical engineers often work on power grid optimization and renewable energy integration. That kind of reasoning is completely out of reach for traditional collaborative filtering.

This also solves the cold-start problem with unexpected elegance. When a brand new member joins with only a job title and headline, the LLM can infer likely interests from its world knowledge — no engagement history required. Traditional recommendation systems are essentially blind to new users until they've accumulated enough clicks to build a signal.

The Numerical Feature Problem (and the Fix Every Developer Should Know)

Here's the most replicable technical insight in LinkedIn's entire writeup, and it's one that will bite anyone who naively feeds structured data into an LLM.

When LinkedIn first built their post prompts, they passed engagement counts as raw numbers: views:12345. Sensible enough. But the model treated "12345" exactly like any other sequence of text tokens — it had no understanding that this number represented popularity, or that 12,345 was meaningfully different from 234 or 890,000. The correlation between popularity features and embedding similarity was nearly zero (-0.004). Essentially, the model was ignoring the signal entirely.

Their fix was elegant: convert continuous numerical values into percentile buckets, then wrap them in special tokens.

views:12345 became <view_percentile>71</view_percentile>

This tells the model: "this post is in the 71st percentile of view counts." The percentile value (1–100) tokenizes as one or two tokens with stable, learnable representations. The model can now distinguish low-popularity posts (0–20th percentile) from viral ones (90th+) in a way that actually sticks in the embedding. The result was a 30x improvement in correlation between popularity and embedding similarity, and a 15% improvement in recall@10 across the retrieval system.

Here's a clean Python implementation of this pattern you can apply to your own LLM-powered retrieval systems:

import numpy as np
 
class PercentileFeatureEncoder:
    """
    Encode numerical features as percentile bucket tokens for LLM prompts.
 
    LinkedIn's key insight: raw numbers lose ordinal meaning when tokenized.
    "views:12345" -> meaningless tokens.
    "<view_percentile>71</view_percentile>" -> learnable signal.
 
    Impact at LinkedIn: 30x correlation improvement, +15% recall@10.
    """
 
    def __init__(self, feature_name: str):
        self.feature_name = feature_name
        self._sorted_ref: list[float] | None = None
 
    def fit(self, values: list[float]) -> "PercentileFeatureEncoder":
        self._sorted_ref = sorted(values)
        return self
 
    def encode(self, value: float) -> str:
        if self._sorted_ref is None:
            raise RuntimeError("Call .fit() before .encode()")
        idx = int(np.searchsorted(self._sorted_ref, value, side="right"))
        percentile = min(int(idx / len(self._sorted_ref) * 100), 100)
        return f"<{self.feature_name}_percentile>{percentile}</{self.feature_name}_percentile>"
 
 
def build_post_prompt(
    post: dict,
    view_encoder: PercentileFeatureEncoder,
    engagement_encoder: PercentileFeatureEncoder,
) -> str:
    """Convert structured post data into an LLM-ready prompt string."""
    view_token = view_encoder.encode(post["view_count"])
    eng_token  = engagement_encoder.encode(post["engagement_rate"])
 
    return (
        f"format:{post['format']} "
        f"author:{post['author_name']} | {post['author_headline']} | {post['industry']}\n"
        f"reach:{view_token} engagement:{eng_token}\n"
        f"text:{post['text']}"
    )
 
 
# --- example ---
rng = np.random.default_rng(42)
 
corpus_views       = rng.lognormal(mean=7, sigma=2, size=50_000).tolist()
corpus_engagements = rng.beta(a=2, b=20, size=50_000).tolist()
 
view_enc = PercentileFeatureEncoder("view").fit(corpus_views)
eng_enc  = PercentileFeatureEncoder("engagement").fit(corpus_engagements)
 
post = {
    "format": "article",
    "author_name": "Jane Smith",
    "author_headline": "Staff ML Engineer",
    "industry": "Technology",
    "view_count": 12_345,
    "engagement_rate": 0.08,
    "text": "How we replaced five retrieval pipelines with a single LLM embedding model.",
}
 
print(build_post_prompt(post, view_enc, eng_enc))
# format:article author:Jane Smith | Staff ML Engineer | Technology
# reach:<view_percentile>71</view_percentile> engagement:<engagement_percentile>64</engagement_percentile>
# text:How we replaced five retrieval pipelines with a single LLM embedding model.

Apply this pattern any time you're feeding numerical signals — counts, rates, scores, durations — into an LLM as part of a prompt or embedding input. Raw numbers are a trap.

Training Strategy: Hard Negatives and the Positives-Only Insight

LinkedIn trained their dual encoder with InfoNCE loss on member-to-post engagement pairs. They used two types of negatives:

• Easy negatives: randomly sampled posts that were never shown to the member
• Hard negatives: posts that were shown to the member but received zero engagement

Hard negatives force the model to learn subtle distinctions — "relevant but not quite right" versus "genuinely valuable." Adding just two hard negatives per member in each training batch improved recall@10 by +3.6%, a significant gain from a deceptively simple change.

The second critical training insight was what to include in a member's engagement history. Initially, LinkedIn included all posts that were shown to the member — both those they engaged with and those they scrolled past. This degraded performance and wasted GPU memory (attention complexity is quadratic in sequence length).

Filtering to only positively-engaged posts improved the signal dramatically and had compounding efficiency benefits:

• 37% reduction in per-sequence memory footprint
• 40% more training sequences per batch with fixed GPU memory
• 2.6× faster training iteration due to reduced sequence length

The lesson is one that runs counter to the "more data is always better" instinct: training signal quality beats quantity, especially when you're dealing with expensive sequence-based models. The positives-only change improved both quality and cost simultaneously — a rare win.

Sequential Ranking: Your Feed as a Professional Story

Retrieval gets candidates into the pool. Ranking decides what you actually see. LinkedIn rebuilt this layer too, with a model they call the Generative Recommender (GR).

Traditional ranking models score each post in isolation: given a member profile and a candidate post, output a relevance score. This completely ignores temporal patterns — the fact that what you found interesting yesterday shifts what's relevant today, and that your professional journey over the past six months shapes what you're ready to engage with next.

The GR model processes more than 1,000 of your historical interactions as an ordered sequence, treating your engagement history as a professional narrative. Instead of asking "is this post relevant to this profile?", it asks "is this post the next logical chapter in this member's professional story?" It's the same intuition behind sequence-to-sequence LLMs applied to recommendation — and it changes what kinds of patterns the model can learn.

For your feed, this means the system can detect things like: you've been heavily engaged in operational content lately, and now something at the intersection of operations and your core expertise should feel like a natural next step — even if neither keyword appeared explicitly in either context.

Infrastructure: Disaggregating CPU and GPU at Scale

None of this is free. Running LLM inference in real time for a feed that has to respond in milliseconds, at 1.3 billion users, required serious infrastructure work.

LinkedIn's biggest architectural shift was disaggregating CPU-bound feature processing from GPU-heavy model inference — running each type of compute on the hardware it's suited for, rather than creating GPU bottlenecks waiting on CPU work. They also:

• Wrote custom C++ data loaders to eliminate Python multiprocessing overhead during training
• Built a custom Flash Attention variant to optimize attention during inference
• Parallelized checkpointing to reclaim GPU memory during training runs
• Maintained three nearline pipelines running continuously: prompt generation, embedding generation, and GPU-accelerated ANNS indexing — ensuring that new posts get embeddings within minutes of publication, not hours

The nearline architecture is elegant: every new post triggers prompt generation, then LLM inference to produce embeddings, then insertion into a GPU-accelerated approximate nearest neighbor index. By the time a member loads their feed, the freshest relevant content is already indexed and retrievable.

What This Means for Developers

You're probably not running at 1.3 billion users, but the architectural patterns here generalize broadly. Here's what's directly transferable:

1. Percentile-encode all numerical features. Don't pass raw counts, rates, or scores to LLMs. Convert them to percentile buckets with special tokens. It's a 30-minute change with potentially significant quality improvement.

2. Hard negatives punch above their weight. If you're fine-tuning any embedding model for retrieval, generating hard negatives from your actual impression data is worth the engineering effort. Two per positive sample was enough to move LinkedIn's needle by 3.6%.

3. Positives-only training history is better and cheaper. Filtering to engaged history only improves signal quality and cuts training costs. These two goals usually conflict — here they don't.

4. Treat user history as a sequence, not a set. If you're building a recommender, ranking with a model that understands temporal patterns will beat independent per-item scoring. The order matters.

5. Disaggregate your compute explicitly. If you're building nearline serving pipelines, be intentional about which work goes to CPU vs. GPU. Letting the GPU wait on Python data loading is silent performance death.

6. Multi-teacher distillation enables policy + behavior. LinkedIn's breakthrough was using one teacher model trained on product policy (what should be shown) and a second trained on click prediction (what gets clicked) and distilling both into a single student model. This pattern lets teams encode qualitative judgment alongside quantitative optimization signals.

Final Thoughts

What LinkedIn has published here is rare: a genuinely transparent engineering postmortem about replacing an entire class of infrastructure at production scale, complete with concrete numbers, failed approaches, and the specific insights that unlocked the wins.

The broader takeaway is that the "LLMs as retrieval engines" pattern is maturing fast. For years, LLMs were primarily generation tools — you prompted them and read the output. The LinkedIn architecture treats a fine-tuned LLM as a universal understanding layer: a model that can encode members, posts, jobs, or any other structured entity into a shared semantic space where relevance is a geometric relationship.

That's a fundamentally different paradigm from keyword search, collaborative filtering, or even older embedding approaches. And as LinkedIn's engineering blog makes clear, getting it to work at scale requires as much careful engineering discipline as it does ML sophistication.

The code you ship tomorrow probably doesn't serve a billion users. But the percentile encoding trick, the hard negatives strategy, and the positives-only training insight? Those work at any scale.

Sources: LinkedIn Engineering Blog · VentureBeat interview with Tim Jurka · VentureBeat: Why LinkedIn says prompting was a non-starter · LinkedIn large-scale retrieval paper