Part III: Viewer-conditioned projection at billion scale

The first two posts made a diagnosis and tested it. Item recommendation works on a bipartite graph, social recommendation works on a unipartite one, and that difference pushes the viewer–candidate relevance matrix toward high effective rank, which is why the shared-embedding-and-attention playbook underfits and keeps underfitting no matter how many layers you put on top of it. The fix, if you take the diagnosis seriously, is to stop giving every candidate one fixed vector and instead make its representation depend on who is looking, to turn the candidate embedding from a noun into a verb.

That is easy to say and, at first glance, impossible to do. If every viewer gets its own way of seeing every candidate, the obvious implementation is a per-viewer projection matrix \(W_v\), and at a billion viewers that is on the order of sixty-four terabytes of parameters before you have trained a single one of them. So the real question is not whether to make the embedding viewer-conditioned; it is whether you can do it without the cost exploding. This post is the answer, and it has three moving parts: a candidate embedding that is deliberately viewer-independent, a per-viewer operator that is deliberately low-rank, and a serving trick that moves the entire operator over to the query side so the candidates never have to know it is there.

The division of labor

Write the score for a candidate \(c\) under viewer \(v\) as

with three pieces. \(E_c\) is the candidate’s embedding, and it is viewer-independent, it holds only what is stably true about \(c\). \(W_v\) is a \(d \times d\) operator that belongs to the viewer, and it is where all the viewer-dependence lives. \(a_v\) is a small learned query head. The point of writing it this way is the separation: everything that changes from viewer to viewer goes into \(W_v\), and everything that is a property of the candidate alone goes into \(E_c\). This is exactly what the bipartite-origin argument from the first post demands. In a unipartite graph the only thing stable about a node is its structure; the viewer-dependence has to be factored out into an operator rather than averaged into a shared vector.

The candidate embedding is a noun. In social recommendation it has to be a verb. The verb is \(W_v\).

What the candidate embedding actually is

If \(E_c\) is supposed to be viewer-independent, the natural question is what could possibly be stable about a person that does not depend on who is asking. The answer is the only thing a person has that an item does not have in richer form: the pattern of how they connect. A person has no stable content semantics in the way an item does, there is no genre or brand or price to bind to, so the viewer-independent part of the embedding has to be a structural fingerprint. Degree, triangle count, the community or communities they sit in, the distribution of small motifs around them, a snapshot of where personalized PageRank sends them. This is what it means, concretely, to say the embedding encodes edges rather than interests.

There are a few ways to build that fingerprint, in increasing order of cost and fidelity. The cheapest is a plain learnable identifier, one vector per person, randomly initialized, trained end to end, which lets the training signal discover the structure implicitly and scales to billions trivially because it is just a table. Slightly richer is to compute explicit structural features, in the style of role-discovery work, and project them down; richer still is a shallow one- or two-hop graph network over the neighborhood; and the most faithful is a substructure tokenizer, which I would not use here because, as the first post noted, it collapses into straight chains at the kind of degree the real graph has.

The version I would actually start with is the cheapest one, the random-initialization identifier, for a specific experimental reason: it isolates the contribution of \(W_v\). If the only viewer-aware component in the whole model is the operator, then any lift you measure is the operator’s lift, not the fingerprint’s, and you can ablate the structural features in afterwards to see what they add on top.

There is one correctness constraint worth stating plainly, because it is the kind of thing that looks like a detail and silently ruins the model. The candidate embedding cannot be random and frozen. It has to be random and trained. The reason is structural: as we will see in a moment, the serving trick moves \(W_v\) over to the query side, which means the per-viewer query no longer depends on \(c\), which means every scrap of information about \(c\) has to live in \(E_c\) itself. A frozen random table carries no information about the candidate, and a model built on it will learn nothing. The training signal has to be allowed to push the random initialization into something that encodes the candidate’s stable identity. Relatedly, \(W_v\) has to be a function of \(v\) alone and never of \(c\), not because that is fashionable, but because putting candidate information into the viewer-side operator breaks the serving pattern that makes the whole thing affordable.

Four ways to build that fingerprint, cheapest first:

The operator, kept low-rank

The expensive object is \(W_v\), and the move that makes it tractable is to never build the full \(d \times d\) matrix. Instead factor it as a low-rank residual,

This is the standard low-rank-adaptation form, and I want to be clear that the form itself is borrowed rather than invented; what matters is where it lands.[1] The identity term is what keeps the candidate’s base embedding intact, and the rank-\(r\) term is where the viewer does its re-projecting. Set \(r = 0\) and the whole thing collapses to \(W_v = I\), which is plain matrix factorization, a useful ablation origin, because it lets you read the lift from viewer-conditioning directly off the gap between \(r = 0\) and \(r > 0\). In practice a small \(r\), something like 4 to 16, is the working range; at \(d = 128\) and \(r = 8\) the operator is a couple of kilobytes per viewer rather than tens of kilobytes, and across a billion viewers that is the difference between something you can store and something you cannot.

Where the operator comes from

Even a couple of kilobytes per viewer, stored as an independent table, is multiple terabytes at a billion users, and it has a worse problem than size: a brand-new viewer has no factors until you have trained on them, so you have no cold-start. The fix is to not store the factors at all but to generate them. Keep one shared, small network that takes a viewer’s structural-and-activity fingerprint as input and produces \((U_v, V_v)\) as output. Now the per-viewer state is just the fingerprint, the same kind of object as the candidate embedding, and the mapping from fingerprint to operator is a single set of weights shared across everyone.

This is where the design stops being a pile of tricks and becomes one idea. Both the candidate embedding and the viewer’s operator are derived from structural fingerprints; the candidate’s fingerprint is read off as a static noun, and the viewer’s fingerprint is turned into a dynamic verb by the generator. A viewer who has just arrived gets a projection immediately, from their fingerprint, before they have generated a single training example. The actual intelligence of the system is concentrated in that one mapping, from how someone connects to how they see, and everything else is bookkeeping.

Stored factors versus a generated operator:

The trick that makes it serve

This is the part I find most satisfying, because it dissolves what looked like the hardest constraint. The score is \(\langle a_v, W_v E_c\rangle\), which reads as “project every candidate by this viewer’s matrix and then compare”, and if that were really how you had to serve it, you would be stuck re-projecting the entire candidate table for every viewer. But inner products are symmetric in a useful way:

Because \(W_v^{\top} = I + V_v U_v^{\top}\), computing \(z_v\) is \(z_v = a_v + V_v(U_v^{\top} a_v)\), which is a handful of cheap matrix-vector products, \(O(dr)\) per viewer, with \(r\) in the single digits. The operator has moved entirely to the query side.

The consequence is that the candidate table stays exactly what it was in any ordinary two-tower system: one static set of vectors, precomputed once, indexed for approximate nearest-neighbor search.[2] Per viewer you compute a single vector \(z_v\) and run a standard ANN lookup against the static table. None of the viewer-conditioned machinery ever touches the candidate side. This is not a new serving pattern, it is the pattern every large retrieval system already uses, but it is the reason the per-viewer operator costs nothing at serve time. You get viewer-conditioning for free, in the sense that it changes the per-request work from “re-project a billion candidates” to “compute one extra vector and do the lookup you were doing anyway.”

Does it scale

The accounting comes out clean. The candidate table at a billion users and 128 dimensions in half precision is around 256 gigabytes, which is large but ordinary for a production embedding store and fits comfortably in memory. The per-viewer query is a few thousand flops and is irrelevant to latency. The generator is a small network measured in megabytes. The approximate-nearest-neighbor index over a billion vectors is a solved problem at the scale of the systems that already do this for items.

The one place scale actually bites is not the model but the training data. The model itself is small, a table, a small generator, a query head, so the bottleneck is moving trillions of edges through it, not computing on them. That is the same edge-movement problem that every billion-scale graph system has, and it is handled the same way: subsample, correct for popularity bias, train data-parallel. The other quiet cost is the structural fingerprint, because the things that make it informative, triangle counts, personalized PageRank, are exactly the multi-hop quantities that are expensive to keep fresh on a graph this large. The constraint is to keep the fingerprint shallow, one or two hops, and recompute it incrementally rather than from scratch each refresh. With those two caveats, the answer to whether it scales to billions of users and trillions of edges is yes, and the caveats are the same caveats everyone in this regime already lives with.

The bill:

Why not just bilateralize the attention

The objection I hear most often at this point is that there is a simpler, more fashionable fix: take the sequence model and make its attention bilateral, so the viewer’s friending history attends to the candidate’s history and the candidate’s attends back. Friending is mutual, the argument goes, so model both directions and let attention sort it out.

I want to engage this seriously because it is half right. Bilateral attention does solve something real, which is reciprocity. A friend request is not satisfied when one side wants it; both sides have to consent, and a model that only predicts \(p(v \to c)\) misses the \(p(c \to v)\) half. Folding both directions in and combining them is a genuine improvement, and it belongs somewhere, specifically in the acceptance model, the downstream stage that asks “given we showed this candidate, will the connection actually form.” That is where it is the right tool.

What it does not do is replace the relevance operator, for the same reason ordinary attention could not, which the first post laid out. Attention is still a rank-\(d_h\) bilinear operator; making it run in two directions gives you two rank-bounded passes, not one high-rank one.[3] The mutual-friend signal, the thing that actually drives friend suggestion, does pass through bilateral attention, because a shared neighbor appears in both histories, but it passes through as one soft, anonymous co-occurrence term among hundreds of thousands of cross-terms, drowned out and projected into a low-rank similarity. It is not the hard, identity-specific, degree-weighted count that a feature like Adamic-Adar computes directly. And the deeper problem is that the whole bilateral formulation grew up on bipartite matching, dating, recruiting, where the two sides are genuinely different types and the topology fixes the roles. A social graph is unipartite; bolting a bilateral model onto it quietly re-imposes the bipartite assumption that the topology does not support, and it breaks in exactly the place where viewer-conditioning lives.

There is a clean way to settle the argument, and it is the same ablation I would run on anything in this space. If bilateral attention really captured the structural signal, then adding explicit graph features like Adamic-Adar and personalized PageRank on top of it should give roughly zero lift, because the signal would already be in the attention. I would bet a reasonable amount that it does not give zero lift, that the features still move the metric, by a meaningful margin, because the structural signal is not something attention discovers. It is something you compute and hand it.

What is actually new

I have been careful through this series to separate the diagnosis from the mechanism, and the mechanism deserves the same honesty, because most of it is not new. Per-user projection in recommendation is not new; McAuley and colleagues proposed learning a projection matrix for each user that takes the place of user-specific factors a decade ago.[4] Per-user low-rank adapters are not new; the recent recommender–LLM literature maintains a distinct adapter per user and routes between them.[5] The low-rank residual form is low-rank adaptation. The fingerprint idea is role discovery. The serving pattern is two-tower retrieval. The hypernetwork that generates the operator from features is a hypernetwork. None of these pieces, taken alone, is a contribution.

What is new, and what I would actually defend, is narrower and more useful than any single mechanism, and it is not a necessity theorem. The chain I would like to be able to write, unipartite therefore high-rank therefore per-viewer projection, breaks at the last link: the query-side identity above pins the score matrix at rank at most \(d\) for any \(r\), so an operator that collapses onto the query side is not the thing that answers a rank objection. What the unipartite topology does force is asymmetry, because one shared embedding per node and a symmetric dot product make the score of \((v, c)\) and the score of \((c, v)\) the same number, which is the wrong shape for a directed relation; per-viewer projection is one way to break that symmetry, and the cheaper way is already sitting inside this design, because \(a_v\) and \(E_c\) are two separate per-node tables and \(\langle z_v, E_c\rangle\) is asymmetric at \(r = 0\). The asymmetry argument therefore reaches \(r = 0\) and stops there, and I would rather say that than pretend the rank argument picked this operator out of the space. What is left, and I think it is enough, is the diagnosis and the assembly. The diagnosis is that the unipartite topology, not sparsity and not model capacity, is what breaks the item-recommendation playbook on people, because it puts the dominant signal, triadic closure, in a motif the bipartite topology is structurally forbidden from containing, and that signal is a function of the neighborhood around a pair rather than of the pair itself. The assembly is what makes the operator serve at a billion users: a viewer-independent structural candidate table, a generated low-rank operator, and the query-side move that keeps the candidate side untouched. And the operator’s own contribution is more mundane than a rank argument and still worth having: the viewer’s vector stops being a stored row in a table and becomes a generated function of the viewer’s own structure, which is where the cold-start, the generalization across viewers, and the storage that does not grow with the number of viewers come from. The diagnosis explains why the problem is different; the query-side factorization is why the response is affordable. That is the whole claim, and I would rather state it at that size than inflate it.

Reused versus novel:

Closing the floor

This is the last of the structural posts, and it is worth saying plainly where it leaves things. The floor is built: a candidate embedding that is a structural noun, a per-viewer operator that is a generated verb, and a serving trick that makes the verb free. It is also, as the second post insisted, a middle floor and not the top one. It does not replace the graph features; it does not express the full higher-order structure that a tensor or a hypergraph would; it does not get you out of the bilinear world. What it does is get the viewer’s side of that frame from structure rather than from a lookup table, in the specific regime of a billion viewers whose relevance function changes shape for each of them, at a cost that is dominated by the edge movement you were going to pay for anyway.

What is left, if you wanted to take it further, is the evidence, running the metric-versus-dimension curve on a real social graph and watching the baselines saturate where the viewer-conditioned model keeps climbing, and running the Adamic-Adar-on-top-of-attention ablation to settle the bilateral question with a number rather than an argument. That is a smaller and more empirical piece, and it is where I would point anyone who finished these three posts still unsure whether the diagnosis holds. The diagnosis is the part I am most confident in. The floor is the part I am most useful in. Whether the floor is as good as the diagnosis predicts is, in the end, an experiment, and experiments, unlike arguments, are easy to read.

Notes

[1] The low-rank residual form is from Hu et al., LoRA (2021). The form is borrowed; what this post claims is the placement, a per-viewer operator rather than a fine-tuning adapter.

[2] Static candidate side, online query side, with an ANN index: the two-tower retrieval pattern is the production standard at YouTube, Google, Allegro, and other large retrieval systems.

[3] The attention rank ceiling is Theorem 1 of Bhojanapalli et al., “Low-Rank Bottleneck in Multi-Head Attention Models” (ICML 2020), cited in the first post.

[4] See McAuley et al., “Improving Latent Factor Models via Personalized Feature Projection” (CIKM 2015): learn a projection matrix per user that takes the place of user-specific factors.

[5] See RecLoRA (2024): maintain a distinct LoRA per user and route between them with a meta-LoRA.