NeurIPS 2026 | More Skills, More Misrouting: Scaling Laws for Large Agent Skill Libraries

NeurIPS 2026 | More Skills, More Misrouting: Scaling Laws for Large Agent Skill Libraries

Give an Agent 10 skills, and it will almost never pick the wrong one. Expand the library to 500, without changing the model, the task, or the presence of the correct skill, and it begins to misroute with increasing frequency. More troublingly, it almost never invents a tool that does not exist, so the logs make everything look normal.

Our NeurIPS 2026 paper, The Scaling Laws of Skills in LLM Agent Systems, studies this phenomenon systematically. The experiments span 15 frontier LLMs, 1,141 real-world skills, and more than 3 million routing and execution decisions. The conclusion is that skill libraries have scaling laws of their own—laws that can be measured, predicted, and used as a blueprint for repair.

One-Minute Overview

• More skills make routing harder. Every time the skill library doubles, single-step routing accuracy falls by about three percentage points. All 15 models follow $Acc(N)=a-b\ln N$, with R² above 0.96.
• The right shelf, the wrong book. Ninety-four percent of routing errors occur within the functional cluster that contains the correct skill.
• 0% hallucination, 18.3% accuracy. As a query loses its anchors, more than 80% of requests are captured by real “black-hole skills” already in the library.
• Execution can rescue routing. State produced by a correctly executed upstream step gives the hardest downstream decisions roughly four times the gain seen on medium-difficulty decisions.
• The library can be fixed without changing the model. Law-guided library optimization raises routing accuracy from 71.3% to 91.7% and the ClawBench pass rate from 49.3% to 61.6%.

The Capability Is There, but the Agent Cannot Find It

Most previous research on skills asks whether a model can use tools or select the correct API. The skill library is usually treated as background configuration: a handful to a few dozen carefully chosen tools with clean boundaries.

Real Agent skill libraries, however, keep growing. Today they contain file operations, Python, and database queries; tomorrow they add Git, testing, search, and browsers; later come internal company tools, user-defined workflows, and new skills distilled by the Agent from past tasks. The library does not merely grow—it becomes crowded. Descriptions converge, functions overlap, boundaries blur, and several candidates often appear capable of completing the same task.

We therefore asked a different question: if the task remains unchanged and the correct skill is always available, what happens as the Agent faces more and more candidates? We scale the library size N from 10 to 20, 50, 100, 200, and 500, while extending multi-step pipelines from one step to ten. In every condition, the correct skill is guaranteed to be present, and the remaining slots are filled with domain-stratified distractors. Every failure observed here is therefore not a missing capability, but a capability that was visible and still not selected.

The 1,141 experimental skills come from public Agent Skills, Claude Code, MCP, and community GitHub repositories. After deduplication, they cover 14 software-automation domains. The 4,075 tasks were constructed by human annotators, with only unanimously accepted gold labels retained (κ = 0.91). The model set spans 15 systems, including GPT, Claude, Gemini, GLM, Kimi, Doubao, DeepSeek, and Qwen.

Law One: Doubling the Skills, Accuracy Drops by 3 %.

The first result is remarkably stable: single-step routing accuracy declines linearly with the logarithm of library size.

Acc(N) = a − b ln N

For all 15 models, the fit has R² above 0.96. On average, every doubling of the skill library costs about three percentage points of accuracy. The slope b varies by model and captures how quickly that model gets lost. Using 80% accuracy as a reference, the fitted curve for the strongest model, GPT-5.4, does not cross the line until roughly 1,200 skills, whereas 9 of the 15 models fall below it with fewer than 25 skills.

A three-point drop in a single step may sound manageable, but an Agent is a multi-step system. In a multi-step pipeline, losses do not merely compound; they are worse than a prediction based on independent repetitions of single-step routing. One wrong turn pushes the plan into the wrong semantic neighborhood. Stepwise accuracy is U-shaped: the first step benefits from a clear user intent, the final steps narrow back to a concrete goal, and the middle is most fragile. By then, the original intent has been diluted across several transformations, while the number of candidates that all look plausible as the next action is at its highest. An Agent completing a task does not mean it stayed on the correct path throughout.

The Problem Is Not Size, but Crowding

A deeper analysis shows that library size is only a coarse proxy. A Git skill does not compete with a database skill, and a PDF-processing skill is rarely confused with an npm package manager. The truly difficult choices are neighbors such as Git Merge, Git Rebase, and Git Conflict Resolver—skills that sit close to the correct answer in semantic space while retaining partially overlapping functional boundaries.

The data are direct: 94% of routing errors occur within the functional cluster containing the correct skill. The model recognizes whether a task concerns databases, code, or files, yet picks the wrong option among similar skills—like finding the right shelf and taking the wrong book. Our local Competition Index (CI) explains failures for individual skills (R² = 0.55) more than twice as well as library size N alone (R² = 0.26).

More counterintuitively, the most similar skill is not the most dangerous. Errors concentrate in the semantic-similarity “danger band” [0.55, 0.75): irrelevant candidates are easy to reject, while near-duplicates prompt the model to discriminate carefully. The hardest candidates are similar enough to look plausible, but not similar enough to trigger caution. This also means retrieval recall is not always better when it is broader. Expanding the candidate set from Top-20 to Top-50 often adds little useful information and many more competitors from the danger band.

A skill description is therefore not merely an introduction; it defines a decision boundary. Replacing name-only entries with descriptions that include constraints and examples reduces the routing decay slope b from 0.32 to 0.08. Yet longer descriptions are not always better. In a small library, counterexamples such as “do not confuse this with another skill” can clarify boundaries; in a large library, the same wording may bring competitors into view that the model would otherwise never have considered. A description strategy that works in a small library may not scale to a large one.

Black-Hole Skills: 0% Hallucination, 18.3% Accuracy

As we progressively remove concrete anchors from a query—file names, object names, tool-specific verbs, and explicit schemas—errors evolve in a fixed sequence: first local substitution between neighboring skills, then drift within the functional cluster, and finally a new failure mode, capture by black-hole skills.

A black-hole skill is a real library entry with an overly general description. Once query anchors disappear, it becomes a gravitational center in semantic space, pulling in requests that belong to other skills. Its formation is conjunctive: an ambiguous query alone is not enough, and a generic skill alone is not enough. The attractor emerges only when an ambiguous query and an ambiguous skill occur together. That combination is routine in real systems: benchmark prompts are often well specified, but real users are not; the boundaries among a few dozen hand-designed tools may be pristine, but those of a skill library that has evolved for months may not be.

The clearest evidence is numerical. Under weak anchoring, the hallucination rate is 0%, yet routing accuracy is only 18.3%, and more than 80% of requests are hijacked by skills already in the library. The Agent never selects a nonexistent skill: the call is valid, the format is correct, and the API may even return normally, but the Agent is consistently and earnestly wrong. We have treated skill hallucination as the primary risk of tool use; in a large library, the subtler danger is in-library hijack. No hallucination does not imply correct routing.

Law Two: Realized State Can Rescue Routing

That is only half the story. A real Agent executes a skill after selecting it, and execution produces concrete state: files, code, errors, database output, and test results. Once that state enters the context, how does the routing law change?

Begin with the cleanest case: a two-step pipeline in which no execution result is passed between steps. Across 1,690 skill pairs, joint success is almost exactly the product of the success rates of the two steps, P(A∩B)≈P(A)P(B), with a mean deviation below 0.01. Merely placing two skills in the same pipeline creates no synergy on its own.

What changes the picture is state realization. When an upstream skill executes correctly and its output is inserted verbatim into the downstream context, the next step no longer sees an abstract request to “handle a problem in this project.” It sees a specific failing test, a generated file, or an explicit API error. The resulting rescue follows a simple form: ΔP(B∣A)=2α(1−P(B))P(A),2α≈0.76

In other words, rescue is constrained jointly by upstream success and the downstream headroom that remains. Its distribution is especially revealing: harder downstream decisions are rescued more. The gain for the hardest tier is roughly four times that of the medium-difficulty tier. Easy decisions do not lack information in the first place; the truly difficult ones are those for which the original query alone does not reveal which skill should come next. Execution does not simply run a routing decision—it reshapes the next routing problem.

State can, of course, propagate errors as well. The effect of an incorrect upstream execution depends on the strength of the dependency between steps. Under strong dependency, the downstream step cannot avoid the bad input and performance falls by more than 15%. Under weak dependency, it can ignore the incorrect state and complete the original task directly, producing even a small 2.8% gain. Designing an Agent workflow is therefore not merely a matter of chaining capabilities; the dependency structure among skills determines how errors propagate.

One Parameter Connects the Two Laws

The routing law and the execution law appear separate, but they share one variable: the routing decay slope b. The larger b is, the faster a model becomes lost as the library expands. Fewer correct upstream skills then produce less correct state, leaving fewer opportunities for downstream rescue. Library size, routing collapse, correct state, and execution rescue thus form a single chain.

We do not refit the model with execution data. Instead, we substitute a and b from the routing experiments directly into the independently measured rescue equation. The resulting prediction is significantly correlated with the observed rescue gains across all 15 models (Spearman ρ = 0.74, p < 0.005), while the cross-model regression gives α=0.50−3.4b. The same library property determines both when an Agent begins to get lost and how effectively execution can recover it afterward.

Same Model, Better-Organized Skill Library: 71.3% → 91.7%

If these laws are real, they should guide system design in reverse. We therefore build a Law-Guided Auto Skill Manager that does not retrain the model or train a separate router. Instead, it diagnoses where the skill library is failing and applies mechanism-specific interventions: nearest-neighbor auditing and description-boundary rewriting for strongly competing skill pairs; prompt anchoring for cross-family drift; and removal or narrowing for overly general skills that may become black-hole skills.

We evaluate at N = 150, close to the collapse transition of a model with the median b, using GPT-5.4-mini as the router. Before optimization, we fix the model, tasks, gold labels, distractors, parser, and prompt template; prompt anchoring is enabled identically before and after optimization. On 1,600 predefined held-out tasks, held-out routing accuracy rises from 71.3% to 91.7%, while the in-library hijack rate falls from 22.4% to 4.1%.

The ablations align one-to-one with the earlier mechanism analysis. Rewriting the boundaries of high-confusion skill pairs alone adds 12.8 percentage points of accuracy; removing overly abstract skills alone adds 4.9 points. Combining the two yields additional synergy, consistent with the finding that a black hole requires both an ambiguous query and an ambiguous skill. Rather than designing a complex algorithm first and explaining it afterward, we observe the scaling behavior, identify the failure mechanisms, and derive interventions from those mechanisms.

The organized library also improves downstream end-to-end execution. With runtime context gating and closure checks, the average ClawBench pass rate rises from 49.3% to 61.6%, and ClawMark rises from 28.4% to 34.5%. A skill library is not merely an Agent configuration file; it is part of the capability stack itself.

More Skills ≠ Better Agent

When a model gains parameters, it does not have to explicitly choose the correct one from billions of parameters at every inference step. Skills are different: every added skill contributes both a capability and a routing candidate. Expanding a skill library therefore always introduces two opposing forces—capability growth and routing interference. The former lets an Agent do more; the latter makes it harder to find the right capability at the right time.

As Agents begin to generate, accumulate, and update skills automatically, a mature system may soon face hundreds or thousands of them. The questions will then change: Should a new skill enter the global routing pool, or be exposed only in certain domains? Which existing skill does it overlap with, and should they be merged or split? Which boundaries must be explicit in the description? Which skills have become semantic attractors? When should execution state be used, and when should it be isolated to prevent error propagation? Together, these questions point to an increasingly important discipline: skill-library governance.

Our conclusions are established for flat natural-language skill libraries. Learned routers, hierarchical libraries, retrieval architectures, and fine-tuning may change the exact slopes and transition points, but the qualitative logarithmic decay also reproduces beyond the libraries curated for our experiments. Agent scaling should not be judged solely by the foundation model; the capability system around it has scaling behavior of its own.

Scaling capability is one problem. Organizing capability is another.

---

Paper: The Scaling Laws of Skills in LLM Agent Systems (NeurIPS 2026) https://arxiv.org/abs/2605.16508 Authors: Evolvent AI Team