Do AI Agent Skills Help Weaker Models More? We Measured It

In an earlier post we made a claim and admitted we hadn't tested it: that our front-end agent skills should help weaker, cheaper models more than a frontier one. The logic was sound — handed-to-you knowledge is worth less to a model that already knows it — but logic isn't evidence. So we ran the experiment. This is what the numbers say.

The Setup

We re-ran the same build tasks with two weaker executors — Haiku 4.5 and Sonnet 4.6 — with the skills and without them, and graded with the identical instruments we used on the frontier model (Opus 4.8). The number that matters is the lift: how much the skill improves the output, measured at each model tier.

Two tests: the seven knowledge skills (graded on objective correctness), and the process "visual polish" skill (judged blind on quality).

Result 1: The Weaker the Model, the Bigger the Lift

On correctness, the pattern is clean and monotonic:

The skill pulls every tier up to 100% — but the weaker the model, the bigger the gap it closes. The lift roughly triples from the frontier model to the smallest one. The unaided weak models kept dropping the exact conventions the skills encode (a consistent sizing strategy, accessibility-respecting breakpoints, mobile-nav state, performance signals); the skill restored them every time. Hypothesis confirmed.

Result 2: Taste Transfers Down — Verification Doesn't

The quality skill was more interesting. Judged blind, the skilled output won 3 of 4 head-to-head pairs on the weaker models (the frontier model, running the full render loop, had won both of its pairs). The reviewer again flagged the unaided versions as the generic-AI default and the skilled ones as "custom-designed."

"The winning designs favor confident palettes and editorial type; the losing ones lean on the generic defaults — purple/indigo gradients, gradient headline text, uniform centered cards."

But the one loss is the most useful result in the whole study. The smallest model, even with the skill, lost one pair — not for looking generic, but for shipping a broken-contrast call-to-action (white text on a white background). That is precisely the kind of execution defect the skill's render-and-refine loop is designed to catch by looking at the rendered page — and a weaker, browserless model couldn't run that loop reliably.

So the skill's two halves behave differently as models get weaker: the shipped taste (a vetted design system, anti-pattern guardrails) transfers down-tier and keeps the output from looking generic. The verification (render, critique, fix) doesn't — it needs both a browser and enough capability to judge a screenshot.

What We Changed

The findings have a clear fix: if a weak model can't run the looking-and-refining loop, give it a verification floor that doesn't depend on the model's judgement. So we added to the polish skill:

• A dependency-free automated quality gate — a script that flags the deterministic defects (missing focus states, no reduced-motion guard, missing viewport tag, content hidden until JS, the generic-AI palette, flat shadows, obvious contrast failures) and must pass before shipping. In testing it correctly failed the generic baselines and passed the reference page.
• A scenario matrix — what verification to run for each combination of model strength and browser availability, with the automated gate as the constant that holds in every case.

The gate is a heuristic, not a renderer — it can't see true pixel contrast — so when a browser is available the full render check still runs. But it turns "I think it's fine" into a checked result regardless of how capable the model is.

What This Means for Teams

• Knowledge skills are a bigger win on cheaper models — arguably their best use case. If you run a small or budget model in production, well-built skills are where a lot of your quality comes from.
• Don't expect the same on a frontier model — it already knows the conventions, so a correctness skill barely moves it.
• Shipped taste travels; self-verification doesn't. A design system lifts any model's output; the looking-and-fixing loop needs capability and a browser.
• Give every tier an automated gate. It's how you stop a cheap model from shipping the bug it can't see itself.

How SplatDev Applies This

We build AI-enhanced development systems tuned to the model you actually run — frontier or budget — with the skills, guardrails, and automated gates that make the output reliable at that tier. And we measure it: where the gains are, where they aren't, and what to add to close the gap.

Running a cost-sensitive model and want more quality out of it?

Contact SplatDev — Build Better With AI

Or email us at contact@splatdev.com