What are AI evals? How marketing teams prove AI work is getting better

When a marketing team uses AI every week, whether through a chat assistant or an AI marketing agent, the work gets judged the way any draft does. Someone reads the post or the email, fixes what's wrong, and decides it's good enough to go out. That settles the draft in front of them. It can't settle the questions that come up a few months in: are this quarter's drafts better than last quarter's, did last week's change to the prompt help or hurt, and will the work hold up when the provider replaces the model behind it? Each of those questions asks for a comparison over time, and a draft read on its own offers nothing to compare.

An AI eval is a repeatable test of AI work: a fixed set of real tasks, a written standard for good output, a way to score each result against that standard, and a record of the scores kept over time. Teams that build AI products use evals to test each change before it ships, but the method needs no code. A marketing team can run one with a folder of past briefs, a one-page rubric, and a spreadsheet, and get evidence in place of an impression: the same assignments, scored the same way, before and after every change.

The how-to sections follow the order in which a team builds the test. Fixed cases come first, because every later score depends on them. Next come a rubric and two reviewers who learn to apply it the same way, and then an AI judge, which earns a place only once its scores match those reviewers'. Later sections cover where AI judges go wrong, how to check for regressions when a model, prompt, or skill changes, and how to sample work that runs every week, with a worked example of a rerun and templates for each step at the end.

What an AI eval is made of

An eval has four parts, and each one protects the comparison from a different kind of noise. Anthropic's engineers, in a 2026 guide to evaluating AI agents, describe tasks with defined inputs and success criteria, graders that score each attempt, and a transcript that records everything that happened during the attempt. In marketing work, the four parts look like this:

Take any part away and the comparison stops meaning anything. Without fixed test cases, this month's scores describe different assignments from last month's, so a rise could simply mean the briefs got easier. Without a written standard, a score reflects whatever the reviewer happened to notice that day, the habit OpenAI's evaluation guidance calls vibe-based evals. Without a consistent grader, a change in the scores might come from a change of reviewer. And without a record, the next run has nothing to be compared with.

View the four parts of an AI eval diagram at full size (opens in a new tab).

An eval also differs from the one-off evaluation a team runs before it adopts an AI product. Our guide to evaluating an AI marketing agent walks through that process: a few representative assignments, repeated and reviewed, that end in a decision about a supervised pilot. A standing eval stays in place after the decision, and the team reruns it whenever something that shapes the output changes. The one-off evaluation answers whether to adopt something. The standing eval answers whether the work is still good.

Why one good AI result proves little

A single strong draft shows that the AI produced good work for that brief once. It doesn't show how often that happens or whether it will still happen next month, for three reasons.

The first is variation. The same brief can return a strong draft on one run and a weak one on the next, so a good result might be the typical result or the lucky one, and only repeated runs tell you which. Our guide to why AI gives different answers to the same question explains where the variation comes from and how to reduce it. An eval measures how much of it is left.

The second is that models change. In a 2023 study, Lingjiao Chen, Matei Zaharia, and James Zou tested the same model service in March and again in June and found that GPT-4's accuracy at identifying prime numbers had fallen from 84% to 51%, among other shifts in its behavior. They concluded that services like it need continuous monitoring. Providers also retire models. As of October 2026, Anthropic's deprecation policy gives at least 60 days' notice before it retires a publicly released model, after which requests to that model fail, and it advises testing with newer models well before the retirement date. A team that never chooses to change models will still have to, on the provider's schedule.

The third is that teams change things themselves. A new instruction in the prompt, a revised brief template, an updated brand guide, or a new version of the skill the AI follows can improve one kind of output and quietly damage another. Without a standing test, the team hears about the damage from a reader.

Anthropic's engineers report that teams with evals can adopt a new model in days, while teams without them face weeks of testing. For a marketing team, that is the difference between rerunning a known set of briefs and finding a new model's weak spots one published post at a time.

Five kinds of AI eval and what each one catches

Evals differ in who or what does the scoring and what each output is compared against. A standing eval usually combines several of the five kinds below, each where it works best.

The five kinds trade speed against judgment. Anthropic's documentation on building evaluations ranks grading methods the same way: checks written in code are the fastest and most reliable but lack nuance, human grading is the most flexible but slow and expensive, and grading by a language model is fast and suits complex judgment but should be tested for reliability before it is scaled. The same guidance prefers many automatically graded questions to a few graded by hand, which suits engineering teams testing software at volume.

Marketing quality is mostly a matter of judgment, so add the kinds in a deliberate order. People define the standard and apply it first. Automatic checks then take over what a machine can test without judgment, such as length, links, required sections, and claims your team has ruled out in its AI usage policy. An AI judge takes over judgment calls only after its scores match the people's scores on the same outputs.

How to build a golden set from your own marketing work

A golden set, often called a golden dataset, is a fixed collection of test cases that you score the same way every time the eval runs. Each case is a real brief, the material that came with it, and, where you have one, an approved output. A marketing team is well placed to build one, because the briefs and the approved work already exist.

Choose one job and gather its cases

Start with one recurring job your team already knows how to judge, such as the weekly LinkedIn post or the monthly customer newsletter. A set that mixes every kind of work can show that something got worse without showing which kind of work it was. Gather the job's recent briefs together with what was actually published. The published version, after a person's edits, is the closest thing you have to an approved output.

How many cases you need

Anthropic's engineers suggest that 20 to 50 simple tasks drawn from real failures is a good start, because early on each change has a large effect and a small set is enough to see it. For one marketing job, 20 to 30 cases is a practical first set: enough to include the hard cases below, and few enough for two people to score every output by hand while the standard settles. Add cases as failures teach you something.

Cover the cases that break

A set of easy briefs tells you little, because every version passes it. OpenAI's guidance recommends covering typical, edge, and adversarial cases, and Anthropic's engineers recommend testing the cases where a behavior should happen as well as those where it shouldn't. For marketing work, that means four groups:

• Typical cases are the everyday briefs, and they should make up most of the set.
• Hard cases include a thin brief, sources that conflict, a feature only some customers have, or a topic where a careless claim could cause trouble.
• Past failures are briefs where AI work went wrong before, such as an invented statistic or a post written for the wrong audience, kept so the mistake can't return unnoticed.
• Should-not cases are briefs where the right response is to ask, flag, or decline, such as a request for a customer figure the source material doesn't contain. A good output says the figure is missing. A bad one invents it.

Record what each case is for

Write every case down in the same shape: the brief exactly as it was given, the context and sources supplied with it, the approved output or a note on what a good output must contain, the gates that apply, and one sentence on why the case is in the set. That last line keeps the set honest, because a case nobody can explain is one nobody will notice has gone stale.

Treat the approved output as a guide rather than a template. Anthropic's engineers describe a reference solution as a known working output that passes every grader, but two excellent posts can share few sentences. The reference should show the points a good output covers, not the words it must use.

Keep some cases back, and keep the set current

Hold a few cases out of everyday use. If the person improving the prompt checks every change against every case, the prompt slowly fits those cases, and the set stops being a fair test of briefs the prompt hasn't seen. A handful of held-back cases, scored only when you make a decision, shows whether an improvement is real.

Give the set versions. When you add a case, retire one whose facts have gone stale, or change a reference output, note the date and the reason. An unrecorded change to the set looks exactly like a change in the work.

View the golden set diagram at full size (opens in a new tab).

How to write a rubric reviewers agree on

A rubric turns "is this good?" into questions two people can answer the same way. Susan Brookhart, reviewing rubric research in a 2018 paper for educators, describes a rubric as criteria that say what to look for, paired with descriptions of the work at each level of performance, and she recommends descriptive language over evaluative language for those levels. A level written as a judgment, such as "good use of evidence", gives the reviewer nothing to check. A level written as a description, such as "every claim matches a source supplied with the brief", can be checked against the draft.

Check the gates before you score anything

Some failures make an output unusable however well it scores elsewhere: an unsupported material claim, a claim your team has ruled out, a missing required part, the wrong format or length, or a broken required link. Our guide to evaluating an AI marketing agent calls these hard gates, and the same split works for evals. Check the gates first as pass or fail, record each failure on its own line, and never average a failed gate into a quality score. A well-written post with an invented statistic still can't be published.

Five criteria for marketing work

Score each output on five criteria, one at a time:

• Accurate: every claim matches its source. Our guide to preventing AI hallucinations sets out the claim check behind this criterion.
• On-brand: the writing follows your voice rules. Our brand voice guide shows how to turn voice traits into observable choices a reviewer can check.
• Specific: the work uses details that only this brief and its sources supply. Generic substance is one of the five signals of AI slop.
• Useful: the work does the job the brief set, for the reader it named.
• Ready to ship: the amount of editing that stands between this output and publication.

Brookhart calls a rubric that scores each criterion separately an analytic rubric, and one that gives a single overall score a holistic rubric. The holistic kind is quicker, but the analytic kind shows which quality changed, and that is what an eval needs to show. One overall score can say a post got worse. The criteria say whether it became less accurate or less like your brand.

View the gates and criteria diagram at full size (opens in a new tab).

Describe four levels for each criterion

Most of the rubrics Brookhart reviewed had four or five performance levels. Four suits marketing work: an even number removes the comfortable middle score, so every rating is a decision, and each extra level adds another boundary two reviewers have to agree on. Adapt the anchors below to your own rules.

Write anchors for the failures you actually see. If reviewers keep marking posts down for burying the call to action, that belongs in the anchors for "useful", in words a reviewer can check.

Expect to rewrite the rubric after the first round

In a 2024 study of people setting up AI judges, Shreya Shankar and colleagues named the reason criteria drift: people need criteria to grade outputs, but grading outputs is also how they work out what their criteria are. Score a first batch, rewrite the anchors that made reviewers hesitate, and give the rubric a version number. When a revision changes what a level means, rescore the earlier baseline with the new version, because scores from two versions of a rubric can't be compared.

How to calibrate two reviewers before you trust the scores

A rubric works only when two people using it reach the same scores, and the only way to know whether they do is to measure. Brookhart calls agreement between raters arguably the most important kind of reliability for a rubric, because matching work to a level description takes judgment. Anthropic's engineers apply the same test to the tasks themselves: a good task is one where two domain experts would independently reach the same pass or fail verdict.

1. Choose 10 to 20 outputs from the golden set, including some you expect to be weak.
2. Have two reviewers score them independently, one criterion at a time across all the outputs, with one sentence of evidence for each score.
3. Compare the scores and count the agreements.
4. Find the cause of each disagreement: an unclear anchor, a rule only one reviewer knows, or a real difference in judgment. Fix the anchor or add an example. Don't average two scores into a third, because the gap between them is information about the rubric.
5. Score a fresh batch and measure again, until agreement holds at the level you set in advance.

Measure agreement with two numbers

Percent agreement is the share of outputs on which the reviewers gave the same score. It is easy to misread, because two reviewers who pass almost everything will agree often by chance. Cohen's kappa corrects for that by comparing the agreement you observed with the agreement you'd expect by chance, on a scale where 0 means no better than chance and 1 means perfect agreement. Report both numbers.

In an illustrative calculation, two reviewers mark 20 outputs pass or fail on "ready to ship", counting a 3 or a 4 as a pass. Reviewer A passes 12 outputs and reviewer B passes 13. They both pass 11, both fail 6, and disagree on 3.

Eighty-five percent sounds strong, but the kappa of 0.68 is more modest, because two reviewers who each pass more than half the outputs would agree on about half of them by chance alone. Mary McHugh, in a 2012 review of the kappa statistic for health researchers, notes that many texts recommend 80% agreement as the minimum acceptable, and she treats any kappa below 0.60 as inadequate. On her scale, the illustrative 0.68 is moderate agreement. Those thresholds come from a field where a disagreement can mean a wrong diagnosis, and a marketing team can set its own. Set the threshold before you measure, and calibrate again when a reviewer joins or the rubric changes.

What LLM-as-a-judge is and how it works

LLM-as-a-judge is the practice of asking a language model to score or compare AI outputs against written criteria, in place of or alongside a human reviewer. The model doing the scoring is called the judge. For a marketing team, the appeal is arithmetic: a golden set of 25 cases, run twice on each of two models and scored on five criteria, produces 500 scores, more than anyone can read by hand after every change.

A 2023 study by Lianmin Zheng and colleagues set out three ways to use a judge:

• Single-answer grading: the judge scores one output on a scale. It scales most easily, but it can miss subtle differences between two outputs, and the authors note that absolute scores are likely to shift more than comparisons when the judge model changes.
• Pairwise comparison: the judge sees two outputs for the same brief and picks the better one or calls a tie. It picks up differences a scale would blur, but the number of possible pairs grows quickly as you add versions.
• Reference-guided grading: the judge grades each output against a reference answer. On math questions, where judges were often misled by the answers they were grading, having the judge write its own answer first and grade against it cut wrong verdicts from 14 of 20 to 3 of 20.

G-Eval, a 2023 method from Yang Liu and colleagues, has the judge write out evaluation steps from the criteria before it fills in a scoring form. Its authors also warn that judges built on language models may favor text written by language models.

Zheng's study also measured how closely a judge matches people. The strongest judge agreed with expert human voters 85% of the time on comparisons without ties, slightly more often than the experts agreed with one another (81%). Those comparisons, though, covered general questions on writing, reasoning, math, coding, and other topics, judged mostly by graduate students, not marketing work for a particular brand. A strong judge can approximate general human preferences, but that doesn't mean it knows your product facts or your voice. OpenAI's guidance, like Anthropic's, asks for a check first: use human feedback to calibrate automated scoring.

The documented biases of AI judges and how to counter them

Researchers who test judges keep finding the same ways they go wrong. In a 2025 study, Jiayi Ye and colleagues catalogued 12 types of judge bias and found that even judges with good overall performance showed significant biases on certain tasks. Each bias below has a countermeasure a marketing team can build into the way it runs the judge.

Three habits make any judge more dependable. Ask for the evidence and reasoning before the score, as Wang and colleagues did and as Anthropic's documentation recommends. Give each criterion its own judge call, the way Anthropic's engineers grade separate dimensions, so a strong impression of the voice doesn't leak into the accuracy score. And let the judge answer Unknown when the material doesn't settle the question, as Anthropic's engineers suggest, so a guess doesn't pass as a verdict.

View the six judge biases diagram at full size (opens in a new tab).

Two more options trade cost for reliability. A few examples of good judgments in the prompt raised the strongest judge's consistency in Zheng's study from 65.0% to 77.5%, though the authors note that consistency isn't accuracy and that the longer prompts made each call about four times as expensive. And in a 2024 preprint, an early result rather than a settled one, Pat Verga and colleagues found that a panel of smaller judges from different model families agreed with people better than one large judge did, showed less bias from any one model, and cost over seven times less.

When you can trust an AI judge's scores

Treat a judge like a new reviewer joining the team. It earns trust by matching the people who set the standard, on outputs they have already scored.

1. Take the calibration sample your two reviewers scored and use their agreed scores as the answer key.
2. Run the judge on the same outputs with the same rubric, one criterion per call, with evidence before each score.
3. Measure the judge's agreement with the answer key the way you measured the reviewers', with percent agreement and kappa, against a threshold set beforehand. A sensible bar is that the judge agrees with your reviewers about as often as they agree with each other.
4. Read every disagreement. Thakur and colleagues found that judges with high percent agreement can still assign very different scores, so look for patterns. A judge that scores one criterion higher than people do, or misses one kind of failure such as an invented statistic, needs a narrower job or a different model.
5. Check again whenever the judge's model or prompt changes. Absolute scores are likely to shift with a new judge model, so the judge you checked last quarter is effectively a new judge after an upgrade.

View the reviewer calibration and judge check diagram at full size (opens in a new tab).

For the judge itself, choose a capable model from a different family than the one that wrote the work, which lowers the risk of self-preference. Thakur's results also point to the strongest judge you can use, even when a lower-cost model writes the drafts. If no single judge passes your check on a criterion, test a panel or keep that criterion with people.

The judge's role ends before release. It can sort outputs, flag likely failures, and show people where to look, and that is how it saves review time. It shouldn't approve claims about your product, prices, customers, or results, because its documented weaknesses are exactly the ones that let a confident, well-written error through. A person checks every material claim before anything is published, whatever the judge scored.

Offline evals, online evals, and regression checks

An offline eval tests a setup before it reaches real work: you run the golden set on the new model, prompt, or skill and compare the scores with the current version's. An online eval checks work after it ships, through reviewers' scores on a sample of real outputs, how much they edit, how often drafts are accepted, and what happens when the work goes out. Each covers the other's blind spot. The golden set can't contain every brief the team will write, and live samples arrive only after a problem has reached real work. Anthropic's engineers describe the same layering, with automated evals run on each change and model upgrade, and monitoring and transcript reading once the work is live. OpenAI's guidance asks teams to run evals on every change and to grow the set over time.

Capability evals and regression evals

Anthropic's engineers also separate two purposes. A capability eval asks whether a setup can do something it doesn't yet do well, so it should start with a low pass rate and show progress. A regression eval protects what already works, so it should pass nearly every time, and any failure is a reason to stop and look. Cases graduate from the first kind to the second once they pass reliably. For a marketing team, the golden set's typical cases and past failures form the regression set, and a few stretch briefs for a format the team is still developing form a small capability set.

When to run a regression check

Rerun the regression set whenever something that shapes the output changes:

• a new model, whether you chose it or the provider is retiring the current one;
• an edit to the prompt, the brief template, or the instructions for a recurring job;
• a new version of a skill the AI follows, which our guide to AI agent skills explains is worth re-testing after a model update even when nothing in the skill has changed;
• new brand rules, product facts, or source material;
• a change to the tools or settings the AI uses.

If the change is a model switch, our guide to choosing an AI model by task covers which candidates are worth testing and how to weigh their cost. The regression check confirms that the one you chose didn't break anything.

How to compare the results

1. Run every case at least twice on both the old and the new setup. One run per case can't separate a weaker setup from an unlucky draft.
2. Compare the gates first. Any new gate failure stops the change until you know its cause.
3. Compare each criterion separately. A single average can hide a fall in accuracy behind a rise in voice.
4. Run pairwise comparisons on a sample, in both orders, to catch differences the scores blur.
5. Read the outputs and run records behind every failure and a few of the passes. Anthropic's engineers end their guide with the same advice, to read the transcripts: scores show where something changed, and the records show why.

A worked example: rerunning a golden set before a model change

This example is fictional, and every number in it is illustrative. Plinthwise makes booking software for physiotherapy clinics. Its two-person marketing team uses an AI agent to draft a weekly LinkedIn post for clinic owners from a short brief and a source pack: product notes, release notes, and a customer story the customer has approved for use.

Over six months, the team built a golden set of 24 cases from its own briefs: 14 typical posts, 4 hard cases (among them a feature available on only one plan and a brief whose release notes contradicted each other), 4 past failures, and 2 should-not cases in which the brief asks for a figure the source pack doesn't contain. Six of the 24 are held back from day-to-day prompt editing. The two reviewers had already calibrated, and the team's judge, a model from a different family than the one that writes the posts, had matched their agreed scores closely enough to clear the threshold the team set.

Then the provider announced that the model behind the weekly post would be retired in two months. The team picked a replacement and needed to know whether it was ready for this job and what had to change first. Each of the 24 cases ran twice on the current model and twice on the candidate, giving 48 outputs for each. Automatic checks tested length and the required link, the judge scored the five criteria one per call and compared each case's outputs in both orders, and a reviewer read every gate failure and a sample of the passes.

The averages, taken over the outputs that passed every gate, make the candidate look like an easy upgrade: better on voice and specificity, and preferred in more cases. The gates say otherwise. Both unsupported claims came from the same hard case, a post about reducing missed appointments, for which the source pack had no figure. The current model wrote around the gap and said the product helps clinics reduce missed appointments. In both runs, the candidate wrote that it cuts no-shows by 30%, a number nobody had approved.

The reviewer opened the run records to find out why. The brief template asked for a proof point in every post, and the source pack offered none for this topic. The current model had treated the instruction loosely, while the candidate followed it literally and filled the gap. The fault was in the brief template, and the new model had exposed it.

The team changed the template, not the model. It added a short list of approved figures to every source pack and an instruction to say in the review notes when no approved figure supports a point, instead of supplying one. Then it reran the hard cases, the should-not cases, and the six held-back cases on both models. The unsupported claims fell to zero, the other scores barely moved, and the held-back cases held up as well, which suggested the fix was real and not tuned to the cases the team had been looking at.

Plinthwise moved the weekly post to the candidate with the revised template, added a variant of the missed-appointments brief to its past failures, and logged the date, both models, the template and rubric versions, the scores, and the reason for the switch. Reviewers still check every claim before a post is published. Without the gates, the averages would have recommended the same switch with the invented figure still in it.

View the Plinthwise rerun example at full size (opens in a new tab).

How to evaluate recurring and scheduled AI work

Recurring work is where attention fades fastest. A weekly digest that gets a close read in its first month may get only a skim by its third, while sources move and product facts go stale in the background.

Before a job runs on a schedule, run it against its golden set and agree what a usable result looks like. Our guide to automating marketing with AI agents calls this writing the acceptance test and makes it the first step of putting an automation into production. Then score the first runs in full. Once several runs in a row pass cleanly, sampling can take over. For internal work such as that digest, our guide to human-in-the-loop review describes the change as moving from reading everything to checking each run's sources and changes and reading a sample in full. As that guide notes, the less often a person reviews each output, the more testing the setup needs first.

A workable sampling plan has three parts:

• automatic checks on every run, for the gates a rule can test;
• a full rubric score on a regular sample, such as one run in four, chosen in advance so the sample isn't only the runs that already looked odd;
• a full read of every run that failed, finished only partly, or was flagged for attention, because those runs are the most likely to hide a new kind of failure.

Track a few numbers across runs: gate failures, criterion scores, how much reviewers edit, and how often drafts are accepted without a rewrite. The automation guide recommends measuring review acceptance and corrections in place of counting completed runs, and the rubric gives those measures a standard to be judged against.

Use the activity record to find out why

Anthropic's engineers advise grading what an agent produced rather than the path it took, so that a good result reached by an unexpected route isn't marked down. The path still matters when something goes wrong. For marketing work, the useful parts of the record are the sources the AI opened, the files it read, what it searched for, what it changed, and anything it flagged as unverified.

When a score drops, open the record before you change anything. A post can lose accuracy because the model got worse, but also because a source page moved, an old file was read in place of the new one, or the brief changed. Each of those needs a different fix, and the record tells you which one you're looking at.

View the sampling plan diagram at full size (opens in a new tab).

How to turn eval results into fixes

An eval is worth running only if its results change something. Match each pattern in the scores to the place where the problem started, because a fix at the source repairs every later run, while a fix to one output repairs only that output.

Most of these fixes land in the brief or in the saved context the AI reads, not in individual outputs. As the automation guide puts it, if reviewers disagree about what good work looks like, the agent will inherit that disagreement. Changing the model won't settle it.

Feed every new failure back into the golden set as a past-failure case, retire cases whose facts no longer hold, and keep a short decision log with the date, what changed, the scores before and after, and what you decided. After a few months, the log shows which changes helped, which didn't, and which failures keep coming back.

Evals answer whether the work meets your standard. Whether that work wins customers, saves time, or earns its cost is a separate measurement with different evidence, which our guide to measuring the ROI of AI in marketing covers. Keep the two apart: a high eval score doesn't show business value, and good business results don't show that the work was accurate.

How golden-set scoring works with Ampere

Golden-set scoring is a way of working with Ampere, the AI marketing agent, rather than a separate feature, and each part of the method maps onto parts of Ampere you may already use.

Keep the eval in a Project. Upload the golden set, the rubric, and the scoring sheet as Project files, in formats such as Markdown, Word, Excel, or PDF. Project context files can be summarized and made available to Ampere in later sessions, so every rerun starts from the same cases and the same standard.

Ask Ampere to run or score the set. In a session, you can ask Ampere to draft each brief in the set, or to score a batch of drafts against the rubric one criterion at a time, evidence first, and fill in the scoring sheet as a spreadsheet in the session workspace. The session workspace shows the files, sources, and work in progress, so a reviewer can open what Ampere read before trusting what it scored. Scoring by Ampere is scoring by an AI judge, with the biases described earlier, so check it against your reviewers before relying on it, and remember that a model scoring its own drafts may favor them. If your Organization has a second model provider enabled, you can score in a session that uses a different model family from the one that wrote the drafts.

Rerun the set when you change models. The model selector describes each option by task fit and relative cost. Switching to another model in the same provider family keeps the session, while switching to another provider's model can start a new session and leave the original session's attachments behind, which is why the golden set belongs in Project files rather than in one session.

Bring in a second reviewer and watch recurring runs. Sessions can be shared with view, edit, or owner access, so a colleague can score the same outputs for calibration. A scheduled task that runs a saved prompt can be bound to a Project, and its run history can be filtered to runs that need attention, finished partly, or failed, which are the runs a sampling plan reads in full. For LinkedIn text posts, polls, and newsletters, Ampere's packages can include review notes and claim checks, which give the accuracy check a place to start.

Ampere-maintained marketing skills are skills evaluated against defined task, tool, and output expectations. That is Ampere testing the methods it ships, not a test of your briefs, your voice, or your standard, which is the job of your golden set. AI outputs may be inaccurate, incomplete, or non-unique, so people review the work, and nothing is published without you.

Templates: golden-set case, rubric, scoring sheet, and judge prompt

Copy these into a document or spreadsheet and adapt the wording to your own work. Each one matches a step earlier in this guide. Start with the record for a single case:

Case ID: (for example, POST-07)
Group: typical / hard / past failure / should-not
Held back from everyday prompt editing: yes / no
Why this case is in the set (one sentence):

Brief, exactly as given:

Context and sources supplied with the brief:

Approved output, or what a good output must contain:

Gates that apply:
[ ] Every material claim matches a supplied source
[ ] No claim from the ruled-out list
[ ] Required parts present:
[ ] Length and format:
[ ] Required links work

Added on:
Last reviewed:
Set version:

The rubric repeats the gates and anchors from earlier in a form you can paste into a shared document and edit:

RUBRIC, version 1, for: (name of the job)
Date:

GATES: pass / fail / unknown. Check these before scoring anything.
Record each failed gate on its own line. Never average one into a score.
1. Every material claim matches a source supplied with the brief.
2. No claim from the ruled-out list appears.
3. Every required part is present.
4. Length and format meet the brief.
5. Required links work.

CRITERIA: score each one separately from 1 to 4.
Write one sentence of evidence for each score.

Accurate
4  Every claim, number, name, and date matches a supplied source.
3  One small slip, such as a rounded figure, that changes no claim.
2  A claim stretches its source, or a source is cited for something it doesn't say.
1  A claim has no source or contradicts one. This also fails gate 1.

On-brand
4  Follows every voice rule; a regular reader would recognize us.
3  Follows the rules, with a phrase or two a reviewer would change.
2  Breaks a voice rule more than once, or uses wording we rule out.
1  Could have come from any company.

Specific
4  Uses details only this brief and its sources supply.
3  Mostly specific, with one generic passage.
2  Several passages would fit any company's post.
1  Swapping in another company's name would change nothing.

Useful
4  Gives the named reader something to use or decide, in line with the brief's goal.
3  Serves the goal, but the main point arrives late or a step is missing.
2  On topic, but doesn't serve the goal or the named reader.
1  Misses the goal or addresses the wrong reader.

Ready to ship
4  Publishable after a proofread.
3  Needs light line edits.
2  Needs a section rewritten.
1  Needs to start over.

The scoring sheet keeps one row per output, including the failures, and a short summary for each setup you compare. Its first line is the spreadsheet's header row: paste it into the first cell of a new sheet and split it into columns at the commas.

Run date,Case ID,Run (1 or 2),Setup (model / prompt version / skill version),Grader (reviewer or judge),Gate 1,Gate 2,Gate 3,Gate 4,Gate 5,Accurate,On-brand,Specific,Useful,Ready to ship,Evidence for each score,Notes

ROWS: one per output. Keep every row, including failures.
Gates: pass / fail / unknown. Criteria: 1 to 4.

SUMMARY FOR EACH SETUP
Gate failures (count, with case IDs):
Average for each criterion, over outputs that passed every gate:
Pairwise wins, losses, and ties (a win counts only if it holds in both orders):

AGREEMENT BETWEEN TWO GRADERS (pass = 3 or 4 on the criterion)
N = outputs both graders scored
Percent agreement = outputs with the same verdict / N
Chance agreement = (A passes / N) × (B passes / N) + (A fails / N) × (B fails / N)
Kappa = (percent agreement - chance agreement) / (1 - chance agreement)
Threshold, set before measuring:

DECISION AND REASON:

The judge prompt scores one criterion per call, asks for evidence before the score, checks claims only against the sources you supply, and allows an Unknown answer. Write a version for each criterion by swapping in its levels:

You are scoring one marketing draft on one criterion.
Score only this criterion. The other criteria are scored separately.

Criterion: Accurate
Levels:
4 = Every claim, number, name, and date matches a source below.
3 = One small slip, such as a rounded figure, that changes no claim.
2 = A claim stretches its source, or a source is cited for something it does not say.
1 = A claim has no source or contradicts one.

Brief:
(Paste the brief.)

Sources supplied with the brief:
(Paste the sources.)

Draft:
(Paste the draft.)

Instructions:
1. List every factual claim in the draft.
2. Under each claim, quote the source passage that supports it, or write "no supporting source".
3. Judge claims only against the sources above.
4. A citation, quote, or link inside the draft is not evidence unless the sources support the claim.
5. Length is not a quality. Do not reward a draft for being longer.
6. If the sources do not let you decide, answer Unknown and say what is missing.
7. Write your evidence first. Put the score alone on the last line.
8. Write the score as one of: Score: 4, Score: 3, Score: 2, Score: 1, Score: Unknown.

For a pairwise comparison, give the judge both drafts and ask which better meets the criterion, or whether they tie. Then run the comparison again with the drafts in the opposite order, and count a win only when both runs agree.

Sources and further reading

• Lianmin Zheng and colleagues, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track. Ways to use a judge, position and length bias, and agreement with experts.
• Peiyi Wang and colleagues, Large Language Models are not Fair Evaluators, ACL 2024. Answer order and how to correct for it.
• Arjun Panickssery, Samuel R. Bowman, and Shi Feng, LLM Evaluators Recognize and Favor Their Own Generations, NeurIPS 2024. Self-preference.
• Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori Hashimoto, Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, COLM 2024. Length bias.
• Jiayi Ye and colleagues, Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge, ICLR 2025. Twelve judge biases, including authority bias, tested with invented citations.
• Aman Singh Thakur and colleagues, Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges, GEM workshop at ACL 2025. Limits of percent agreement, and leniency.
• Pat Verga and colleagues, Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models, preprint, April 2024. Panels of judges.
• Yang Liu and colleagues, G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment, EMNLP 2023. Evaluation steps before a scoring form.
• Jie Huang and colleagues, Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024. Self-correction without outside feedback.
• Shreya Shankar and colleagues, Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences, UIST 2024. Criteria drift.
• Lingjiao Chen, Matei Zaharia, and James Zou, How is ChatGPT's behavior changing over time?, arXiv, 2023 (version 3, October 31, 2023, which reports the figures cited here). Behavior changes in one model service over three months.
• Anthropic, Model deprecations, Claude documentation, read October 13, 2026. Notice before a model is retired.
• Anthropic, Define success criteria and build evaluations, Claude documentation, read October 13, 2026. Grading methods and model-based grading.
• Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, and Jiri De Jonghe, Demystifying evals for AI agents, Anthropic Engineering, January 9, 2026. Tasks, graders, transcripts, and capability and regression evals.
• OpenAI, Evaluation best practices, OpenAI API documentation, read October 13, 2026. Eval sets, judge types, and calibration.
• Susan M. Brookhart, Appropriate Criteria: Key to Effective Rubrics, Frontiers in Education, April 10, 2018. Criteria, level descriptions, and agreement between raters.
• Mary L. McHugh, Interrater reliability: the kappa statistic, Biochemia Medica, 2012. Percent agreement and Cohen's kappa.

Frequently Asked Questions

What are the different types of AI evals?

The main types of AI evals are human review, rubric scoring, pairwise preference, reference checks, and automatic checks. Human review sets the standard and catches problems that take judgment. Rubric scoring tracks each quality across runs, pairwise preference compares two versions of the same task, reference checks compare output with approved facts, and automatic checks test rules such as length and links on every run.

How do you set up AI evals?

Set up AI evals by choosing one recurring job and collecting 20 to 30 real briefs with their sources and approved outputs, including hard cases and past failures. Write a rubric with pass/fail gates and described score levels, calibrate two reviewers on the same sample, and add an AI judge only after it matches them. Then rerun the set after every change.

What is LLM-as-a-judge?

LLM-as-a-judge is the practice of asking a language model to score or compare AI outputs against written criteria, in place of or alongside a human reviewer. The judge can grade one output on a scale, pick the better of two, or grade against a reference answer. It makes scoring at volume practical, but its scores need checking against human reviewers first.

How do you evaluate AI-generated content?

Evaluate AI-generated content with a written rubric instead of an overall impression. Check pass/fail gates first, such as unsupported claims and missing required parts, then score each quality separately, such as accuracy, brand voice, and usefulness, against described levels, with a sentence of evidence for each score. AI-content detectors don't help, because they estimate whether a machine wrote a text, not whether it meets your standard.

What is a golden dataset in AI?

A golden dataset is a fixed collection of test cases scored the same way every time an eval runs, so results from different runs can be compared. For a marketing team, each case is a real brief, its sources, and an approved output or a note on what a good output must contain. For one recurring job, 20 to 30 cases is a practical start.

Is LLM-as-a-judge robust?

Not on its own. Studies of LLM-as-a-judge have found judges favoring the answer in one position, longer answers, their own writing, and answers with invented citations. Running comparisons in both orders, scoring one criterion per call, asking for evidence first, and checking the judge against calibrated reviewers make it dependable enough to sort and flag work, but not to approve claims customers will read.

Which LLM should you use for LLM-as-a-judge?

For LLM-as-a-judge, use a capable model from a different family than the one that wrote the work, ideally the strongest available, even if a lower-cost model writes the drafts. A different model lowers the risk of the judge favoring its own writing, and one study found that only the best and largest judges came reasonably close to human scores. Check any judge against your reviewers first.

Can AI evaluate its own work?

AI can evaluate its own work, but its scores are a weak check. One study found that judges favor their own writing over text people rated as equally good, and another found that models struggle to correct their own reasoning without outside feedback. For scores you rely on, use a different model checked against human reviewers, and have people approve every material claim.