How to evaluate an AI marketing agent with your own work

When you're considering an AI marketing agent, a demonstration can help you picture the work it might take on. You might watch it turn a product brief into a launch email and a landing page, then imagine handing over the next campaign on your own calendar. The harder question comes afterward: would you use the result, and how much work would your team still need to do before it was ready?

That question is easier to answer with an assignment you know well. You know which product claims need evidence, what your customers care about, and what your team expects to receive at the end. A launch email might read smoothly while making a promise your product team has never approved. A landing page might look finished but arrive as an image that nobody can edit. When you review the work against a familiar brief, you can see these problems and judge how much effort they would take to fix.

An evaluation gives that review some structure. You choose a few assignments that reflect the work you would actually delegate, supply the context the agent needs, and decide what would make each result acceptable. You then keep the output and the review notes, including mistakes, missing files, corrections, time, and cost. Repeating the assignments helps you see whether a good first result holds up. Together, those records give you a basis for recommending a pilot and explaining where a person still needs to be involved.

This guide walks through that process using Northstar Relay, a fictional B2B SaaS company that sells an AI assistant for support teams. The company wants help with research, launch messaging, and campaign materials, so we'll follow one assignment in each area through to a pilot recommendation. Every example output, score, time, and cost is invented for teaching; no real agent or provider was tested for this article. You can use the same method and the downloadable sheet with your own work.

What an agent evaluation tells you

When an agent writes a launch email, the model is only one part of what determines the result. The product also decides how the agent reads your files, finds other information, uses tools, and asks for permission. Your setup affects the result too: an agent with the current product brief has a different starting point from one working from an old website page. An evaluation of the complete agent needs to include those conditions. A test of the model alone cannot tell you how well the whole arrangement will work for your team.

Amplitude's agent-evaluation guide makes this distinction in its vendor guidance. Independent WebArena research also evaluates whether agents complete tasks in a defined web environment. These sources help explain why the full task matters, though neither establishes how well an agent will handle your marketing work.

Benchmarks, demonstrations, feature lists, and copy samples can all help you decide what to investigate. A benchmark compares performance on its particular tasks. A demonstration lets you see a product being used, while a feature list helps you check whether it supports the work you need. A copy sample gives you something to judge for phrasing and tone. Your own assignment adds the context those materials can't supply: the sources you trust, the decisions your buyers are making, and the files your colleagues need to use.

Much evaluation guidance is written for developers. OpenAI's agent-workflow guide, for example, explains how to review records of an agent's actions and build repeatable tests. You don't need to start with that technical machinery to review a launch email. Start by checking the result against the brief. When something goes wrong, the record of the agent's actions can help you understand whether it missed an input, misread a source, or used a tool incorrectly.

If you need the mechanics first, read how AI agents work. Use the agent briefing guide to prepare the assignment, then use the evaluation sheet below to record how the agent handled it.

Choose representative marketing assignments

Choose assignments that recur often enough to matter and that someone on your team knows how to review. A recently completed piece of work can be a useful starting point because you already have the brief, sources, and an approved result to refer to. Include different kinds of difficulty: finding and interpreting evidence, making a persuasive argument, and producing files that another person can use. A strong result on one kind of work gives you a reason to test it further; it tells you little about the others.

Northstar Relay's audience is support operations leaders at mid-market software companies. Its approved product description says the assistant helps teams organize support knowledge and draft responses for human review. The company has no approved evidence for a percentage time saving, an autonomous resolution rate, or a security certification. Those limits let reviewers distinguish a supported product benefit from a promise the agent has invented.

The evaluation pack contains a product fact sheet, two approved customer interview excerpts, a voice guide, three past pages, and a list of claims that require substantiation. The voice guide asks for direct, specific writing without grand claims. Each assignment draws on this common material, with additional instructions for the result the team needs.

For research and synthesis, the agent compares three publicly available approaches to organizing support knowledge and writes a two-page decision memo. The memo needs to explain what the sources say, what that means for Northstar Relay, and what remains uncertain. Reviewers should be able to follow a material claim back to the passage that supports it.

A well-sourced memo can still contain a weak recommendation. The reviewer therefore also asks whether the agent has explained the tradeoffs and connected its advice to Northstar Relay's situation. A list of facts about three approaches is a useful input, but the assignment asks the agent to help the team make a decision. The competitor analysis guide explains how to keep that distinction clear when reviewing research.

For writing and messaging, the agent produces a launch email and landing-page outline for an approved product update. It receives the audience, fact sheet, voice guidance, and an explicit instruction to avoid quantified performance claims unless an approved source supports them. The reviewer checks the factual promises first, then considers whether the message explains something the buyer would care about and gives them a sensible next step.

This is also where you can distinguish familiar brand language from useful marketing. Copy may use the company's preferred words and still fail to explain why the update matters. Review it against the messaging framework as well as the voice guide, so that sounding familiar doesn't become the only standard.

For creative production, the agent delivers a campaign concept, two social images, a landing-page wireframe, editable copy, and a source list in an organized folder. Someone on the team should open the files in the software they would normally use and try a small edit, such as changing a headline or replacing an image. That establishes whether the handoff is usable, beyond what its preview suggests.

Review the visual work at its intended size as well. Text that looks fine when enlarged may be difficult to read in a social feed, and a layout may crop badly in its required format. Keep these craft judgments separate from the delivery requirement: a beautiful flattened image still fails this assignment if the requested editable copy or source file is missing.

View the three assignments at full size (opens in a new tab).

Prepare comparable conditions and record the setup

Prepare the source pack and review criteria before running the first assignment. Each candidate should receive the same objective, brand examples, deadline, required deliverables, prohibited claims, and standard of review. If the inputs contradict one another, say which source takes priority. Otherwise, you may end up judging how an agent guessed at an unresolved brief.

Keep a short record of what each candidate received and could access. Include the uploaded files, enabled tools, product and model labels shown to you, date, and manual setup. If a model name or product version isn't available, record that it wasn't disclosed. The purpose is to make the conditions understandable to someone who wasn't there when you ran the task.

Products don't all accept context in the same way. One may have a dedicated place to save brand guidance; another may expect files to be attached to the session. Use the setup each product needs, supply equivalent material, and record the time you spend doing it. Forcing the same sequence of clicks would make the comparison less useful if one product then lacked information that the other could read.

Identical prompts are useful when the question is how different models respond to the same prompt. When you're choosing a complete product, the more useful question is how it performs with a reasonable setup your team could repeat. Keep improvements visible: if you rewrite the brief after a weak result, record a new test condition. A fresh attempt with the original brief and a revision made after feedback answer different questions.

Use approved or synthetic inputs for the first trial, and give the agent access only to the material the assignment needs. Before using confidential information, check the data-handling terms, retention controls, connected-app access, and administrator permissions for your plan. Confirm the settings in the account you will actually use. A control described in general product documentation may depend on a plan or configuration you haven't enabled. OpenAI's business-data page and ChatGPT agent documentation provide examples of those distinctions.

Keep the initial assignments within research and draft creation. State that sending email, posting, changing settings, and publishing are prohibited, and use available permissions to restrict those actions. A written instruction is worth checking, but it should not be your only protection for a live account.

If you later need to evaluate an approval step, test it separately in a disposable destination after your organization authorizes the test. Review what the agent asks you to approve and whether the eventual action matches that request. Record who reviews the work and who can authorize the action. NIST's AI Risk Management Framework provides guidance on documenting the test context and human oversight.

View the test conditions at full size (opens in a new tab).

Separate critical failures from graded quality

Before looking at the outputs, agree which errors would prevent the team from using them. These are the pass/fail requirements, sometimes called hard gates. For Northstar Relay, a run fails if it invents or misstates a material product claim, cites a source that does not support such a claim, exposes confidential input, acts without authorization, or omits a required usable file. These failures need their own record because a strong score for tone or design cannot compensate for them.

Check the evidence for each requirement. A working citation link only proves that a page is available; you still need to read the passage and judge whether it supports the claim as written. If the record doesn't let you judge a requirement, mark it unknown and explain what would be needed to check it. Leaving an approval step untested, for example, is different from observing the agent ask for approval correctly.

For runs that have enough evidence to review, score four qualities from 0 to 4: evidence and accuracy (35%); strategic and audience usefulness (30%); brand fit and clarity (20%); and deliverable craft and completeness (15%). Zero means unusable on that dimension; two means substantial revision; four means ready for normal editorial review. These are illustrative weights chosen for Northstar Relay; choose yours according to the work you need.

For evidence and accuracy, look at whether claims are correct and whether you can trace them to their sources. For strategy and audience usefulness, consider whether the work helps the intended buyer understand an issue or make a decision. A recommendation needs to follow from the evidence and the buyer's situation; confidently worded advice is not enough on its own.

Brand fit covers the company's voice and the clarity of the writing. Keep it separate from strategic quality so familiar phrasing doesn't disguise a weak argument. Craft covers the finish and completeness of the deliverable, including visual hierarchy, readable text, and useful file formats. The scores should help a reviewer explain where the work needs attention, rather than replace the review itself.

Record the work involved as well as the result. Measure how long the agent takes to return its output, and separately record the time a person spends setting up, reviewing, and revising it. Note revision rounds, charges, approval requests, and unexpected actions. This lets you distinguish a fast first draft from a result that actually saves your team time. Say what each cost includes so that one candidate's subscription fee isn't being compared with another candidate's charge for a single run.

Leave room for questions the trial couldn't answer. Draft assignments won't show how a campaign performs with buyers, and a trial using synthetic material won't establish how the product handles confidential customer data. Microsoft Research's human-AI interaction guidance is useful when considering whether people can understand and control the system. The particular rubric and thresholds still need to reflect the work you're evaluating.

View the rubric at full size (opens in a new tab).

Copy the assignment record

Fill out the assignment record before starting, then attach a result record for each attempt. Keeping the original brief beside the output helps a reviewer distinguish a failure to follow instructions from a requirement that was never stated. If you change the objective, inputs, or pass conditions, note that change before running the task again.

Assignment ID and owner:
Objective and intended audience:
Inputs and source priority:
Brand context and examples supplied:
Required deliverables and file formats:
Allowed tools and destinations:
Prohibited actions and claims:
Critical failures to check:
Quality criteria and weights:
Evidence to preserve (sources, files, screenshots, action history):
Trial conditions (date, product, model, tools, permissions, setup, prompt):
Elapsed time / human setup / human review / revision effort:
Metered cost boundary and amount:
Reviewer 1 / Reviewer 2 / disagreement resolution:
Gate result / quality score / unknowns / decision:

Download the blank evaluation sheet (CSV). It keeps one row per run, so a reviewer can sort failed gates without losing the scores or conditions. The automation guide covers ongoing tasks and approvals after a limited pilot becomes appropriate.

A completed example: six synthetic runs

The table below shows how a completed evaluation record might look. It contains six synthetic runs, representing two fresh attempts at each of Northstar Relay's three assignments. “Agent A” and “Model M” are fictional labels, and every output description, gate result, score, time, and cost is invented for teaching.

The illustrative date is 2026-09-25 (synthetic). Each attempt uses the same source pack and constraints, allows draft-only file creation and public web research, and prohibits external publication. Approval requests are outside the example's scope, so approval behavior remains unknown in all six trials. The charges shown include only metered agent and tool use; they exclude subscription fees and human labor.

View the annotated synthetic failure at full size (opens in a new tab).

The W1 result shows why the pass/fail check comes before an overall judgment of quality. Its claim of “40% faster onboarding” has no support in the supplied material. The 4/4 brand and craft scores explain why an editor might initially like the draft, but the claim must be removed or substantiated before publication. The result remains a failed attempt even if an editor can repair it.

W2 gives the editor supported claims to work with, so the remaining work is closer to ordinary editorial revision. C1 needs a different repair: the missing editable copy and source file must be delivered. Keeping these reasons in the record tells the team what kind of supervision or rework each assignment may need.

How the weighted scores are calculated

The four quality weights add up to 100. To calculate a result, divide each rating by the maximum rating of 4, multiply by that quality's weight, and add the contributions. The formula is (evidence ÷ 4 × 35) + (strategy ÷ 4 × 30) + (brand ÷ 4 × 20) + (craft ÷ 4 × 15).

R1 has ratings of 3, 3, 2, and 3. Its calculation is 3/4×35 + 3/4×30 + 2/4×20 + 3/4×15 = 70/100. W1 has ratings of 0, 2, 4, and 4, giving 0/4×35 + 2/4×30 + 4/4×20 + 4/4×15 = 50/100, with a failed factual gate. Keeping the failure beside the number prevents a reader from treating 50 as permission to use the work.

The six synthetic scores are 70, 62.5, 50, 75, 62.5, and 62.5. Their sum is 382.5, so the mean across all six runs is 382.5 ÷ 6 = 63.75/100. Four of the six runs passed the observable requirements for these draft assignments. Approval behavior remains unknown because no approval step was tested. The four passing scores total 270, giving a pass-only mean of 270 ÷ 4 = 67.5/100. The second average describes the quality of the four passing outputs; it leaves out the two failures and must be read alongside them. Neither average tells you how often the agent would succeed across all your marketing work.

What the time and cost figures include

The synthetic elapsed total is 18 + 12 + 9 + 11 + 26 + 24 = 100 minutes across six runs. The human setup and review total is 7 + 8 + 11 + 6 + 13 + 9 = 54 minutes. These are different measures: someone may be doing other work while the agent runs, or reviewing one output while another is being produced. Adding the two totals would not give a meaningful measure of labor or end-to-end completion time.

Synthetic metered agent and tool charges total $1.20 + $1.05 + $0.70 + $0.75 + $2.60 + $2.40 = $8.70 for six runs, excluding reviewer labor and subscriptions. The example leaves revision time and the cost of rework unknown. In your evaluation, record both before concluding that a candidate is cheaper or faster to use.

View the repeated-trial interpretation at full size (opens in a new tab).

What the repeated attempts tell you

The two writing attempts used the same brief, yet W1 invented a performance statistic while W2 stayed within the facts. If the team had kept only W2, it would have missed a reason to retain careful claim review. If it had kept only W1, it would not have seen that the agent could produce a more useful draft under the same conditions. Keeping both makes the next pilot question clearer: can the agent stay within the approved claims across more assignments?

Two attempts cannot answer how often this error occurs. The chosen tasks may leave out harder topics or unfamiliar formats, and later runs may use different models, tools, or permissions. Record those conditions so you can distinguish a change in the setup from variation between runs. A small sample is useful for finding problems and choosing what to investigate next, as long as the recommendation stays within what you observed.

Help reviewers apply the rubric consistently

Before scoring the full set, ask two reviewers to assess the same few outputs independently. One might be responsible for product facts and another for audience and brand judgment. Have each person point to the sentence, source passage, or file behind a rating, then compare their reasons. This early discussion helps you discover whether the rubric gives them enough guidance to make comparable judgments.

Suppose one reviewer gives W2 a 2 for brand fit and another gives it a 4. Averaging them to 3 leaves the disagreement unexplained. One reviewer may be rewarding familiar language while the other is judging whether the message suits the audience. They may also be reading an ambiguous voice guide differently. Clarify the criterion, retain the original ratings, and explain how the reviewers reached the final score.

For a factual disagreement, return to the approved source and check the claim together. For a judgment about style or strategy, record any disagreement that remains instead of suggesting the final score is more precise than it is. An agent's self-assessment can help identify passages to discuss, but a person still needs to check the work.

Write a recommendation for the next pilot

The recommendation should say which assignments the evidence supports, what a reviewer must check, and what needs to be tested next. Someone reading it should be able to trace the recommendation back to the outputs and the review notes.

Northstar Relay's synthetic decision report recommends a supervised pilot for research memos and first drafts of messaging. A person would check citations and product claims before using either. Creative concept work could join the pilot if reviewers open every required file and confirm that they can edit it. Publishing, sending messages, and changing account settings remain outside the approved scope.

The report records two failed gates: W1's unsupported statistic and C1's missing deliverables. It also lists what the example cannot establish: confidential-data handling in the actual account, approval behavior with connected tools, performance on a wider range of assignments, campaign results, and costs after revisions.

The proposed pilot includes five more writing assignments drawn from different product updates, two creative handoffs with file-opening checks, and an authorized approval test in a disposable destination. A named reviewer would preserve the sources and output files, record revision time, and stop the pilot if the agent exposed confidential information or acted without authorization. The extra assignments would help the team judge those tasks; they would still be too few to establish a general failure rate.

A broader decision would need evidence from the additional work and permissions the agent would receive. Revisit the affected assignments when a material change to the product, model, tools, permissions, source material, or workflow could change the result. The record you've kept gives you a starting point for that review: you can repeat familiar tasks and see what has improved, what has become harder to use, and what still needs a person's attention. Once a pilot becomes part of the team's week, our guide to the five stages of an AI-native marketing team covers who owns the context, the review, and the recurring work.

Open the evaluation reference at full size (opens in a new tab) or download the image (opens in a new tab). Keep the completed sheet with the output files and reviewed sources so the person approving the pilot can inspect the work behind your recommendation.

Sources and further reading

• Amplitude: What is AI agent evaluation? — vendor explainer of task, trace, and human review; not evidence of marketing outcomes.
• OpenAI: Evaluate agent workflows — current provider guidance for traces, graders, and repeatable evaluation datasets; aimed at builders.
• WebArena research paper — independent benchmark of end-to-end web-agent tasks; its results do not measure marketing deliverables.
• NIST AI Risk Management Framework Core — context, measurement, human oversight, and limitations of generalization.
• Microsoft Research: Guidelines for Human-AI Interaction — human-centered design and control guidance.
• OpenAI business data and ChatGPT agent documentation — provider claims and plan-specific permissions and data controls; confirm the terms for any product you test.
• Google Search Central: people-first content, generative AI content guidance, and its generative AI search guide — useful checks when evaluating publishable content; none guarantees search visibility.

Frequently Asked Questions

What is AI agent evaluation?

AI agent evaluation means testing an agent on defined tasks to see whether its work is usable and how much correction it needs. The model is only one part of what gets tested. The product decides how the agent reads files, finds information, uses tools, and asks for permission, and the material you supply shapes the result as well. For marketing work, a familiar assignment makes the clearest test, because reviewers already know which claims need evidence and what the finished work should include.

What metrics are most important for evaluating an AI marketing agent?

Pass/fail requirements agreed in advance come first, because a high quality score cannot make up for a failed one. A run fails if it invents or misstates a material product claim, exposes confidential input, acts without authorization, or omits a required usable file. Score each reviewable run from 0 to 4 on evidence and accuracy, audience usefulness, brand fit, and craft. Then record the agent's elapsed time, human setup and review time, revision rounds, and charges.

Which marketing tasks should you use to test an AI marketing agent?

Use recurring assignments that reflect work you would actually delegate and that someone on your team already knows how to review. A recently completed project makes a good starting point because its brief, sources, and approved result are on hand for comparison. Cover different kinds of difficulty, such as finding and interpreting evidence, writing a persuasive message, and producing files a colleague can edit. A strong result on one kind of work tells you little about the others.

How should you test an AI marketing agent before deployment?

Test an AI marketing agent before deployment on research and draft assignments, giving it access only to the material each assignment needs. Prohibit sending, posting, publishing, and settings changes in the brief, and restrict those actions with the product's permissions too, because a written instruction shouldn't be your only protection for a live account. Test any approval step separately in a disposable destination once your organization authorizes it. A supervised pilot with a named reviewer and clear stop conditions comes next.

What is the difference between evaluating an LLM and evaluating an AI agent?

Evaluating a large language model, or LLM, tests how the model responds to prompts. Evaluating an AI agent tests the whole arrangement that produces the work: the model, the product that decides how the agent reads files and uses tools, and the material the agent receives. Identical prompts suit a model comparison. When you're choosing an agent product, the more useful question is how it performs when given equivalent facts and permissions through a reasonable setup your team could repeat.

What tools do you need to evaluate an AI marketing agent?

You can evaluate an AI marketing agent with an assignment record for each task, a spreadsheet that keeps one row per run, and the saved outputs, sources, and files; the guide includes a copyable record and a blank evaluation sheet. Developer evaluation tooling, built around records of an agent's actions and repeatable tests, isn't needed to judge a launch email against its brief. When a result goes wrong, though, the action record can show whether the agent missed an input, misread a source, or used a tool incorrectly.

How do you measure the ROI of an AI marketing agent?

An evaluation measures the cost and effort side of an AI marketing agent's ROI. Draft assignments won't show how a campaign performs with buyers, so a trial cannot establish the return. State what each recorded charge includes, so one candidate's subscription fee isn't compared with another's charge for a single run. Track the agent's elapsed time apart from the human time spent setting up and reviewing its work, and record revision time and the cost of rework before concluding that a candidate is cheaper or faster.

What privacy risks should you assess before testing an AI marketing agent?

Assess how the product would handle confidential information before an AI marketing agent sees any: check the data-handling terms, retention controls, connected-app access, and administrator permissions for your plan. Confirm the settings in the account you'll actually use, because a control described in general product documentation may depend on a plan or configuration you haven't enabled. Begin with approved or synthetic inputs, and count any exposure of confidential input as a failed run. A synthetic trial won't establish how the product handles confidential customer data.

What are the challenges in evaluating AI marketing agents?

Challenges in evaluating AI marketing agents include polished work that hides a disqualifying error, small samples, and reviewers who disagree. In the guide's fictional example, a launch page earns top brand and craft scores while inventing a performance statistic, so pass/fail checks must come before quality scores. Two attempts can reveal an error without showing how often it occurs. When reviewers disagree, compare their reasons and keep the original ratings instead of averaging the difference away.

Can AI agent benchmarks tell you which AI marketing agent to choose?

No. AI agent benchmarks compare performance on their own tasks, so they can help you decide what to investigate but can't show how an agent will handle your marketing work. WebArena, an independent benchmark, tests whether agents complete tasks in a defined web environment, and its results do not measure marketing deliverables. Your own assignments add what a benchmark lacks: the sources you trust, the decisions your buyers are making, and the files your colleagues need to use.