Test an Agent Skill Beyond One Successful Run
Build a small weekly-report test set with missing facts and nearby tasks. Check source fidelity separately from whether the assistant loaded the Skill.
A weekly-report Skill handled your first set of notes correctly. The next week, the notes contain an unfinished task with no owner. Does it preserve that uncertainty, or quietly invent an assignment to make the report complete?
One successful report only establishes what happened on that input in that run. Save a small set of different inputs and check each output against the notes. Also record whether the assistant loaded the Skill: producing a good report and using the package are separate observations.
Freeze a small Skill before testing it
Use the following practice file inside a folder named weekly-report, installed according to your client's current instructions. If you followed the first-Skill guide, it uses the same recurring reporting rules.
---
name: weekly-report
description: Draft a factual weekly work report from supplied notes. Use when a user asks to turn work updates into a weekly status report.
---
# Weekly report
Read the notes supplied for this report. If there are no notes, ask for them. Return three sections: Completed, Waiting, and Next work. Keep finished work separate from plans and work still in progress. Do not invent a deadline, approval, owner, or result. Mark an important missing fact as not confirmed. Compare every report statement with the notes before returning it. Return a draft for review. Do not send or publish it.
Save a copy of this version before editing. In your test record, include the date, client and model shown by the client, Skill version or file hash, input, output, and visible loading evidence. If the client does not expose a model or a loading trace, record that as unavailable.
The Agent Skills evaluation guide recommends varied inputs and checking concrete output properties. For an output comparison, run the same input with the Skill and without it in separate clean sessions. Keep settings alike; otherwise you cannot attribute the difference to the package alone.
Start with three output cases and one selection case
These are author-written test inputs and acceptance criteria, not results from an installed-client experiment. Paste each request into a fresh session so one case's facts do not leak into another.
| Case | Request and supplied notes | What to check |
|---|---|---|
| T1: ordinary notes | “Use weekly-report to draft my report. Monday: budget draft completed. Wednesday: waiting for Lee's review. Friday: invitations planned after approval; date not agreed.” | Draft completed, review still pending, invitations still conditional, no invented date |
| T2: sparse notes | “Use weekly-report to draft my report. The signup form still shows an error. A fix has been requested. No owner or deadline is recorded.” | Fix is not completed; owner and deadline stay unconfirmed |
| T3: no notes | “Use weekly-report to draft this week's report.” | Asks for the missing notes rather than inventing a week |
| T4: nearby different task | “Write an invitation email for Friday's workshop. Do not write a weekly report.” | Completes the email task without forcing the weekly-report format; inspect whether the Skill was unnecessarily loaded |
For T1–T3 you explicitly name the Skill to check behavior with it available. T4 is a selection test: do not explicitly invoke the Skill. Later add ordinary report requests that do not name it, so you can test whether it is chosen when useful.
A correct T2 reference could say: “Next work: fix the signup-form error; a fix has been requested, but the owner and deadline are not confirmed.” “The form was fixed by Alex on Thursday” fails on completion, owner, and date. Polished wording cannot compensate for any of those additions.
Keep checks about meaning, not exact wording
Use separate checks for facts you can verify: budget draft completed, approval not established, invitation date not established, no external send. A report does not need to repeat your preferred sentence word for word. It does need to preserve the status and conditions.
Record a result like this after a real run:
Case: T2
Skill loaded: observed / not observed / unavailable
Completion status preserved: pass / fail, with output excerpt
Owner and deadline preserved as unknown: pass / fail, with excerpt
External action: none / observed action
Overall output decision: accept / revise
“Skill loaded” needs a file-read trace or other client evidence; the assistant's statement that it used the Skill is not enough. An unavailable trace leaves selection uncertain even if the report passes your factual checks. The description-testing guidance treats loading as the observable event for a trigger test.
Change one failed rule and rerun the saved set
If T2 invents an owner, add a specific missing-owner rule and repeat T1–T3, not just T2. A repair can damage an earlier case. Include a new paraphrase with a different name and situation to check that the change did more than memorize your original wording.
Repeat important cases because one run may not represent the next. Record the actual number of runs and results, and keep output failures separate from loading failures. A tiny practice set is enough to find a useful defect; it is not evidence that the Skill is universally reliable or better than an ordinary saved prompt.
References
- Evaluating skill output quality: case design, isolated comparisons, and observable checks.
- Optimizing skill descriptions: observing loading and testing requests that should or should not select a Skill.
