Add Bmad-Method files and some brand-design artifacts!
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
# Description optimization: the trigger-eval loop
|
||||
|
||||
A skill's description is its only trigger. The router reads it, decides whether the user's request belongs to this skill, and either loads it or moves on. A description that is too narrow stays quiet when it should fire; one that is too broad fires on requests it cannot serve. This loop measures real firing against a held-out test set and improves the description until it triggers on what it should and stays silent on what it should not, without the improver ever overfitting to the cases it is being graded on.
|
||||
|
||||
The whole loop runs through the adapter, so "did the skill fire" means the skill-load event the runtime emits, defined in `references/platform-adapter.md`. No model name appears anywhere in this loop; the adapter forwards whatever a runtime needs.
|
||||
|
||||
## Step 1: generate the query set
|
||||
|
||||
Generate about twenty near-miss queries, roughly half that should trigger the skill and half that should not. The signal lives in the near misses, so make the should-not queries share keywords, domain, and phrasing with the should queries. A should-not query that obviously belongs to another skill teaches the description nothing, because any wording already handles it. The pairs that matter are the ones a careless reader would lump together: a request to build a workflow versus a request to debug an existing one, a request to write a brief versus a request to critique a brief someone already wrote.
|
||||
|
||||
Each query is a `{query, should_trigger}` record:
|
||||
|
||||
```json
|
||||
{ "query": "help me turn my deploy script into a reusable skill", "should_trigger": true }
|
||||
{ "query": "my deploy script keeps failing on the rollback step", "should_trigger": false }
|
||||
```
|
||||
|
||||
Aim for variety in surface form (casual speech, a pasted error, a one-line ask, a paragraph of context) so the description is tested against the shapes real requests arrive in, not one tidy template.
|
||||
|
||||
## Step 2: stratified 60/40 split
|
||||
|
||||
Split the queries into a train set and a test set, 60 percent train and 40 percent test, stratified so the should and should-not ratio is preserved in both halves. Stratifying matters because an unstratified split can land most of the should-not queries in one half and leave the improver blind to the false-positive problem on train, or leave the test set unable to detect it.
|
||||
|
||||
The split is fixed once at the start of the loop and never reshuffled between rounds, because reshuffling would let a query that exposed a weakness in one round hide in the train set the next. The improver works only from the train set. It never sees the test queries, their labels, or the test score, which is what keeps the loop honest.
|
||||
|
||||
## Step 3: measure real triggering
|
||||
|
||||
Run every query through the adapter with the current description in place, several times per query because firing is probabilistic. The trigger rate for a query is the fraction of runs that produced the skill-load event. Turn each rate into a verdict against a threshold (a query "triggers" when its rate clears the bar, for example more than half its runs loaded the skill), then score against the labels:
|
||||
|
||||
- a should-trigger query that triggered is a true positive,
|
||||
- a should-trigger query that stayed quiet is a false negative (the description is too narrow here),
|
||||
- a should-not query that triggered is a false positive (the description is too broad here),
|
||||
- a should-not query that stayed quiet is a true negative.
|
||||
|
||||
Score train and test separately. The train score and its per-query verdicts are what the improver sees; the test score is recorded but withheld.
|
||||
|
||||
## Step 4: improve from train failures, test blinded
|
||||
|
||||
Hand the improver the current description, the train queries with their labels, and the train verdicts, and ask for a rewritten description that fixes the train failures. False negatives mean the description needs to claim ground it is leaving uncovered; false positives mean it needs to draw a sharper boundary against the near misses it is wrongly catching. The improver works the train failures only and never sees a test query or the test score, so it cannot tune to the held-out set.
|
||||
|
||||
Also hand the improver the descriptions it already tried and why each fell short, so it tries something structurally different rather than nudging the same wording round after round. Without this, the loop tends to oscillate between two phrasings that each fix one failure and reintroduce the other. Feeding the history back pushes the improver toward a different cut of the boundary: reframing around intent instead of keywords, naming the adjacent skill the near misses belong to, or moving a qualifier from the trigger clause into the body.
|
||||
|
||||
Keep the description within whatever length and format bounds the runtime enforces (character cap, no angle brackets, and so on); a rewrite that triggers well but violates the bound is not a candidate.
|
||||
|
||||
## Step 5: re-measure and iterate
|
||||
|
||||
Apply the new description, re-measure train and test, and record both scores plus the description text for this round. Continue for up to five rounds. Stop early if train reaches a clean separation (all should fire, all should-not stay quiet) and the test score agrees, because more rounds past a clean split only invite overfitting.
|
||||
|
||||
## Step 6: pick the winner by test score
|
||||
|
||||
After the rounds finish, pick the description with the best test score, not the best train score. Train measures how well the improver fixed the failures it could see; test measures whether that fix generalizes to queries it never saw, which is the only thing that matters in production. When two rounds tie on test, prefer the one with the better train score as the tiebreaker, and failing that the shorter, sharper description.
|
||||
|
||||
Report the winning description, its test score, and the round-by-round trail (each description, its train score, its test score) so the choice is auditable and a human can override it. Log the trail to the run's memlog through `scripts/memlog.py` as the loop runs, one `event` entry per round capturing the description tried and the train and test scores, so a resumed or audited run reads the progression cleanly.
|
||||
|
||||
## Why each guard is here
|
||||
|
||||
| Guard | What it prevents |
|
||||
|---|---|
|
||||
| near-miss should-not queries | a test set so easy the description never has to draw a real boundary |
|
||||
| 60/40 stratified split | a split that hides the false-positive or false-negative problem in one half |
|
||||
| fixed split across rounds | a weakness escaping into the train set on a later round |
|
||||
| test score blinded from improver | the improver tuning its wording to the held-out queries |
|
||||
| pick by test score, not train | shipping a description that fixed the visible failures but does not generalize |
|
||||
| prior attempts fed back | the loop oscillating between two phrasings instead of finding a new boundary |
|
||||
@@ -0,0 +1,91 @@
|
||||
# Eval format and the four modes
|
||||
|
||||
A case is the unit of evaluation. Every case is `input + rubric + optional state_prefix`. The same case shape feeds all four modes; what changes is which invocations the runner sets up and how the result is judged.
|
||||
|
||||
## The case
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "create-1",
|
||||
"input": "I want a brief for InsuLens, a claims-triage tool for mid-market insurers. Notes are in evals/insulens/files/memo.md",
|
||||
"rubric": [
|
||||
"brief.md exists and its word count is between 250 and 1500",
|
||||
"brief.md names InsuLens and the mid-market insurer segment",
|
||||
"brief.md incorporates at least two specific points from memo.md without inventing claims absent from it"
|
||||
],
|
||||
"state_prefix": null,
|
||||
"files": ["evals/insulens/files/memo.md"]
|
||||
}
|
||||
```
|
||||
|
||||
Field semantics:
|
||||
|
||||
- `id`: stable identifier; used as the case's folder name in the run.
|
||||
- `input`: the realistic, messy user request. Use real file paths, company names, typos, and casual speech, because a polished input tests a situation the skill rarely meets. The runner sends this verbatim to the invocation, after prepending any `state_prefix`.
|
||||
- `rubric`: a list of named expectations, each gradeable to `{text, passed, evidence}` by the grader. The strong-versus-weak taxonomy below decides whether each one is worth keeping.
|
||||
- `state_prefix`: optional bracketed prime that places the skill mid-workflow (see below). Null or absent means the skill starts cold.
|
||||
- `files`: optional fixture paths staged into the case's clean working directory before the run. A bare filename lands at the workspace root; a nested path keeps its directory structure, so the input can reference it verbatim. Sources resolve against `--project-root`, then the cases file's directory, then as absolute paths.
|
||||
|
||||
For trigger cases the shape is lighter: a `query` and a `should_trigger` boolean, because there is no artifact to grade, only whether the skill fired. Those cases are covered in `platform-adapter.md` and `description-optimization.md`.
|
||||
|
||||
## state_prefix: turn simulation in one shot
|
||||
|
||||
Most multi-turn skills can be evaluated single-shot if the case is designed right. The `state_prefix` is the trick that makes mid-workflow points reachable without a multi-turn simulator. It is a bracketed prime prepended to the input that tells the skill where in its own flow this turn lands and what the user already said:
|
||||
|
||||
```
|
||||
[the skill has already worked through discovery; on turn 4 the user was asked about stakeholders and responded:] User said: "just me and a PM"
|
||||
```
|
||||
|
||||
The runner prepends the `state_prefix` to `input` and sends the combined text as a single message. One input then exercises any mid-workflow moment: a clarifying turn, a correction, a resume after an interruption. This replaces the deferred multi-turn simulator for everything except cases where the conversation arc itself is the deliverable.
|
||||
|
||||
Subjective skills (coaching, brainstorming, design facilitation) skip the rubric and rely on human judgment. The `state_prefix` still earns its place there, because it lets a human see the exact mid-run moment they want to judge.
|
||||
|
||||
## Strong versus weak expectations
|
||||
|
||||
The grader's job is easier and the result is more honest when an expectation is discriminating, meaning a wrong output cannot pass it. A weak expectation is worse than no expectation, because a green check on it reads as proof when it is noise. The grader flags weak expectations when it sees them; write them out of the rubric before they ship.
|
||||
|
||||
Weak patterns to avoid:
|
||||
|
||||
- Filename-only checks. "brief.md exists" passes for an empty file. Pair existence with a content check.
|
||||
- Wholly subjective phrasing. "the brief is high quality" cannot be graded. State the property concretely.
|
||||
- Tautologies. Anything that follows automatically from the prompt being understood proves nothing.
|
||||
|
||||
Strong patterns for artifact correctness:
|
||||
|
||||
- Specific facts that must appear, such as "incorporates at least two findings from section X."
|
||||
- Structural claims a wrong output would fail, such as "word count between 250 and 1500."
|
||||
- Negative assertions, such as "does not introduce content from unrelated sections."
|
||||
- Frontmatter checks, such as "frontmatter contains title, status, created (ISO 8601), updated."
|
||||
- Bounded output blocks, such as "the final message contains a JSON object with intent='create'."
|
||||
|
||||
Strong patterns for process discipline:
|
||||
|
||||
- Side-artifact existence paired with content, such as ".memlog.md captures the pricing decision with its rejected alternative and rationale."
|
||||
- Transcript tool-call patterns, such as "the transcript contains a call invoking bmad-editorial-review-prose."
|
||||
- Phase ordering, such as "the polish call occurs after the brief Write and before the final JSON block."
|
||||
- Read-only enforcement, such as "the input brief.md is byte-identical to the fixture and no Write or Edit targeted it."
|
||||
- Bidirectional fidelity, such as "every decision in the memlog is reflected in the brief, and no claim in the brief is absent from the input or the memlog."
|
||||
|
||||
Most process-discipline checks are deterministic reads of the transcript and filesystem, so the grader confirms them by quoting evidence rather than judging.
|
||||
|
||||
## The four modes in detail
|
||||
|
||||
### Baseline: skill versus bare model
|
||||
|
||||
Run the case input twice in parallel in the same turn, once wrapped by the skill and once against the bare model with nothing around it. The bare-model run is the long-term floor. The skill earns its existence only by producing something the bare model cannot, so when the skill stops beating the bare model the right call is retirement, not another patch. Use baseline when the user asks whether the skill is worth keeping, or as the release check.
|
||||
|
||||
### Variant: full versus stripped smallest-version
|
||||
|
||||
Run the full skill against a stripped smallest-version of the same skill (passed as `--variant-path`), or against a snapshot of the prior version for an edit, on the same input. This is the two-version comparison made runnable, and it settles the leanness scanner's defend-against-absence findings. If the two outputs tie on the dimension the section was supposed to protect, the section is decoration and gets cut. If the small version is materially and durably worse, the section earned its keep. Variant is how a suspected piece of ceremony gets a real verdict instead of an argument.
|
||||
|
||||
### Quality: output versus rubric
|
||||
|
||||
Grade a single config's output against the named rubric with the read-only grader in `references/grader.md`. The grader gives no partial credit, puts the burden of proof on a passing grade, and flags any non-discriminating assertion. Use quality when a rubric exists and the user wants to know whether the output meets it, independent of any comparison.
|
||||
|
||||
### Trigger and description
|
||||
|
||||
Generate near-miss should-trigger and should-not-trigger queries that share keywords, split them, measure real firing through the adapter, and improve the description across bounded rounds with the held-out scores blinded from the improver. The full loop, including the split ratio, the round bound, and feeding prior failed attempts back, is in `references/description-optimization.md`. Trigger detection itself is "did the skill load," abstracted per runtime in `references/platform-adapter.md`.
|
||||
|
||||
## Getting a skill to behave non-interactively
|
||||
|
||||
Single-shot modes need the skill to produce its deliverable without stopping to ask. Most multi-turn skills expose a headless flag or keyword that suppresses clarifying questions and ends with a structured status block. Trigger it from the input: the literal `Run headless.` at the start, a skill-specific keyword from the skill's own headless section, or enough context that no clarification is genuinely needed. The `state_prefix` also helps here, because a turn that already supplies the answer the skill would ask for keeps the run moving. If a skill has no headless path and the input cannot satisfy its questions, either add a headless mode to the skill or accept that this case needs a human in the loop.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Grader: LLM-as-judge contract
|
||||
|
||||
The grader inspects one case's captured transcript and artifacts and answers, per expectation, whether it held. It writes its verdict to `grading_path` so the grade lives in the case folder, not just in a subagent's reply. It is otherwise read-only against the run folder: it does not execute the skill, fix an artifact, or rerun anything; its only job is to judge what was produced and cite the evidence.
|
||||
|
||||
The grader has a second job that matters as much as the first: it critiques the rubric. A passing grade on a weak assertion is worse than useless, because it reads as proof while measuring nothing, so the grader flags assertions that a wrong output would also pass and names important outcomes that no assertion covers.
|
||||
|
||||
## Inputs
|
||||
|
||||
The grader receives:
|
||||
|
||||
- `case_id`: identifier for this case.
|
||||
- `input`: the message that was sent to the skill, including any prepended `state_prefix`.
|
||||
- `rubric`: the list of expectation strings it grades, each independently.
|
||||
- `transcript_path`: absolute path to the run's transcript, in the schema the adapter defines.
|
||||
- `artifacts_dir`: absolute path to the directory of files the skill wrote.
|
||||
- `grading_path`: absolute path where the grader writes `grading.json`.
|
||||
|
||||
## Process
|
||||
|
||||
1. Read the transcript. It is line-ordered events in the adapter's schema. Note the input that was sent, every tool call the skill made (with its name and arguments), the order those calls happened in, the final message (often a JSON status block for headless runs), and any errors.
|
||||
|
||||
2. List and read the artifacts. Walk `artifacts_dir` and open the files each expectation implicates. Read their contents rather than trusting filenames, and note modification times when ordering or read-only behavior is in scope.
|
||||
|
||||
3. Grade each expectation independently. Identify what kind of check it is and gather the matching evidence:
|
||||
|
||||
- Artifact existence + content ("brief.md exists AND names X") → open the file, read it, check the content matches; existence alone never passes a content claim.
|
||||
- Transcript tool-call patterns ("transcript contains a Skill call to X") → scan for `tool_use` events with the matching `name` and `input`; quote the matching event.
|
||||
- Phase ordering ("the polish call occurs after the Write and before the final JSON block") → find each landmark's line number or event index and verify the order.
|
||||
- Read-only enforcement ("input file is byte-identical; no Write/Edit targeted it") → compare content against the fixture AND scan the transcript for any Write/Edit whose `input.file_path` falls in the protected path.
|
||||
- Frontmatter checks → parse the frontmatter, verify each named field and its format.
|
||||
- Output-block checks ("final message contains a JSON object with intent='create'") → take the last assistant message's text, extract the object, check the field.
|
||||
- Bidirectional fidelity ("every decision in the log appears in the artifact AND nothing in the artifact lacks a source") → list claims on each side and trace both directions.
|
||||
|
||||
4. Decide pass or fail with specific evidence. Pass only when there is clear evidence the expectation holds and the evidence reflects substance rather than surface compliance, so a file that exists but holds only placeholders fails a content expectation. Fail when no evidence is found, the evidence contradicts the expectation, or the assertion is technically satisfied while the underlying outcome is wrong. Cite the evidence every time by quoting a line, naming a file with its path, or pointing to a tool call by its index and arguments.
|
||||
|
||||
5. Critique the rubric. After grading, surface assertions that look weak, meaning ones that passed but would also pass for a clearly wrong output, and name important outcomes you observed, good or bad, that no assertion checks. Keep the bar at what a rubric author would call a good catch rather than a nit.
|
||||
|
||||
6. Write the verdict to `grading_path` as `grading.json`, then summarize it in your reply.
|
||||
|
||||
## Output
|
||||
|
||||
`grading.json` holds one record per expectation plus a summary and rubric feedback:
|
||||
|
||||
```json
|
||||
{
|
||||
"case_id": "create-1",
|
||||
"expectations": [
|
||||
{
|
||||
"text": "brief.md exists and word count is between 250 and 1500",
|
||||
"passed": true,
|
||||
"evidence": "artifacts/insulens/brief.md, 487 words"
|
||||
},
|
||||
{
|
||||
"text": "the memlog references having ingested the memo as source material",
|
||||
"passed": false,
|
||||
"evidence": ".memlog.md exists but contains only the init entry; no mention of memo.md"
|
||||
}
|
||||
],
|
||||
"summary": { "passed": 1, "failed": 1, "total": 2, "pass_rate": 0.5 },
|
||||
"rubric_feedback": {
|
||||
"weak": [
|
||||
{
|
||||
"assertion": "brief.md exists",
|
||||
"reason": "Existence alone passes for an empty file; pair with a content or word-count check."
|
||||
}
|
||||
],
|
||||
"uncovered": [
|
||||
"The brief invented a competitor not present in the input or the memlog; no assertion would have caught this."
|
||||
],
|
||||
"overall": "Assertions check structure but not content fidelity in two places."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When `weak` and `uncovered` would both be empty, set them to `[]` and `overall` to `"No suggestions; the rubric looks discriminating."`
|
||||
|
||||
## Rules
|
||||
|
||||
- Verdicts come from evidence, not impressions, so quote, name files, and point to event indices.
|
||||
- No partial credit. Each expectation is pass or fail.
|
||||
- The burden of proof is on a passing grade, so when the evidence is uncertain the expectation fails.
|
||||
- Read-only against the run folder except `grading.json`. The grader never edits an artifact.
|
||||
- No silent defaults. If a file or the transcript genuinely cannot be read, mark the affected expectations failed with that as the evidence rather than guessing.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Platform adapter
|
||||
|
||||
Everything runtime-specific in the eval-runner lives here, behind one seam. The rest of the skill, the scripts, the case format, the grader, and the modes are written against this seam and stay platform-agnostic. No model name is hardcoded anywhere; a model is just a value the adapter forwards if a runtime needs one, never a list this skill maintains.
|
||||
|
||||
## The adapter config file
|
||||
|
||||
An adapter is a JSON file the scripts read. A working Claude Code adapter ships at `assets/adapter-claude-code.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "claude-code",
|
||||
"invocation": ["claude", "-p", "{prompt}", "--output-format", "stream-json",
|
||||
"--verbose", "--dangerously-skip-permissions"],
|
||||
"auth_env": "ANTHROPIC_API_KEY",
|
||||
"transcript": { "format": "stdout-jsonl" },
|
||||
"skill_dir": ".claude/skills",
|
||||
"load_signal": { "skill_tool": "Skill", "read_tool": "Read" },
|
||||
"env_passthrough": []
|
||||
}
|
||||
```
|
||||
|
||||
| Key | Required | Meaning |
|
||||
|---|---|---|
|
||||
| `invocation` | yes | argv template for one non-interactive run. `{prompt}` (alias `{query}`) is replaced with the composed input, `{cwd}` with the case's clean working directory. |
|
||||
| `auth_env` | no | name of the one env var the runtime reads for its credential. Forwarded from the host **only when set non-empty** — forwarding an empty string overrides the runtime's own credential fallback and breaks auth. |
|
||||
| `transcript` | no | `{"format": "stdout-jsonl"}` (default; stdout captured as the JSONL transcript) or `{"format": "file", "path": "transcript.jsonl"}` (runtime writes a file in the cwd). |
|
||||
| `skill_dir` | no | directory under the cwd where the runtime discovers skills. Default `.claude/skills`. Used to stage the skill under test and trigger mode's synthetic skill. |
|
||||
| `load_signal` | trigger mode | which tool calls count as a skill load: `{"skill_tool": "Skill", "read_tool": "Read"}` (the defaults). See trigger detection below. |
|
||||
| `env_passthrough` | no | extra host env var names to forward into the run, for runtimes that need more than the auth var. Empty unless a runtime forces it. |
|
||||
|
||||
### Discovery
|
||||
|
||||
`run_evals.py` and `run_triggers.py` locate the adapter in this order:
|
||||
|
||||
1. `--adapter <path>` on the command line.
|
||||
2. `BMAD_EVAL_ADAPTER` env var pointing at a config file.
|
||||
3. `adapter.json` or `.bmad-eval-adapter.json` beside the cases/queries file.
|
||||
|
||||
Nothing found means the run degrades to staging-only (cases prepared, results recorded as skipped). When the current runtime is Claude Code and no project adapter exists, pass `--adapter {skill-root}/assets/adapter-claude-code.json`.
|
||||
|
||||
## Invocation and isolation
|
||||
|
||||
The runner fills the invocation template with the input (any `state_prefix` already prepended) and the clean working directory, runs the command from that directory, and waits for completion. Before invoking, it stages into the cwd: the skill under test at `<cwd>/<skill_dir>/<skill-name>/`, and any case fixtures.
|
||||
|
||||
The subprocess environment is built from scratch, never inherited, so host shell config, memories, and tokens cannot bias the result. It contains exactly: `PATH`, a fresh empty `HOME` at `<case>/.home`, `CLAUDE_CONFIG_DIR` inside that HOME, the `auth_env` var when set non-empty on the host, and any `env_passthrough` keys present on the host. There is no container, no terminal emulation, and no credential file staging.
|
||||
|
||||
For a baseline run the runner issues the same command twice from the same input: once with the skill staged in the working directory and once with nothing staged, so the bare-model floor is measured under identical conditions. For a variant run it stages the full skill in one config and the `--variant-path` skill in the other.
|
||||
|
||||
## Transcript schema
|
||||
|
||||
The transcript tells `run_evals.py` where timing and token counts live and tells the grader how to read tool calls and the final message. The scripts read line-delimited JSON events: `assistant` events carry `message.content[]` items (a `tool_use` item has `name` and `input`; usage blocks carry token counts), and a `result` event's usage block is authoritative for totals. A runtime whose events differ needs its own accounting branch — that branch belongs here, behind the seam, not in a mode or the grader.
|
||||
|
||||
## Trigger detection: "did the skill load"
|
||||
|
||||
Trigger mode does not measure output; it measures whether the description caused the skill to fire. `run_triggers.py` stages a synthetic skill (unique name) in `skill_dir`, sends each query through the invocation command, and scans the transcript for a load. Each query runs several times because firing is probabilistic; the trigger rate is the fraction of runs that loaded the skill.
|
||||
|
||||
Only `tool_use` events count as a load: a `skill_tool` call whose input names the synthetic skill, or a `read_tool` call whose `file_path` falls inside the synthetic skill's directory (its SKILL.md). Whole-transcript substring matching is rejected outright, because the runtime's init event lists every discovered skill by name — a substring match would report 100% trigger rate no matter what the description says.
|
||||
|
||||
## Adding a runtime
|
||||
|
||||
Write an adapter file declaring the keys above; add `skill_dir` and `load_signal` if you want trigger mode. Add no model list and no provider branch anywhere else; if a value beyond these is needed, it belongs in the adapter, not in a script or a prompt.
|
||||
@@ -0,0 +1,55 @@
|
||||
# Self-improvement: the bounded auto-iterate loop
|
||||
|
||||
This is the loop that scans a skill, evaluates it, proposes a fix, applies it, and re-evaluates, repeating until the skill passes or a round bound is hit. It turns a single scan-and-fix pass into a closed loop that keeps going until the evidence says stop. It is the most autonomous mode the runner offers, so it carries the most guardrails: it is opt-in, calibrated to what is at stake, fully logged, and bounded.
|
||||
|
||||
The benchmark is a guardrail, never the judge. The human stays the judge. A green run means the change cleared the bar the loop was given, not that the change is correct, and the loop's job is to do the mechanical iteration a human would otherwise do by hand and then hand back a fix plus the evidence for it.
|
||||
|
||||
## When to run it, and how hard
|
||||
|
||||
The loop is opt-in. It never starts on its own, because applying changes to a skill in a loop is a stronger action than reporting findings, and the user decides when that is warranted.
|
||||
|
||||
Calibrate the aggressiveness to the stakes. A throwaway skill the user is still shaping can take a longer loop and a looser bar, because a wrong iteration costs little and is easy to throw away. A skill that other skills already depend on, or one that is shipped and in use, takes a short loop, a strict pass bar, and a close human read of every applied change, because a regression there propagates. Agree the round bound and the pass condition with the user before the first round, and write both into the memlog so the run is auditable against the terms it was given.
|
||||
|
||||
## The loop
|
||||
|
||||
Each round runs four beats:
|
||||
|
||||
1. Scan. Run the builder's scanners against the skill (the five lenses in `bmad-workflow-builder`: architecture, determinism, customization, enhancement, leanness), and collect the findings. On rounds after the first, scan again rather than trusting the prior scan, because the last fix may have moved something.
|
||||
|
||||
2. Eval. Run the modes that apply to this skill: quality against its rubric where one exists, variant to settle a leanness defend-against-absence finding, baseline to confirm the skill still beats the bare model. The scan says what looks wrong; the eval says whether it measurably is. A finding the eval cannot confirm is a candidate to note for a human, not to auto-fix.
|
||||
|
||||
3. Propose a fix. From the confirmed findings, propose one concrete change. Address the cause the finding names rather than the single case that exposed it (see generalizing, below). Keep the change small enough that the next eval can attribute the delta to it; a round that rewrites five things at once cannot tell you which one moved the score.
|
||||
|
||||
4. Apply and re-eval. Apply the proposed change, then re-run the eval from beat 2 and compare. A round that improves the score and breaks nothing else is kept; a round that regresses any mode is reverted before the next round, because an applied change that made things worse is not a base to build on.
|
||||
|
||||
Stop when the pass condition is met or the round bound is reached, whichever comes first. The bound is a hard stop: hitting it without passing ends the loop and reports the best state reached, it does not earn extra rounds.
|
||||
|
||||
## The full trail goes in memlog
|
||||
|
||||
Every round writes to the run's memlog through `scripts/memlog.py`, so the whole reasoning chain is on disk and nothing the loop decided is hidden in a model's head. Per round, log:
|
||||
|
||||
- a `decision` entry naming the fix proposed and the finding it answers,
|
||||
- an `event` entry recording the re-eval delta (which modes ran, the before-and-after score, what regressed if anything),
|
||||
- a `note` entry when a round is reverted, with why.
|
||||
|
||||
At the end, log a `direction` entry summarizing the final state, whether the pass condition was met, and what a human should still review. Because the trail is append-only and typed, a reviewer reads the run back in order and sees what was tried, what each attempt did to the numbers, and why the loop stopped where it did.
|
||||
|
||||
## Generalize to intent, do not overfit to the case
|
||||
|
||||
The failure that ends most auto-iterate loops is fixing the example instead of the cause. A case fails because the skill mishandled a class of input; patching the skill to special-case that one input passes the case and leaves the class broken, and often the patch is a hardcoded branch that makes the skill worse. Read each finding as a representative of an intent category and fix the category. A case where the skill invented a fact absent from the source is not "handle this memo," it is "the skill does not ground its output in the provided source," and the fix belongs at that level.
|
||||
|
||||
When a proposed fix reaches for ALL-CAPS ALWAYS or NEVER or a stack of MUSTs, treat that as a yellow flag, the same way the leanness scanner does. Shouting at the model is usually a sign the fix is patching a symptom; a sharper outcome statement or a small worked example generalizes where a louder rule does not. Prefer the version that explains the reasoning over the version that issues the command.
|
||||
|
||||
## Why each guard is here
|
||||
|
||||
| Guard | What it prevents |
|
||||
|---|---|
|
||||
| opt-in | a loop applying changes the user never authorized |
|
||||
| stakes calibration | the same aggressiveness on a throwaway and a depended-on skill |
|
||||
| eval confirms the scan | auto-fixing a finding the evidence does not support |
|
||||
| one change per round | a round whose delta cannot be attributed to a specific fix |
|
||||
| revert on regression | building the next round on a change that made things worse |
|
||||
| round bound | a loop that runs away instead of handing back to a human |
|
||||
| full memlog trail | reasoning that lives only in the model and cannot be audited |
|
||||
| benchmark as guardrail, human as judge | treating a green run as proof the change is correct |
|
||||
| generalize to intent | a hardcoded patch that passes the case and leaves the class broken |
|
||||
Reference in New Issue
Block a user