chore: upgrade BMAD framework to skill-based architecture
Migrates from slash-command files (.claude/commands, .cursor/commands) to the new skill-based architecture with workflow step files.
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-teach-me-testing
|
||||
description: 'Teach testing progressively through structured sessions. Use when user says "lets learn testing" or "I want to study test practices"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -60,7 +60,7 @@ Create an ongoing learning companion that teaches testing progressively through
|
||||
## Classification Decisions
|
||||
|
||||
**Workflow Name:** teach-me-testing
|
||||
**Target Path:** {project-root}/src/workflows/testarch/teach-me-testing/
|
||||
**Target Path:** {project-root}/src/workflows/testarch/bmad-teach-me-testing/
|
||||
|
||||
**4 Key Decisions:**
|
||||
|
||||
@@ -927,7 +927,7 @@ teach-me-testing/
|
||||
`{external-project-root}/_bmad-output/bmb-creations/workflows/teach-me-testing/`
|
||||
|
||||
**Target (Production):**
|
||||
`{project-root}/src/workflows/testarch/teach-me-testing/`
|
||||
`{project-root}/src/workflows/testarch/bmad-teach-me-testing/`
|
||||
|
||||
**Command:**
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
name: teach-me-testing
|
||||
name: bmad-teach-me-testing
|
||||
description: 'Teach testing progressively through structured sessions. Use when user says "lets learn testing" or "I want to study test practices"'
|
||||
web_bundle: true
|
||||
---
|
||||
6
_bmad/tea/workflows/testarch/bmad-testarch-atdd/SKILL.md
Normal file
6
_bmad/tea/workflows/testarch/bmad-testarch-atdd/SKILL.md
Normal file
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-testarch-atdd
|
||||
description: 'Generate failing acceptance tests using TDD cycle. Use when the user says "lets write acceptance tests" or "I want to do ATDD"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
# Acceptance Test-Driven Development (ATDD)
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/atdd`
|
||||
**Workflow ID**: `_bmad/tea/testarch/bmad-testarch-atdd`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
@@ -35,11 +35,11 @@ From `workflow.yaml`, resolve:
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/atdd/steps-c/step-01-preflight-and-context.md`
|
||||
`./steps-c/step-01-preflight-and-context.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/atdd/steps-c/step-01b-resume.md`
|
||||
`./steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -71,9 +71,12 @@ const subagentContext = {
|
||||
config: {
|
||||
test_framework: config.test_framework,
|
||||
use_playwright_utils: config.tea_use_playwright_utils,
|
||||
use_pactjs_utils: config.tea_use_pactjs_utils,
|
||||
pact_mcp: config.tea_pact_mcp, // "mcp" | "none"
|
||||
browser_automation: config.tea_browser_automation,
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
provider_endpoint_map: /* from Step 1/3 context, if use_pactjs_utils enabled */,
|
||||
},
|
||||
timestamp: timestamp
|
||||
};
|
||||
@@ -16,7 +16,8 @@ This is an **isolated subagent** running in parallel with E2E failing test gener
|
||||
- Story acceptance criteria from Step 1
|
||||
- Test strategy and scenarios from Step 3
|
||||
- Knowledge fragments loaded: api-request, data-factories, api-testing-patterns
|
||||
- Config: test framework, Playwright Utils enabled/disabled
|
||||
- Config: test framework, Playwright Utils enabled/disabled, Pact.js Utils enabled/disabled (`use_pactjs_utils`), Pact MCP mode (`pact_mcp`)
|
||||
- Provider Endpoint Map (if `use_pactjs_utils` enabled and provider source accessible)
|
||||
|
||||
**Your task:** Generate API tests that will FAIL because the feature is not implemented yet (TDD RED PHASE).
|
||||
|
||||
@@ -113,6 +114,72 @@ test.describe('[Story Name] API Tests (ATDD)', () => {
|
||||
- ✅ Use data factories for test data (from data-factories fragment)
|
||||
- ✅ Include priority tags [P0], [P1], [P2], [P3]
|
||||
|
||||
### 1.5 Provider Source Scrutiny for CDC in TDD Red Phase (If `use_pactjs_utils` Enabled)
|
||||
|
||||
When generating Pact consumer contract tests in the ATDD red phase, provider scrutiny applies with TDD-specific rules. Apply the **Seven-Point Scrutiny Checklist** from `contract-testing.md` (Response shape, Status codes, Field names, Enum values, Required fields, Data types, Nested structures) for both existing and new endpoints.
|
||||
|
||||
**If provider endpoint already exists** (extending an existing API):
|
||||
|
||||
- READ the provider route handler, types, and validation schemas
|
||||
- Verify all seven scrutiny points against the provider source: Response shape, Status codes, Field names, Enum values, Required fields, Data types, Nested structures
|
||||
- Add `// Provider endpoint:` comment and scrutiny evidence block documenting findings for each point
|
||||
- Wrap the entire test function in `test.skip()` (so the whole test including `executeTest` is skipped), not just the callback
|
||||
|
||||
**If provider endpoint is new** (TDD — endpoint not implemented yet):
|
||||
|
||||
- Use acceptance criteria as the source of truth for expected behavior
|
||||
- Acceptance criteria should specify all seven scrutiny points where possible (status codes, field names, types, etc.) — note any gaps as assumptions in the evidence block
|
||||
- Add `// Provider endpoint: TODO — new endpoint, not yet implemented`
|
||||
- Document expected behavior from acceptance criteria in scrutiny evidence block
|
||||
- Wrap the entire test function in `test.skip()` and use realistic expectations from the story
|
||||
|
||||
**Graceful degradation when provider source is inaccessible:**
|
||||
|
||||
1. **OpenAPI/Swagger spec available**: Use the spec as the source of truth for response shapes, status codes, and field names
|
||||
2. **Pact Broker available** (when `pact_mcp` is `"mcp"`): Use SmartBear MCP tools to fetch existing provider states and verified interactions as reference
|
||||
3. **Neither available**: For new endpoints, use acceptance criteria; for existing endpoints, use consumer-side types. Mark with `// Provider endpoint: TODO — provider source not accessible, verify manually` and set `provider_scrutiny: "pending"` in output JSON
|
||||
4. **Never silently guess**: Document all assumptions in the scrutiny evidence block
|
||||
|
||||
**Provider endpoint comments are MANDATORY** even in red-phase tests — they document the intent.
|
||||
|
||||
**Example: Red-phase Pact test with provider scrutiny:**
|
||||
|
||||
```typescript
|
||||
// Provider endpoint: TODO — new endpoint, not yet implemented
|
||||
/*
|
||||
* Provider Scrutiny Evidence:
|
||||
* - Handler: NEW — not yet implemented (TDD red phase)
|
||||
* - Expected from acceptance criteria:
|
||||
* - Endpoint: POST /api/v2/users/register
|
||||
* - Status: 201 for success, 400 for duplicate email, 422 for validation error
|
||||
* - Response: { id: number, email: string, createdAt: string }
|
||||
*/
|
||||
test.skip('[P0] should generate consumer contract for user registration', async () => {
|
||||
await provider
|
||||
.given('no users exist')
|
||||
.uponReceiving('a request to register a new user')
|
||||
.withRequest({
|
||||
method: 'POST',
|
||||
path: '/api/v2/users/register',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: { email: 'newuser@example.com', password: 'SecurePass123!' },
|
||||
})
|
||||
.willRespondWith({
|
||||
status: 201,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: like({
|
||||
id: integer(1),
|
||||
email: string('newuser@example.com'),
|
||||
createdAt: string('2025-01-15T10:00:00Z'),
|
||||
}),
|
||||
})
|
||||
.executeTest(async (mockServer) => {
|
||||
const result = await registerUser({ email: 'newuser@example.com', password: 'SecurePass123!' }, { baseUrl: mockServer.url });
|
||||
expect(result.id).toEqual(expect.any(Number));
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Why test.skip():**
|
||||
|
||||
- Tests are written correctly for EXPECTED behavior
|
||||
@@ -163,6 +230,7 @@ Write JSON to temp file: `/tmp/tea-atdd-api-tests-{{timestamp}}.json`
|
||||
"knowledge_fragments_used": ["api-request", "data-factories", "api-testing-patterns"],
|
||||
"test_count": 3,
|
||||
"tdd_phase": "RED",
|
||||
"provider_scrutiny": "completed",
|
||||
"summary": "Generated 3 FAILING API tests for user registration story"
|
||||
}
|
||||
```
|
||||
@@ -205,6 +273,8 @@ Subagent completes when:
|
||||
- JSON output valid and complete
|
||||
- No E2E/component/unit tests included (out of scope)
|
||||
- Tests follow knowledge fragment patterns
|
||||
- Every Pact interaction has `// Provider endpoint:` comment (if CDC enabled)
|
||||
- Provider scrutiny completed or TODO markers added for new endpoints (if CDC enabled)
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
@@ -213,3 +283,4 @@ Subagent completes when:
|
||||
- Placeholder assertions (expect(true).toBe(true))
|
||||
- Did not follow knowledge fragment patterns
|
||||
- Invalid or missing JSON output
|
||||
- Pact interactions missing provider endpoint comments (if CDC enabled)
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-atdd
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-atdd
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
name: testarch-atdd
|
||||
name: bmad-testarch-atdd
|
||||
description: Generate failing acceptance tests using TDD cycle. Use when user says 'lets write acceptance tests' or 'I want to do ATDD'
|
||||
web_bundle: true
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
# Test Architect workflow: atdd
|
||||
name: testarch-atdd
|
||||
# Test Architect workflow: bmad-testarch-atdd
|
||||
name: bmad-testarch-atdd
|
||||
# prettier-ignore
|
||||
description: 'Generate failing acceptance tests using TDD cycle. Use when the user says "lets write acceptance tests" or "I want to do ATDD"'
|
||||
|
||||
@@ -13,10 +13,10 @@ document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/atdd"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
template: "{installed_path}/atdd-checklist-template.md"
|
||||
installed_path: "."
|
||||
instructions: "./instructions.md"
|
||||
validation: "./checklist.md"
|
||||
template: "./atdd-checklist-template.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-testarch-automate
|
||||
description: 'Expand test automation coverage for codebase. Use when user says "lets expand test coverage" or "I want to automate tests"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -191,6 +191,33 @@ Before starting this workflow, verify:
|
||||
- [ ] Error cases tested (400, 401, 403, 404, 500)
|
||||
- [ ] JWT token format validated (if auth tests)
|
||||
|
||||
### Consumer Contract Tests / CDC (If `use_pactjs_utils` Enabled)
|
||||
|
||||
**Provider Endpoint Comments:**
|
||||
|
||||
- [ ] Every Pact interaction has `// Provider endpoint:` comment
|
||||
- [ ] Comment includes exact file path to provider route handler, OR uses the TODO form when provider is inaccessible
|
||||
- [ ] Comment follows format: `// Provider endpoint: <path> -> <METHOD> <route>` or `// Provider endpoint: TODO — provider source not accessible, verify manually`
|
||||
|
||||
**Provider Source Scrutiny:**
|
||||
|
||||
- [ ] Provider route handlers and/or OpenAPI spec read before generating each interaction
|
||||
- [ ] Status codes verified against provider source (e.g., 201 not assumed 200)
|
||||
- [ ] Field names cross-referenced with provider type/DTO definitions
|
||||
- [ ] Data types verified (string ID vs number ID, date formats)
|
||||
- [ ] Enum/union values extracted from provider validation schemas
|
||||
- [ ] Required request fields and headers checked against provider validation
|
||||
- [ ] Nested response structures match provider's actual response construction
|
||||
- [ ] Scrutiny evidence documented as block comment in each test file
|
||||
|
||||
**CDC Quality Gates:**
|
||||
|
||||
- [ ] Postel's Law enforced: exact values in `withRequest`, matchers in `willRespondWith`
|
||||
- [ ] Response matchers (`like`, `eachLike`, `string`, `integer`) used only in `willRespondWith`
|
||||
- [ ] Provider state names are consistent with provider's state handler naming
|
||||
- [ ] DI pattern used for consumer function imports (actual consumer code, not raw `fetch()`)
|
||||
- [ ] One logical endpoint per Pact interaction (no multi-endpoint interactions)
|
||||
|
||||
### Component Tests (If Applicable)
|
||||
|
||||
- [ ] Component test files created in `tests/component/`
|
||||
@@ -467,6 +494,8 @@ All of the following must be true before marking this workflow as complete:
|
||||
- [ ] **Output file formatted correctly**
|
||||
- [ ] **Knowledge base references applied** and documented (including healing fragments if used)
|
||||
- [ ] **No test quality issues** (flaky patterns, race conditions, hardcoded data, page objects)
|
||||
- [ ] **Provider scrutiny completed or gracefully degraded** for all CDC interactions — each interaction either has scrutiny evidence or a TODO marker (if `use_pactjs_utils` enabled)
|
||||
- [ ] **Provider endpoint comments present** on every Pact interaction (if `use_pactjs_utils` enabled)
|
||||
|
||||
---
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
# Test Automation Expansion
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/automate`
|
||||
**Workflow ID**: `_bmad/tea/testarch/bmad-testarch-automate`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
@@ -40,11 +40,11 @@ From `workflow.yaml`, resolve:
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/automate/steps-c/step-01-preflight-and-context.md`
|
||||
`./steps-c/step-01-preflight-and-context.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/automate/steps-c/step-01b-resume.md`
|
||||
`./steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -77,6 +77,30 @@ Use CLI to explore the application and identify testable pages/flows:
|
||||
|
||||
---
|
||||
|
||||
**If `use_pactjs_utils` is enabled — Provider Endpoint Mapping (all stacks):**
|
||||
|
||||
When consumer-driven contract tests will be generated, build a Provider Endpoint Map during target identification. This applies to all `{detected_stack}` values — frontend, backend, and fullstack consumers all need provider scrutiny.
|
||||
|
||||
1. **Locate provider source and/or OpenAPI spec**: Scan workspace for provider project (from config, monorepo structure, or adjacent repositories). Also check for OpenAPI/Swagger spec files (`openapi.yaml`, `openapi.json`, `swagger.json`) — these document the provider's contract explicitly and can supplement or replace handler code analysis.
|
||||
2. **Map each consumer endpoint** to its provider counterpart:
|
||||
- Provider file path (route handler)
|
||||
- Route pattern (METHOD + path)
|
||||
- Validation schema location (Joi, Zod, class-validator) or OpenAPI request schema
|
||||
- Response type/DTO definition location or OpenAPI response schema
|
||||
- OpenAPI spec path (if available, e.g., `server/openapi.yaml`)
|
||||
3. **Output as "Provider Endpoint Map" table** in the coverage plan:
|
||||
|
||||
```markdown
|
||||
| Consumer Endpoint | Provider File | Route | Validation Schema | Response Type | OpenAPI Spec |
|
||||
| --------------------- | --------------------------------- | ------------------------- | ----------------------------------- | --------------- | ------------------------------------------------- |
|
||||
| GET /api/v2/users/:id | server/src/routes/userHandlers.ts | GET /api/v2/users/:userId | server/src/validation/user.ts | UserResponseDto | server/openapi.yaml#/paths/~1api~1v2~1users~1{id} |
|
||||
| POST /api/v2/users | server/src/routes/userHandlers.ts | POST /api/v2/users | server/src/validation/createUser.ts | UserResponseDto | server/openapi.yaml#/paths/~1api~1v2~1users |
|
||||
```
|
||||
|
||||
4. **If provider source not accessible**: Mark entries with `TODO — provider source not accessible` and note in coverage plan that provider scrutiny will use graceful degradation (see `contract-testing.md` Provider Scrutiny Protocol)
|
||||
|
||||
---
|
||||
|
||||
## 2. Choose Test Levels
|
||||
|
||||
Use `test-levels-framework.md` to select:
|
||||
@@ -74,6 +74,7 @@ const subagentContext = {
|
||||
detected_stack: '{detected_stack}', // "frontend" | "backend" | "fullstack"
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
provider_endpoint_map: /* from Step 2 coverage plan, if use_pactjs_utils enabled */,
|
||||
},
|
||||
timestamp: timestamp
|
||||
};
|
||||
@@ -190,6 +191,7 @@ When `use_pactjs_utils` is enabled, the API test generation subagent (step-03a)
|
||||
- **Provider verification tests**: Using `buildVerifierOptions` for one-call verifier setup
|
||||
- **Message contract tests**: Using `buildMessageVerifierOptions` if async/Kafka patterns detected
|
||||
- **Helper files**: Request filter setup with `createRequestFilter`, shared state constants
|
||||
- **Provider scrutiny**: Subagent reads provider route handlers, types, and validation schemas before generating each interaction (see `contract-testing.md` Provider Scrutiny Protocol)
|
||||
|
||||
When `pact_mcp` is `"mcp"`, the subagent can use SmartBear MCP tools to fetch existing provider states and generate tests informed by broker data.
|
||||
|
||||
@@ -97,8 +97,70 @@ test.describe('[Feature] API Tests', () => {
|
||||
- ✅ Generate request filter helpers in `pact/http/helpers/` using `createRequestFilter({ tokenGenerator: () => string })`
|
||||
- ✅ Generate shared state constants in `pact/http/helpers/states.ts`
|
||||
- ✅ If async/message patterns detected, generate message consumer tests in `pact/message/` using `buildMessageVerifierOptions`
|
||||
- ✅ **Provider endpoint comment MANDATORY** on every Pact interaction: `// Provider endpoint: <path> -> <METHOD> <route>`
|
||||
- ⚠️ **Postel's Law for matchers**: Use `like()`, `eachLike()`, `string()`, `integer()` matchers ONLY in `willRespondWith` (responses). Request bodies in `withRequest` MUST use exact values — never wrap request bodies in `like()`. The consumer controls what it sends, so contracts should be strict about request shape.
|
||||
|
||||
### 1.5 Provider Source Scrutiny (CDC Only)
|
||||
|
||||
**CRITICAL**: Before generating ANY Pact consumer interaction, perform provider source scrutiny per the **Seven-Point Scrutiny Checklist** defined in `contract-testing.md`. Do NOT generate response matchers from consumer-side types alone — this is the #1 cause of contract verification failures.
|
||||
|
||||
The seven points to verify for each interaction:
|
||||
|
||||
1. Response shape
|
||||
2. Status codes
|
||||
3. Field names
|
||||
4. Enum values
|
||||
5. Required fields
|
||||
6. Data types
|
||||
7. Nested structures
|
||||
|
||||
**Source priority**: Provider source code is most authoritative. When an OpenAPI/Swagger spec exists (`openapi.yaml`, `openapi.json`, `swagger.json`), use it as a complementary or alternative source — it documents the provider's contract explicitly and can be faster to parse than tracing through handler code. When both exist, cross-reference them; if they disagree, the source code wins. Document the discrepancy in the scrutiny evidence block (e.g., `OpenAPI shows 200 but handler returns 201; using handler behavior`) and flag it in the output JSON `summary` so it is discoverable by downstream consumers or audits.
|
||||
|
||||
**Scrutiny Sequence** (for each endpoint in the coverage plan):
|
||||
|
||||
1. **READ provider route handler and/or OpenAPI spec**: Find the handler file from `subagentContext.config.provider_endpoint_map` or by scanning the provider codebase. Also check for OpenAPI/Swagger spec files. Extract:
|
||||
- Exact status codes returned (`res.status(201)` / OpenAPI `responses` keys)
|
||||
- Response construction (`res.json({ data: ... })` / OpenAPI `schema`)
|
||||
- Error handling paths (what status codes for what conditions)
|
||||
|
||||
2. **READ provider type/model/DTO definitions**: Find the response type referenced by the handler or OpenAPI `$ref` schemas. Extract:
|
||||
- Exact field names (`transaction_id` not `transactionId`)
|
||||
- Field types (`string` ID vs `number` ID / OpenAPI `type` + `format`)
|
||||
- Optional vs required fields (OpenAPI `required` array)
|
||||
- Nested object structures (OpenAPI `$ref`, `allOf`, `oneOf`)
|
||||
|
||||
3. **READ provider validation schemas**: Find Joi/Zod/class-validator schemas or OpenAPI request body `schema.required`. Extract:
|
||||
- Required request fields and headers
|
||||
- Enum/union type allowed values (`"active" | "inactive"` / OpenAPI `enum`)
|
||||
- Request body constraints
|
||||
|
||||
4. **Cross-reference findings** against consumer expectations:
|
||||
- Does the consumer expect the same field names the provider sends?
|
||||
- Does the consumer expect the same status codes the provider returns?
|
||||
- Does the consumer expect the same nesting the provider produces?
|
||||
|
||||
5. **Document scrutiny evidence** as a block comment in the generated test:
|
||||
|
||||
```typescript
|
||||
/*
|
||||
* Provider Scrutiny Evidence:
|
||||
* - Handler: server/src/routes/userHandlers.ts:45
|
||||
* - OpenAPI: server/openapi.yaml paths./api/v2/users/{userId}.get (if available)
|
||||
* - Response type: UserResponseDto (server/src/types/user.ts:12)
|
||||
* - Status: 201 for creation (line 52), 400 for validation error (line 48)
|
||||
* - Fields: { id: number, name: string, email: string, role: "user" | "admin" }
|
||||
* - Required request headers: Authorization (Bearer token)
|
||||
*/
|
||||
```
|
||||
|
||||
6. **Graceful degradation** when provider source is not accessible (follows the canonical four-step protocol from `contract-testing.md`):
|
||||
1. **OpenAPI/Swagger spec available**: Use the spec as the source of truth for response shapes, status codes, and field names
|
||||
2. **Pact Broker available** (when `pact_mcp` is `"mcp"` in `subagentContext.config`): Use SmartBear MCP tools to fetch existing provider states and verified interactions as reference
|
||||
3. **Neither available**: Generate from consumer types but use the TODO form of the mandatory comment: `// Provider endpoint: TODO — provider source not accessible, verify manually`. Set `provider_scrutiny: "pending"` in output JSON
|
||||
4. **Never silently guess**: Document all assumptions in the scrutiny evidence block
|
||||
|
||||
> ⚠️ **Anti-pattern**: Generating response matchers from consumer-side types alone. This produces contracts that reflect what the consumer _wishes_ the provider returns, not what it _actually_ returns. Always read provider source or OpenAPI spec first.
|
||||
|
||||
### 3. Track Fixture Needs
|
||||
|
||||
Identify fixtures needed for API tests:
|
||||
@@ -145,6 +207,8 @@ Write JSON to temp file: `/tmp/tea-automate-api-tests-{{timestamp}}.json`
|
||||
],
|
||||
"fixture_needs": ["authToken", "userDataFactory", "productDataFactory"],
|
||||
"knowledge_fragments_used": ["api-request", "data-factories", "api-testing-patterns"],
|
||||
"provider_scrutiny": "completed",
|
||||
"provider_files_read": ["server/src/routes/authHandlers.ts", "server/src/routes/checkoutHandlers.ts", "server/src/types/auth.ts"],
|
||||
"test_count": 12,
|
||||
"summary": "Generated 12 API test cases covering 3 features"
|
||||
}
|
||||
@@ -185,6 +249,9 @@ Subagent completes when:
|
||||
- All API tests generated following patterns
|
||||
- JSON output valid and complete
|
||||
- No E2E/component/unit tests included (out of scope)
|
||||
- Every Pact interaction has `// Provider endpoint:` comment (if CDC enabled)
|
||||
- Provider source scrutiny completed or gracefully degraded with TODO markers (if CDC enabled)
|
||||
- Scrutiny evidence documented as block comments in test files (if CDC enabled)
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
@@ -192,3 +259,5 @@ Subagent completes when:
|
||||
- Did not follow knowledge fragment patterns
|
||||
- Invalid or missing JSON output
|
||||
- Ran tests (not subagent responsibility)
|
||||
- Pact interactions missing provider endpoint comments (if CDC enabled)
|
||||
- Response matchers generated from consumer-side types without provider scrutiny (if CDC enabled)
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-automate
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-automate
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
name: testarch-automate
|
||||
name: bmad-testarch-automate
|
||||
description: Expand test automation coverage for codebase. Use when user says 'lets expand test coverage' or 'I want to automate tests'
|
||||
web_bundle: true
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
# Test Architect workflow: automate
|
||||
name: testarch-automate
|
||||
# Test Architect workflow: bmad-testarch-automate
|
||||
name: bmad-testarch-automate
|
||||
# prettier-ignore
|
||||
description: 'Expand test automation coverage for codebase. Use when the user says "lets expand test coverage" or "I want to automate tests"'
|
||||
|
||||
@@ -13,9 +13,9 @@ document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/automate"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
installed_path: "."
|
||||
instructions: "./instructions.md"
|
||||
validation: "./checklist.md"
|
||||
template: false
|
||||
|
||||
# Variables and inputs
|
||||
6
_bmad/tea/workflows/testarch/bmad-testarch-ci/SKILL.md
Normal file
6
_bmad/tea/workflows/testarch/bmad-testarch-ci/SKILL.md
Normal file
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-testarch-ci
|
||||
description: 'Scaffold CI/CD quality pipeline with test execution. Use when the user says "lets setup CI pipeline" or "I want to create quality gates"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
# CI/CD Pipeline Setup
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/ci`
|
||||
**Workflow ID**: `_bmad/tea/testarch/bmad-testarch-ci`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
@@ -35,11 +35,11 @@ From `workflow.yaml`, resolve:
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/ci/steps-c/step-01-preflight.md`
|
||||
`./steps-c/step-01-preflight.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/ci/steps-c/step-01b-resume.md`
|
||||
`./steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -106,16 +106,16 @@ Resolution precedence:
|
||||
|
||||
Determine the pipeline output file path based on the detected `ci_platform`:
|
||||
|
||||
| CI Platform | Output Path | Template File |
|
||||
| ---------------- | ------------------------------------------- | --------------------------------------------------- |
|
||||
| `github-actions` | `{project-root}/.github/workflows/test.yml` | `{installed_path}/github-actions-template.yaml` |
|
||||
| `gitlab-ci` | `{project-root}/.gitlab-ci.yml` | `{installed_path}/gitlab-ci-template.yaml` |
|
||||
| `jenkins` | `{project-root}/Jenkinsfile` | `{installed_path}/jenkins-pipeline-template.groovy` |
|
||||
| `azure-devops` | `{project-root}/azure-pipelines.yml` | `{installed_path}/azure-pipelines-template.yaml` |
|
||||
| `harness` | `{project-root}/.harness/pipeline.yaml` | `{installed_path}/harness-pipeline-template.yaml` |
|
||||
| `circle-ci` | `{project-root}/.circleci/config.yml` | _(no template; generate from first principles)_ |
|
||||
| CI Platform | Output Path | Template File |
|
||||
| ---------------- | ------------------------------------------- | ----------------------------------------------- |
|
||||
| `github-actions` | `{project-root}/.github/workflows/test.yml` | `./github-actions-template.yaml` |
|
||||
| `gitlab-ci` | `{project-root}/.gitlab-ci.yml` | `./gitlab-ci-template.yaml` |
|
||||
| `jenkins` | `{project-root}/Jenkinsfile` | `./jenkins-pipeline-template.groovy` |
|
||||
| `azure-devops` | `{project-root}/azure-pipelines.yml` | `./azure-pipelines-template.yaml` |
|
||||
| `harness` | `{project-root}/.harness/pipeline.yaml` | `./harness-pipeline-template.yaml` |
|
||||
| `circle-ci` | `{project-root}/.circleci/config.yml` | _(no template; generate from first principles)_ |
|
||||
|
||||
Use templates from `{installed_path}` when available. Adapt the template to the project's `test_stack_type` and `test_framework`.
|
||||
Use templates from `./` when available. Adapt the template to the project's `test_stack_type` and `test_framework`.
|
||||
|
||||
---
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-ci
|
||||
workflowPath: {project-root}/src/workflows/testarch/ci
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-ci
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-ci
|
||||
workflowPath: {project-root}/src/workflows/testarch/ci
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-ci
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
name: testarch-ci
|
||||
name: bmad-testarch-ci
|
||||
description: Scaffold CI/CD quality pipeline with test execution. Use when user says 'lets setup CI pipeline' or 'I want to create quality gates'
|
||||
web_bundle: true
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
# Test Architect workflow: ci
|
||||
name: testarch-ci
|
||||
# Test Architect workflow: bmad-testarch-ci
|
||||
name: bmad-testarch-ci
|
||||
# prettier-ignore
|
||||
description: 'Scaffold CI/CD quality pipeline with test execution. Use when the user says "lets setup CI pipeline" or "I want to create quality gates"'
|
||||
|
||||
@@ -13,9 +13,9 @@ document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/ci"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
installed_path: "."
|
||||
instructions: "./instructions.md"
|
||||
validation: "./checklist.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-testarch-framework
|
||||
description: 'Initialize test framework with Playwright or Cypress. Use when the user says "lets setup test framework" or "I want to initialize testing framework"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
# Test Framework Setup
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/framework`
|
||||
**Workflow ID**: `_bmad/tea/testarch/bmad-testarch-framework`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
@@ -35,11 +35,11 @@ From `workflow.yaml`, resolve:
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/framework/steps-c/step-01-preflight.md`
|
||||
`./steps-c/step-01-preflight.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/framework/steps-c/step-01b-resume.md`
|
||||
`./steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user