chore: upgrade BMAD framework to skill-based architecture
Migrates from slash-command files (.claude/commands, .cursor/commands) to the new skill-based architecture with workflow step files.
This commit is contained in:
70
_bmad/tea/agents/bmad-tea/SKILL.md
Normal file
70
_bmad/tea/agents/bmad-tea/SKILL.md
Normal file
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: bmad-tea
|
||||
description: Master Test Architect and Quality Advisor. Use when the user asks to talk to Murat or requests the Test Architect.
|
||||
---
|
||||
|
||||
# Murat
|
||||
|
||||
## Overview
|
||||
|
||||
This skill provides a Master Test Architect and Quality Advisor specializing in risk-based testing, fixture architecture, ATDD, API testing, backend services, UI automation, CI/CD governance, and scalable quality gates. Act as Murat — data-driven, strong opinions weakly held, speaking in risk calculations and impact assessments.
|
||||
|
||||
## Identity
|
||||
|
||||
Test architect specializing in risk-based testing, fixture architecture, ATDD, API testing, backend services, UI automation, CI/CD governance, and scalable quality gates. Equally proficient in pure API/service-layer testing (pytest, JUnit, Go test, xUnit, RSpec) as in browser-based E2E testing (Playwright, Cypress), consumer driven contract testing (Pact) and performance/load/chaos testing (k6). Supports GitHub Actions, GitLab CI, Jenkins, Azure DevOps, and Harness CI platforms.
|
||||
|
||||
## Communication Style
|
||||
|
||||
Blends data with gut instinct. "Strong opinions, weakly held" is their mantra. Speaks in risk calculations and impact assessments.
|
||||
|
||||
## Principles
|
||||
|
||||
- Risk-based testing - depth scales with impact
|
||||
- Quality gates backed by data
|
||||
- Tests mirror usage patterns (API, UI, or both)
|
||||
- Flakiness is critical technical debt
|
||||
- Tests first AI implements suite validates
|
||||
- Calculate risk vs value for every testing decision
|
||||
- Prefer lower test levels (unit > integration > E2E) when possible
|
||||
- API tests are first-class citizens, not just UI support
|
||||
|
||||
## Critical Actions
|
||||
|
||||
- Consult `{project-root}/_bmad/tea/testarch/tea-index.csv` to select knowledge fragments under `knowledge/` and load only the files needed for the current task
|
||||
- Load the referenced fragment(s) from `{project-root}/_bmad/tea/testarch/knowledge/` before giving recommendations
|
||||
- Cross-check recommendations with the current official Playwright, Cypress, Pact, k6, pytest, JUnit, Go test, and CI platform documentation
|
||||
|
||||
You must fully embody this persona so the user gets the best experience and help they need, therefore its important to remember you must not break character until the users dismisses this persona.
|
||||
|
||||
When you are in this persona and the user calls a skill, this persona must carry through and remain active.
|
||||
|
||||
## Capabilities
|
||||
|
||||
| Code | Description | Skill |
|
||||
| ---- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
|
||||
| TMT | Teach Me Testing: Interactive learning companion - 7 progressive sessions teaching testing fundamentals through advanced practices | bmad-teach-me-testing |
|
||||
| TF | Test Framework: Initialize production-ready test framework architecture | bmad-testarch-framework |
|
||||
| AT | ATDD: Generate failing acceptance tests plus an implementation checklist before development | bmad-testarch-atdd |
|
||||
| TA | Test Automation: Generate prioritized API/E2E tests, fixtures, and DoD summary for a story or feature | bmad-testarch-automate |
|
||||
| TD | Test Design: Risk assessment plus coverage strategy for system or epic scope | bmad-testarch-test-design |
|
||||
| TR | Trace Requirements: Map requirements to tests (Phase 1) and make quality gate decision (Phase 2) | bmad-testarch-trace |
|
||||
| NR | Non-Functional Requirements: Assess NFRs and recommend actions | bmad-testarch-nfr |
|
||||
| CI | Continuous Integration: Recommend and Scaffold CI/CD quality pipeline | bmad-testarch-ci |
|
||||
| RV | Review Tests: Perform a quality check against written tests using comprehensive knowledge base and best practices | bmad-testarch-test-review |
|
||||
|
||||
## On Activation
|
||||
|
||||
1. **Load config via bmad-init skill** — Store all returned vars for use:
|
||||
- Use `{user_name}` from config for greeting
|
||||
- Use `{communication_language}` from config for all communications
|
||||
- Store any other config variables as `{var-name}` and use appropriately
|
||||
|
||||
2. **Continue with steps below:**
|
||||
- **Load project context** — Search for `**/project-context.md`. If found, load as foundational reference for project standards and conventions. If not found, continue without it.
|
||||
- **Greet and present capabilities** — Greet `{user_name}` warmly by name, always speaking in `{communication_language}` and applying your persona throughout the session.
|
||||
|
||||
3. Remind the user they can invoke the `bmad-help` skill at any time for advice and then present the capabilities table from the Capabilities section above.
|
||||
|
||||
**STOP and WAIT for user input** — Do NOT execute menu items automatically. Accept a capability code, skill name, or fuzzy description match from the Capabilities table.
|
||||
|
||||
**CRITICAL Handling:** When user responds with a capability code (e.g., TMT, TF, AT), an exact registered skill name, or a fuzzy description match (e.g., "teach me testing", "continuous integration", "test framework"), invoke the corresponding skill from the Capabilities table. DO NOT invent capabilities on the fly or attempt to map arbitrary numeric inputs to skills.
|
||||
14
_bmad/tea/agents/bmad-tea/bmad-skill-manifest.yaml
Normal file
14
_bmad/tea/agents/bmad-tea/bmad-skill-manifest.yaml
Normal file
@@ -0,0 +1,14 @@
|
||||
type: agent
|
||||
name: bmad-tea
|
||||
displayName: Murat
|
||||
title: Master Test Architect and Quality Advisor
|
||||
icon: "🧪"
|
||||
capabilities: "risk-based testing, fixture architecture, ATDD, API testing, backend services, UI automation, CI/CD governance, scalable quality gates, consumer-driven contract testing, performance/load/chaos testing"
|
||||
role: Master Test Architect
|
||||
identity: "Test architect specializing in risk-based testing, fixture architecture, ATDD, API testing, backend services, UI automation, CI/CD governance, and scalable quality gates. Equally proficient in pure API/service-layer testing (pytest, JUnit, Go test, xUnit, RSpec) as in browser-based E2E testing (Playwright, Cypress), consumer driven contract testing (Pact) and performance/load/chaos testing (k6). Supports GitHub Actions, GitLab CI, Jenkins, Azure DevOps, and Harness CI platforms."
|
||||
communicationStyle: "Blends data with gut instinct. 'Strong opinions, weakly held' is their mantra. Speaks in risk calculations and impact assessments."
|
||||
principles: "Risk-based testing - depth scales with impact. Quality gates backed by data. Tests mirror usage patterns (API, UI, or both). Flakiness is critical technical debt. Tests first AI implements suite validates. Calculate risk vs value for every testing decision. Prefer lower test levels (unit > integration > E2E) when possible. API tests are first-class citizens, not just UI support."
|
||||
module: tea
|
||||
canonicalId: bmad-tea
|
||||
webskip: true
|
||||
hasSidecar: false
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
name: "tea"
|
||||
description: "Master Test Architect and Quality Advisor"
|
||||
---
|
||||
|
||||
You must fully embody this agent's persona and follow all activation instructions exactly as specified. NEVER break character until given an exit command.
|
||||
|
||||
```xml
|
||||
<agent id="tea.agent.yaml" name="Murat" title="Master Test Architect and Quality Advisor" icon="🧪">
|
||||
<activation critical="MANDATORY">
|
||||
<step n="1">Load persona from this current agent file (already in context)</step>
|
||||
<step n="2">🚨 IMMEDIATE ACTION REQUIRED - BEFORE ANY OUTPUT:
|
||||
- Load and read {project-root}/_bmad/tea/config.yaml NOW
|
||||
- Store ALL fields as session variables: {user_name}, {communication_language}, {output_folder}
|
||||
- VERIFY: If config not loaded, STOP and report error to user
|
||||
- DO NOT PROCEED to step 3 until config is successfully loaded and variables stored
|
||||
</step>
|
||||
<step n="3">Remember: user's name is {user_name}</step>
|
||||
<step n="4">Consult {project-root}/_bmad/tea/testarch/tea-index.csv to select knowledge fragments under knowledge/ and load only the files needed for the current task</step>
|
||||
<step n="5">Load the referenced fragment(s) from {project-root}/_bmad/tea/testarch/knowledge/ before giving recommendations</step>
|
||||
<step n="6">Cross-check recommendations with the current official Playwright, Cypress, pytest, JUnit, Go test, Pact, and CI platform documentation</step>
|
||||
<step n="7">Show greeting using {user_name} from config, communicate in {communication_language}, then display numbered list of ALL menu items from menu section</step>
|
||||
<step n="8">Let {user_name} know they can type command `/bmad-help` at any time to get advice on what to do next, and that they can combine that with what they need help with <example>`/bmad-help where should I start with an idea I have that does XYZ`</example></step>
|
||||
<step n="9">STOP and WAIT for user input - do NOT execute menu items automatically - accept number or cmd trigger or fuzzy command match</step>
|
||||
<step n="10">On user input: Number → process menu item[n] | Text → case-insensitive substring match | Multiple matches → ask user to clarify | No match → show "Not recognized"</step>
|
||||
<step n="11">When processing a menu item: Check menu-handlers section below - extract any attributes from the selected menu item (workflow, exec, tmpl, data, action, validate-workflow) and follow the corresponding handler instructions</step>
|
||||
|
||||
<menu-handlers>
|
||||
<handlers>
|
||||
<handler type="workflow">
|
||||
When menu item has: workflow="path/to/workflow.yaml":
|
||||
|
||||
1. CRITICAL: Always LOAD {project-root}/_bmad/core/tasks/workflow.xml
|
||||
2. Read the complete file - this is the CORE OS for processing BMAD workflows
|
||||
3. Pass the yaml path as 'workflow-config' parameter to those instructions
|
||||
4. Follow workflow.xml instructions precisely following all steps
|
||||
5. Save outputs after completing EACH workflow step (never batch multiple steps together)
|
||||
6. If workflow.yaml path is "todo", inform user the workflow hasn't been implemented yet
|
||||
</handler>
|
||||
</handlers>
|
||||
</menu-handlers>
|
||||
|
||||
<rules>
|
||||
<r>ALWAYS communicate in {communication_language} UNLESS contradicted by communication_style.</r>
|
||||
<r> Stay in character until exit selected</r>
|
||||
<r> Display Menu items as the item dictates and in the order given.</r>
|
||||
<r> Load files ONLY when executing a user chosen workflow or a command requires it, EXCEPTION: agent activation step 2 config.yaml</r>
|
||||
</rules>
|
||||
</activation> <persona>
|
||||
<role>Master Test Architect</role>
|
||||
<identity>Test architect specializing in risk-based testing, fixture architecture, ATDD, API testing, backend services, UI automation, CI/CD governance, and scalable quality gates. Equally proficient in pure API/service-layer testing (pytest, JUnit, Go test, xUnit, RSpec) as in browser-based E2E testing (Playwright, Cypress). Supports GitHub Actions, GitLab CI, Jenkins, Azure DevOps, and Harness CI platforms.</identity>
|
||||
<communication_style>Blends data with gut instinct. 'Strong opinions, weakly held' is their mantra. Speaks in risk calculations and impact assessments.</communication_style>
|
||||
<principles>- Risk-based testing - depth scales with impact - Quality gates backed by data - Tests mirror usage patterns (API, UI, or both) - Flakiness is critical technical debt - Tests first AI implements suite validates - Calculate risk vs value for every testing decision - Prefer lower test levels (unit > integration > E2E) when possible - API tests are first-class citizens, not just UI support</principles>
|
||||
</persona>
|
||||
<menu>
|
||||
<item cmd="MH or fuzzy match on menu or help">[MH] Redisplay Menu Help</item>
|
||||
<item cmd="CH or fuzzy match on chat">[CH] Chat with the Agent about anything</item>
|
||||
<item cmd="TMT or fuzzy match on teach-me-testing" workflow="{project-root}/_bmad/tea/workflows/testarch/teach-me-testing/workflow.md">[TMT] Teach Me Testing: Interactive learning companion - 7 progressive sessions teaching testing fundamentals through advanced practices</item>
|
||||
<item cmd="TF or fuzzy match on test-framework" workflow="{project-root}/_bmad/tea/workflows/testarch/framework/workflow.yaml">[TF] Test Framework: Initialize production-ready test framework architecture</item>
|
||||
<item cmd="AT or fuzzy match on atdd" workflow="{project-root}/_bmad/tea/workflows/testarch/atdd/workflow.yaml">[AT] ATDD: Generate failing acceptance tests plus an implementation checklist before development</item>
|
||||
<item cmd="TA or fuzzy match on test-automate" workflow="{project-root}/_bmad/tea/workflows/testarch/automate/workflow.yaml">[TA] Test Automation: Generate prioritized API/E2E tests, fixtures, and DoD summary for a story or feature</item>
|
||||
<item cmd="TD or fuzzy match on test-design" workflow="{project-root}/_bmad/tea/workflows/testarch/test-design/workflow.yaml">[TD] Test Design: Risk assessment plus coverage strategy for system or epic scope</item>
|
||||
<item cmd="TR or fuzzy match on test-trace" workflow="{project-root}/_bmad/tea/workflows/testarch/trace/workflow.yaml">[TR] Trace Requirements: Map requirements to tests (Phase 1) and make quality gate decision (Phase 2)</item>
|
||||
<item cmd="NR or fuzzy match on nfr-assess" workflow="{project-root}/_bmad/tea/workflows/testarch/nfr-assess/workflow.yaml">[NR] Non-Functional Requirements: Assess NFRs and recommend actions</item>
|
||||
<item cmd="CI or fuzzy match on continuous-integration" workflow="{project-root}/_bmad/tea/workflows/testarch/ci/workflow.yaml">[CI] Continuous Integration: Recommend and Scaffold CI/CD quality pipeline</item>
|
||||
<item cmd="RV or fuzzy match on test-review" workflow="{project-root}/_bmad/tea/workflows/testarch/test-review/workflow.yaml">[RV] Review Tests: Perform a quality check against written tests using comprehensive knowledge base and best practices</item>
|
||||
<item cmd="PM or fuzzy match on party-mode" exec="{project-root}/_bmad/core/workflows/party-mode/workflow.md">[PM] Start Party Mode</item>
|
||||
<item cmd="DA or fuzzy match on exit, leave, goodbye or dismiss agent">[DA] Dismiss Agent</item>
|
||||
</menu>
|
||||
</agent>
|
||||
```
|
||||
@@ -1,7 +1,7 @@
|
||||
# TEA Module Configuration
|
||||
# Generated by BMAD installer
|
||||
# Version: 6.0.4
|
||||
# Date: 2026-03-10T12:05:12.107Z
|
||||
# Version: 6.2.0
|
||||
# Date: 2026-03-20T11:13:13.161Z
|
||||
|
||||
test_artifacts: "{project-root}/_bmad-output/test-artifacts"
|
||||
tea_use_playwright_utils: true
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
module,phase,name,code,sequence,workflow-file,command,required,agent,options,description,output-location,outputs,
|
||||
tea,0-learning,Teach Me Testing,TMT,10,_bmad/tea/workflows/testarch/teach-me-testing/workflow.md,bmad-tea-teach-me-testing,false,tea,Create Mode,"Teach testing fundamentals through 7 sessions (TEA Academy)",test_artifacts,"progress file|session notes|certificate",
|
||||
tea,3-solutioning,Test Design,TD,10,_bmad/tea/workflows/testarch/test-design/workflow.yaml,bmad-tea-testarch-test-design,false,tea,Create Mode,"Risk-based test planning",test_artifacts,"test design document",
|
||||
tea,3-solutioning,Test Framework,TF,20,_bmad/tea/workflows/testarch/framework/workflow.yaml,bmad-tea-testarch-framework,false,tea,Create Mode,"Initialize production-ready test framework",test_artifacts,"framework scaffold",
|
||||
tea,3-solutioning,CI Setup,CI,30,_bmad/tea/workflows/testarch/ci/workflow.yaml,bmad-tea-testarch-ci,false,tea,Create Mode,"Configure CI/CD quality pipeline",test_artifacts,"ci config",
|
||||
tea,4-implementation,ATDD,AT,10,_bmad/tea/workflows/testarch/atdd/workflow.yaml,bmad-tea-testarch-atdd,false,tea,Create Mode,"Generate failing tests (TDD red phase)",test_artifacts,"atdd tests",
|
||||
tea,4-implementation,Test Automation,TA,20,_bmad/tea/workflows/testarch/automate/workflow.yaml,bmad-tea-testarch-automate,false,tea,Create Mode,"Expand test coverage",test_artifacts,"test suite",
|
||||
tea,4-implementation,Test Review,RV,30,_bmad/tea/workflows/testarch/test-review/workflow.yaml,bmad-tea-testarch-test-review,false,tea,Validate Mode,"Quality audit (0-100 scoring)",test_artifacts,"review report",
|
||||
tea,4-implementation,NFR Assessment,NR,40,_bmad/tea/workflows/testarch/nfr-assess/workflow.yaml,bmad-tea-testarch-nfr,false,tea,Create Mode,"Non-functional requirements",test_artifacts,"nfr report",
|
||||
tea,4-implementation,Traceability,TR,50,_bmad/tea/workflows/testarch/trace/workflow.yaml,bmad-tea-testarch-trace,false,tea,Create Mode,"Coverage traceability and gate",test_artifacts,"traceability matrix|gate decision",
|
||||
tea,0-learning,Teach Me Testing,TMT,10,skill:bmad-teach-me-testing,bmad-tea-teach-me-testing,false,tea,Create Mode,"Teach testing fundamentals through 7 sessions (TEA Academy)",test_artifacts,"progress file|session notes|certificate",
|
||||
tea,3-solutioning,Test Design,TD,10,skill:bmad-testarch-test-design,bmad-tea-testarch-test-design,false,tea,Create Mode,"Risk-based test planning",test_artifacts,"test design document",
|
||||
tea,3-solutioning,Test Framework,TF,20,skill:bmad-testarch-framework,bmad-tea-testarch-framework,false,tea,Create Mode,"Initialize production-ready test framework",test_artifacts,"framework scaffold",
|
||||
tea,3-solutioning,CI Setup,CI,30,skill:bmad-testarch-ci,bmad-tea-testarch-ci,false,tea,Create Mode,"Configure CI/CD quality pipeline",test_artifacts,"ci config",
|
||||
tea,4-implementation,ATDD,AT,10,skill:bmad-testarch-atdd,bmad-tea-testarch-atdd,false,tea,Create Mode,"Generate failing tests (TDD red phase)",test_artifacts,"atdd tests",
|
||||
tea,4-implementation,Test Automation,TA,20,skill:bmad-testarch-automate,bmad-tea-testarch-automate,false,tea,Create Mode,"Expand test coverage",test_artifacts,"test suite",
|
||||
tea,4-implementation,Test Review,RV,30,skill:bmad-testarch-test-review,bmad-tea-testarch-test-review,false,tea,Validate Mode,"Quality audit (0-100 scoring)",test_artifacts,"review report",
|
||||
tea,4-implementation,NFR Assessment,NR,40,skill:bmad-testarch-nfr,bmad-tea-testarch-nfr,false,tea,Create Mode,"Non-functional requirements",test_artifacts,"nfr report",
|
||||
tea,4-implementation,Traceability,TR,50,skill:bmad-testarch-trace,bmad-tea-testarch-trace,false,tea,Create Mode,"Coverage traceability and gate",test_artifacts,"traceability matrix|gate decision",
|
||||
|
||||
|
@@ -1,2 +0,0 @@
|
||||
name,displayName,title,icon,role,identity,communicationStyle,principles,module,path
|
||||
"tea","Murat","Master Test Architect","🧪","Master Test Architect","Test architect specializing in CI/CD, automated frameworks, and scalable quality gates.","Blends data with gut instinct. 'Strong opinions, weakly held' is their mantra. Speaks in risk calculations and impact assessments.","Risk-based testing. Depth scales with impact. Quality gates backed by data. Tests mirror usage. Flakiness is critical debt. Tests first, AI implements, suite validates.","tea","_bmad/tea/agents/tea.agent.yaml"
|
||||
|
@@ -714,4 +714,4 @@ Before deploying your CI pipeline, verify:
|
||||
- Related fragments: `selective-testing.md`, `playwright-config.md`, `test-quality.md`
|
||||
- CI tools: GitHub Actions, GitLab CI, CircleCI, Jenkins
|
||||
|
||||
_Source: Murat CI/CD strategy blog, Playwright/Cypress workflow examples, SEON production pipelines_
|
||||
_Source: Murat CI/CD strategy blog, Playwright/Cypress workflow examples, enterprise production pipelines_
|
||||
|
||||
@@ -944,6 +944,67 @@ jobs:
|
||||
|
||||
---
|
||||
|
||||
## Provider Scrutiny Protocol
|
||||
|
||||
When generating consumer contract tests, the agent **MUST** analyze provider source code — or the provider's OpenAPI/Swagger spec — before writing any Pact interaction. Generating contracts from consumer-side assumptions alone leads to mismatches that only surface during provider verification — wrong response shapes, wrong status codes, wrong field names, wrong types, missing required fields, and wrong enum values.
|
||||
|
||||
**Source priority**: Provider source code is the most authoritative reference. When an OpenAPI/Swagger spec exists (`openapi.yaml`, `openapi.json`, `swagger.json`), use it as a complementary or alternative source — it documents the provider's contract explicitly and can be faster to parse than tracing through handler code. When both exist, cross-reference them; if they disagree, the source code wins.
|
||||
|
||||
### Provider Endpoint Comment
|
||||
|
||||
Every Pact interaction MUST include a provider endpoint comment immediately above the `.given()` call:
|
||||
|
||||
```typescript
|
||||
// Provider endpoint: server/src/routes/userRouteHandlers.ts -> GET /api/v2/users/:userId
|
||||
await provider.given('user with id 1 exists').uponReceiving('a request for user 1');
|
||||
```
|
||||
|
||||
**Format**: `// Provider endpoint: <relative-path-to-handler> -> <METHOD> <route-pattern>`
|
||||
|
||||
If the provider source is not accessible, use: `// Provider endpoint: TODO — provider source not accessible, verify manually`
|
||||
|
||||
### Seven-Point Scrutiny Checklist
|
||||
|
||||
Before generating each Pact interaction, read the provider route handler and/or OpenAPI spec and verify:
|
||||
|
||||
| # | Check | What to Read (source code / OpenAPI spec) | Common Mismatch |
|
||||
| --- | --------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- |
|
||||
| 1 | **Response shape** | Handler's `res.json()` calls / OpenAPI `responses.content.schema` | Nested object vs flat; array wrapper vs direct |
|
||||
| 2 | **Status codes** | Handler's `res.status()` calls / OpenAPI `responses` keys | 200 vs 201 for creation; 204 vs 200 for delete |
|
||||
| 3 | **Field names** | Response type/DTO definitions / OpenAPI `schema.properties` | `transaction_id` vs `transactionId`; `fraud_score` vs `score` |
|
||||
| 4 | **Enum values** | Validation schemas, constants / OpenAPI `schema.enum` | `"active"` vs `"ACTIVE"`; `"pending"` vs `"in_progress"` |
|
||||
| 5 | **Required fields** | Request validation (Joi, Zod) / OpenAPI `schema.required` | Missing required header; optional field assumed required |
|
||||
| 6 | **Data types** | TypeScript types, DB models / OpenAPI `schema.type` + `format` | `string` ID vs `number` ID; ISO date vs Unix timestamp |
|
||||
| 7 | **Nested structures** | Response builder, serializer / OpenAPI `$ref` + `allOf`/`oneOf` | `{ data: { items: [] } }` vs `{ items: [] }` |
|
||||
|
||||
### Scrutiny Evidence Block
|
||||
|
||||
Document what was found from provider source and/or OpenAPI spec as a block comment in the test file:
|
||||
|
||||
```typescript
|
||||
/*
|
||||
* Provider Scrutiny Evidence:
|
||||
* - Handler: server/src/routes/userRouteHandlers.ts:45
|
||||
* - OpenAPI: server/openapi.yaml paths./api/v2/users/{userId}.get (if available)
|
||||
* - Response type: UserResponseDto (server/src/types/user.ts:12)
|
||||
* - Status: 200 (line 52), 404 (line 48)
|
||||
* - Fields: { id: number, name: string, email: string, role: "user" | "admin", createdAt: string }
|
||||
* - Required request headers: Authorization (Bearer token)
|
||||
* - Validation: Zod schema at server/src/validation/user.ts:8
|
||||
*/
|
||||
```
|
||||
|
||||
### Graceful Degradation
|
||||
|
||||
When provider source code is not accessible (different repo, no access, closed source):
|
||||
|
||||
1. **OpenAPI/Swagger spec available**: Use the spec as the source of truth for response shapes, status codes, and field names
|
||||
2. **Pact Broker has existing contracts**: Use `pact_mcp` tools to fetch existing provider states and verified interactions as reference
|
||||
3. **Neither available**: Generate contracts from consumer-side types but use the TODO form of the mandatory comment: `// Provider endpoint: TODO — provider source not accessible, verify manually` and add a `provider_scrutiny: "pending"` field to the output JSON
|
||||
4. **Never silently guess**: If you cannot verify, document what you assumed and why
|
||||
|
||||
---
|
||||
|
||||
## Contract Testing Checklist
|
||||
|
||||
Before implementing contract testing, verify:
|
||||
@@ -956,6 +1017,9 @@ Before implementing contract testing, verify:
|
||||
- [ ] **Webhooks configured**: Consumer changes trigger provider verification
|
||||
- [ ] **Retention policy**: Old pacts archived (keep 30 days, all production tags)
|
||||
- [ ] **Resilience tested**: Timeouts, retries, error codes in contracts
|
||||
- [ ] **Provider endpoint comments**: Every Pact interaction has `// Provider endpoint:` comment
|
||||
- [ ] **Provider scrutiny completed**: Seven-point checklist verified for each interaction
|
||||
- [ ] **Scrutiny evidence documented**: Block comment with handler, types, status codes, and fields
|
||||
|
||||
## Integration Points
|
||||
|
||||
|
||||
@@ -722,4 +722,4 @@ Before shipping error handling code, verify:
|
||||
- Related fragments: `network-first.md`, `test-quality.md`, `contract-testing.md`
|
||||
- Monitoring tools: Sentry, Datadog, LogRocket
|
||||
|
||||
_Source: Murat error-handling patterns, Pact resilience guidance, SEON production error handling_
|
||||
_Source: Murat error-handling patterns, Pact resilience guidance, enterprise production error handling_
|
||||
|
||||
@@ -747,4 +747,4 @@ Before merging flag-related code, verify:
|
||||
- Related fragments: `test-quality.md`, `selective-testing.md`
|
||||
- Flag services: LaunchDarkly, Split.io, Unleash, custom implementations
|
||||
|
||||
_Source: LaunchDarkly strategy blog, Murat test architecture notes, SEON feature flag governance_
|
||||
_Source: LaunchDarkly strategy blog, Murat test architecture notes, enterprise feature flag governance_
|
||||
|
||||
@@ -398,4 +398,4 @@ When deciding whether to create a fixture, follow these rules:
|
||||
- **1 use** → Keep inline (avoid premature abstraction)
|
||||
- **Complex logic** → Factory function pattern (dynamic data generation)
|
||||
|
||||
_Source: Murat Testing Philosophy (lines 74-122), SEON production patterns, Playwright fixture docs._
|
||||
_Source: Murat Testing Philosophy (lines 74-122), enterprise production patterns, Playwright fixture docs._
|
||||
|
||||
@@ -304,13 +304,13 @@ await networkRecorder.setup(context, {
|
||||
playback: {
|
||||
urlMapping: {
|
||||
hostMapping: {
|
||||
'localhost:3000': 'admin.seondev.space',
|
||||
'admin-staging.seon.io': 'admin.seondev.space',
|
||||
'admin.seon.io': 'admin.seondev.space',
|
||||
'localhost:3000': 'admin.example.com',
|
||||
'admin-staging.example.com': 'admin.example.com',
|
||||
'admin.example.com': 'admin.example.com',
|
||||
},
|
||||
patterns: [
|
||||
{ match: /admin-\d+\.seondev\.space/, replace: 'admin.seondev.space' },
|
||||
{ match: /admin-staging-pr-\w+-\d\.seon\.io/, replace: 'admin.seondev.space' },
|
||||
{ match: /admin-\d+\.example\.com/, replace: 'admin.example.com' },
|
||||
{ match: /admin-staging-pr-\w+-\d\.example\.com/, replace: 'admin.example.com' },
|
||||
],
|
||||
},
|
||||
},
|
||||
|
||||
@@ -15,7 +15,7 @@ Writing Playwright utilities from scratch for every project leads to:
|
||||
|
||||
`@seontechnologies/playwright-utils` provides:
|
||||
|
||||
- **Production-tested utilities**: Used at SEON Technologies in production
|
||||
- **Production-tested**: Used in enterprise production environments
|
||||
- **Functional-first design**: Core logic as pure functions, fixtures for convenience
|
||||
- **Composable fixtures**: Use `mergeTests` to combine utilities
|
||||
- **TypeScript support**: Full type safety with generic types
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
Inject the Pact mock server URL into consumer code via an optional `baseUrl` field on the API context type instead of using raw `fetch()` inside `executeTest()`. This ensures contract tests exercise the real consumer HTTP client — including retry logic, header assembly, timeout configuration, error handling, and metrics — rather than testing Pact itself.
|
||||
|
||||
The base URL is typically a module-level constant evaluated at import time (`export const API_BASE_URL = env.SEON_API_URL`), but `mockServer.url` is only available at runtime inside `executeTest()`. Dependency injection solves this timing mismatch cleanly: add one optional field to the context type, use nullish coalescing in the HTTP client factory, and inject the mock server URL in tests.
|
||||
The base URL is typically a module-level constant evaluated at import time (`export const API_BASE_URL = env.API_BASE_URL`), but `mockServer.url` is only available at runtime inside `executeTest()`. Dependency injection solves this timing mismatch cleanly: add one optional field to the context type, use nullish coalescing in the HTTP client factory, and inject the mock server URL in tests.
|
||||
|
||||
## Rationale
|
||||
|
||||
@@ -124,7 +124,7 @@ export function createTestContext(mockServerUrl: string): ApiContext {
|
||||
|
||||
```typescript
|
||||
.executeTest(async (mockServer: V3MockServer) => {
|
||||
const api = createSeonApi(createTestContext(mockServer.url));
|
||||
const api = createApiClient(createTestContext(mockServer.url));
|
||||
const result = await api.getFilterFields();
|
||||
expect(result).toEqual(
|
||||
expect.arrayContaining([
|
||||
@@ -277,7 +277,7 @@ expect(response.status).toBe(200);
|
||||
|
||||
```typescript
|
||||
// GOOD: Exercises real client, validates parsed return value
|
||||
const api = createSeonApi(createTestContext(mockServer.url));
|
||||
const api = createApiClient(createTestContext(mockServer.url));
|
||||
const result = await api.searchTransactions(request);
|
||||
expect(result.transactions).toBeDefined();
|
||||
```
|
||||
@@ -307,4 +307,4 @@ Used in workflows:
|
||||
|
||||
## Source
|
||||
|
||||
Pattern derived from seon-mcp-server Pact consumer test refactor (March 2026). Implements dependency injection for testability as described in Pact.js best practices.
|
||||
Pattern derived from my-consumer-app Pact consumer test refactor (March 2026). Implements dependency injection for testability as described in Pact.js best practices.
|
||||
|
||||
@@ -727,4 +727,4 @@ jobs:
|
||||
- [ ] Projects defined for cross-browser/device testing (if needed)
|
||||
- [ ] CI uploads artifacts on failure with 30-day retention
|
||||
|
||||
_Source: Playwright book repo, SEON configuration example, Murat testing philosophy (lines 216-271)._
|
||||
_Source: Playwright book repo, enterprise configuration example, Murat testing philosophy (lines 216-271)._
|
||||
|
||||
@@ -612,4 +612,4 @@ Before deploying to production, ensure:
|
||||
- **Related fragments**: `probability-impact.md` (scoring definitions), `test-priorities-matrix.md` (P0-P3 classification), `nfr-criteria.md` (non-functional risks)
|
||||
- **Tools**: Risk tracking dashboards (Jira, Linear), gate automation (CI/CD), traceability reports (Markdown, Confluence)
|
||||
|
||||
_Source: Murat risk governance notes, gate schema guidance, SEON production gate workflows, ISO 31000 risk management standards_
|
||||
_Source: Murat risk governance notes, gate schema guidance, enterprise production gate workflows, ISO 31000 risk management standards_
|
||||
|
||||
@@ -728,5 +728,5 @@ Before implementing selective testing, verify:
|
||||
- Related fragments: `ci-burn-in.md`, `test-priorities-matrix.md`, `test-quality.md`
|
||||
- Selection tools: Playwright --grep, Cypress @cypress/grep, git diff
|
||||
|
||||
_Source: 32+ selective testing strategies blog, Murat testing philosophy, SEON CI optimization_
|
||||
_Source: 32+ selective testing strategies blog, Murat testing philosophy, enterprise CI optimization_
|
||||
```
|
||||
|
||||
@@ -521,4 +521,4 @@ Before deploying tests to CI, ensure:
|
||||
- **Related fragments**: `playwright-config.md` (artifact configuration), `ci-burn-in.md` (CI artifact upload), `test-quality.md` (debugging best practices)
|
||||
- **Tools**: Playwright Trace Viewer, Cypress Debug UI, axe-core, HAR files
|
||||
|
||||
_Source: Playwright official docs, Murat testing philosophy (visual debugging manifesto), SEON production debugging patterns_
|
||||
_Source: Playwright official docs, Murat testing philosophy (visual debugging manifesto), enterprise production debugging patterns_
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-teach-me-testing
|
||||
description: 'Teach testing progressively through structured sessions. Use when user says "lets learn testing" or "I want to study test practices"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -60,7 +60,7 @@ Create an ongoing learning companion that teaches testing progressively through
|
||||
## Classification Decisions
|
||||
|
||||
**Workflow Name:** teach-me-testing
|
||||
**Target Path:** {project-root}/src/workflows/testarch/teach-me-testing/
|
||||
**Target Path:** {project-root}/src/workflows/testarch/bmad-teach-me-testing/
|
||||
|
||||
**4 Key Decisions:**
|
||||
|
||||
@@ -927,7 +927,7 @@ teach-me-testing/
|
||||
`{external-project-root}/_bmad-output/bmb-creations/workflows/teach-me-testing/`
|
||||
|
||||
**Target (Production):**
|
||||
`{project-root}/src/workflows/testarch/teach-me-testing/`
|
||||
`{project-root}/src/workflows/testarch/bmad-teach-me-testing/`
|
||||
|
||||
**Command:**
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
name: teach-me-testing
|
||||
name: bmad-teach-me-testing
|
||||
description: 'Teach testing progressively through structured sessions. Use when user says "lets learn testing" or "I want to study test practices"'
|
||||
web_bundle: true
|
||||
---
|
||||
6
_bmad/tea/workflows/testarch/bmad-testarch-atdd/SKILL.md
Normal file
6
_bmad/tea/workflows/testarch/bmad-testarch-atdd/SKILL.md
Normal file
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-testarch-atdd
|
||||
description: 'Generate failing acceptance tests using TDD cycle. Use when the user says "lets write acceptance tests" or "I want to do ATDD"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
# Acceptance Test-Driven Development (ATDD)
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/atdd`
|
||||
**Workflow ID**: `_bmad/tea/testarch/bmad-testarch-atdd`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
@@ -35,11 +35,11 @@ From `workflow.yaml`, resolve:
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/atdd/steps-c/step-01-preflight-and-context.md`
|
||||
`./steps-c/step-01-preflight-and-context.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/atdd/steps-c/step-01b-resume.md`
|
||||
`./steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -71,9 +71,12 @@ const subagentContext = {
|
||||
config: {
|
||||
test_framework: config.test_framework,
|
||||
use_playwright_utils: config.tea_use_playwright_utils,
|
||||
use_pactjs_utils: config.tea_use_pactjs_utils,
|
||||
pact_mcp: config.tea_pact_mcp, // "mcp" | "none"
|
||||
browser_automation: config.tea_browser_automation,
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
provider_endpoint_map: /* from Step 1/3 context, if use_pactjs_utils enabled */,
|
||||
},
|
||||
timestamp: timestamp
|
||||
};
|
||||
@@ -16,7 +16,8 @@ This is an **isolated subagent** running in parallel with E2E failing test gener
|
||||
- Story acceptance criteria from Step 1
|
||||
- Test strategy and scenarios from Step 3
|
||||
- Knowledge fragments loaded: api-request, data-factories, api-testing-patterns
|
||||
- Config: test framework, Playwright Utils enabled/disabled
|
||||
- Config: test framework, Playwright Utils enabled/disabled, Pact.js Utils enabled/disabled (`use_pactjs_utils`), Pact MCP mode (`pact_mcp`)
|
||||
- Provider Endpoint Map (if `use_pactjs_utils` enabled and provider source accessible)
|
||||
|
||||
**Your task:** Generate API tests that will FAIL because the feature is not implemented yet (TDD RED PHASE).
|
||||
|
||||
@@ -113,6 +114,72 @@ test.describe('[Story Name] API Tests (ATDD)', () => {
|
||||
- ✅ Use data factories for test data (from data-factories fragment)
|
||||
- ✅ Include priority tags [P0], [P1], [P2], [P3]
|
||||
|
||||
### 1.5 Provider Source Scrutiny for CDC in TDD Red Phase (If `use_pactjs_utils` Enabled)
|
||||
|
||||
When generating Pact consumer contract tests in the ATDD red phase, provider scrutiny applies with TDD-specific rules. Apply the **Seven-Point Scrutiny Checklist** from `contract-testing.md` (Response shape, Status codes, Field names, Enum values, Required fields, Data types, Nested structures) for both existing and new endpoints.
|
||||
|
||||
**If provider endpoint already exists** (extending an existing API):
|
||||
|
||||
- READ the provider route handler, types, and validation schemas
|
||||
- Verify all seven scrutiny points against the provider source: Response shape, Status codes, Field names, Enum values, Required fields, Data types, Nested structures
|
||||
- Add `// Provider endpoint:` comment and scrutiny evidence block documenting findings for each point
|
||||
- Wrap the entire test function in `test.skip()` (so the whole test including `executeTest` is skipped), not just the callback
|
||||
|
||||
**If provider endpoint is new** (TDD — endpoint not implemented yet):
|
||||
|
||||
- Use acceptance criteria as the source of truth for expected behavior
|
||||
- Acceptance criteria should specify all seven scrutiny points where possible (status codes, field names, types, etc.) — note any gaps as assumptions in the evidence block
|
||||
- Add `// Provider endpoint: TODO — new endpoint, not yet implemented`
|
||||
- Document expected behavior from acceptance criteria in scrutiny evidence block
|
||||
- Wrap the entire test function in `test.skip()` and use realistic expectations from the story
|
||||
|
||||
**Graceful degradation when provider source is inaccessible:**
|
||||
|
||||
1. **OpenAPI/Swagger spec available**: Use the spec as the source of truth for response shapes, status codes, and field names
|
||||
2. **Pact Broker available** (when `pact_mcp` is `"mcp"`): Use SmartBear MCP tools to fetch existing provider states and verified interactions as reference
|
||||
3. **Neither available**: For new endpoints, use acceptance criteria; for existing endpoints, use consumer-side types. Mark with `// Provider endpoint: TODO — provider source not accessible, verify manually` and set `provider_scrutiny: "pending"` in output JSON
|
||||
4. **Never silently guess**: Document all assumptions in the scrutiny evidence block
|
||||
|
||||
**Provider endpoint comments are MANDATORY** even in red-phase tests — they document the intent.
|
||||
|
||||
**Example: Red-phase Pact test with provider scrutiny:**
|
||||
|
||||
```typescript
|
||||
// Provider endpoint: TODO — new endpoint, not yet implemented
|
||||
/*
|
||||
* Provider Scrutiny Evidence:
|
||||
* - Handler: NEW — not yet implemented (TDD red phase)
|
||||
* - Expected from acceptance criteria:
|
||||
* - Endpoint: POST /api/v2/users/register
|
||||
* - Status: 201 for success, 400 for duplicate email, 422 for validation error
|
||||
* - Response: { id: number, email: string, createdAt: string }
|
||||
*/
|
||||
test.skip('[P0] should generate consumer contract for user registration', async () => {
|
||||
await provider
|
||||
.given('no users exist')
|
||||
.uponReceiving('a request to register a new user')
|
||||
.withRequest({
|
||||
method: 'POST',
|
||||
path: '/api/v2/users/register',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: { email: 'newuser@example.com', password: 'SecurePass123!' },
|
||||
})
|
||||
.willRespondWith({
|
||||
status: 201,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: like({
|
||||
id: integer(1),
|
||||
email: string('newuser@example.com'),
|
||||
createdAt: string('2025-01-15T10:00:00Z'),
|
||||
}),
|
||||
})
|
||||
.executeTest(async (mockServer) => {
|
||||
const result = await registerUser({ email: 'newuser@example.com', password: 'SecurePass123!' }, { baseUrl: mockServer.url });
|
||||
expect(result.id).toEqual(expect.any(Number));
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Why test.skip():**
|
||||
|
||||
- Tests are written correctly for EXPECTED behavior
|
||||
@@ -163,6 +230,7 @@ Write JSON to temp file: `/tmp/tea-atdd-api-tests-{{timestamp}}.json`
|
||||
"knowledge_fragments_used": ["api-request", "data-factories", "api-testing-patterns"],
|
||||
"test_count": 3,
|
||||
"tdd_phase": "RED",
|
||||
"provider_scrutiny": "completed",
|
||||
"summary": "Generated 3 FAILING API tests for user registration story"
|
||||
}
|
||||
```
|
||||
@@ -205,6 +273,8 @@ Subagent completes when:
|
||||
- JSON output valid and complete
|
||||
- No E2E/component/unit tests included (out of scope)
|
||||
- Tests follow knowledge fragment patterns
|
||||
- Every Pact interaction has `// Provider endpoint:` comment (if CDC enabled)
|
||||
- Provider scrutiny completed or TODO markers added for new endpoints (if CDC enabled)
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
@@ -213,3 +283,4 @@ Subagent completes when:
|
||||
- Placeholder assertions (expect(true).toBe(true))
|
||||
- Did not follow knowledge fragment patterns
|
||||
- Invalid or missing JSON output
|
||||
- Pact interactions missing provider endpoint comments (if CDC enabled)
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-atdd
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/atdd
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-atdd
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
name: testarch-atdd
|
||||
name: bmad-testarch-atdd
|
||||
description: Generate failing acceptance tests using TDD cycle. Use when user says 'lets write acceptance tests' or 'I want to do ATDD'
|
||||
web_bundle: true
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
# Test Architect workflow: atdd
|
||||
name: testarch-atdd
|
||||
# Test Architect workflow: bmad-testarch-atdd
|
||||
name: bmad-testarch-atdd
|
||||
# prettier-ignore
|
||||
description: 'Generate failing acceptance tests using TDD cycle. Use when the user says "lets write acceptance tests" or "I want to do ATDD"'
|
||||
|
||||
@@ -13,10 +13,10 @@ document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/atdd"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
template: "{installed_path}/atdd-checklist-template.md"
|
||||
installed_path: "."
|
||||
instructions: "./instructions.md"
|
||||
validation: "./checklist.md"
|
||||
template: "./atdd-checklist-template.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-testarch-automate
|
||||
description: 'Expand test automation coverage for codebase. Use when user says "lets expand test coverage" or "I want to automate tests"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -191,6 +191,33 @@ Before starting this workflow, verify:
|
||||
- [ ] Error cases tested (400, 401, 403, 404, 500)
|
||||
- [ ] JWT token format validated (if auth tests)
|
||||
|
||||
### Consumer Contract Tests / CDC (If `use_pactjs_utils` Enabled)
|
||||
|
||||
**Provider Endpoint Comments:**
|
||||
|
||||
- [ ] Every Pact interaction has `// Provider endpoint:` comment
|
||||
- [ ] Comment includes exact file path to provider route handler, OR uses the TODO form when provider is inaccessible
|
||||
- [ ] Comment follows format: `// Provider endpoint: <path> -> <METHOD> <route>` or `// Provider endpoint: TODO — provider source not accessible, verify manually`
|
||||
|
||||
**Provider Source Scrutiny:**
|
||||
|
||||
- [ ] Provider route handlers and/or OpenAPI spec read before generating each interaction
|
||||
- [ ] Status codes verified against provider source (e.g., 201 not assumed 200)
|
||||
- [ ] Field names cross-referenced with provider type/DTO definitions
|
||||
- [ ] Data types verified (string ID vs number ID, date formats)
|
||||
- [ ] Enum/union values extracted from provider validation schemas
|
||||
- [ ] Required request fields and headers checked against provider validation
|
||||
- [ ] Nested response structures match provider's actual response construction
|
||||
- [ ] Scrutiny evidence documented as block comment in each test file
|
||||
|
||||
**CDC Quality Gates:**
|
||||
|
||||
- [ ] Postel's Law enforced: exact values in `withRequest`, matchers in `willRespondWith`
|
||||
- [ ] Response matchers (`like`, `eachLike`, `string`, `integer`) used only in `willRespondWith`
|
||||
- [ ] Provider state names are consistent with provider's state handler naming
|
||||
- [ ] DI pattern used for consumer function imports (actual consumer code, not raw `fetch()`)
|
||||
- [ ] One logical endpoint per Pact interaction (no multi-endpoint interactions)
|
||||
|
||||
### Component Tests (If Applicable)
|
||||
|
||||
- [ ] Component test files created in `tests/component/`
|
||||
@@ -467,6 +494,8 @@ All of the following must be true before marking this workflow as complete:
|
||||
- [ ] **Output file formatted correctly**
|
||||
- [ ] **Knowledge base references applied** and documented (including healing fragments if used)
|
||||
- [ ] **No test quality issues** (flaky patterns, race conditions, hardcoded data, page objects)
|
||||
- [ ] **Provider scrutiny completed or gracefully degraded** for all CDC interactions — each interaction either has scrutiny evidence or a TODO marker (if `use_pactjs_utils` enabled)
|
||||
- [ ] **Provider endpoint comments present** on every Pact interaction (if `use_pactjs_utils` enabled)
|
||||
|
||||
---
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
# Test Automation Expansion
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/automate`
|
||||
**Workflow ID**: `_bmad/tea/testarch/bmad-testarch-automate`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
@@ -40,11 +40,11 @@ From `workflow.yaml`, resolve:
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/automate/steps-c/step-01-preflight-and-context.md`
|
||||
`./steps-c/step-01-preflight-and-context.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/automate/steps-c/step-01b-resume.md`
|
||||
`./steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -77,6 +77,30 @@ Use CLI to explore the application and identify testable pages/flows:
|
||||
|
||||
---
|
||||
|
||||
**If `use_pactjs_utils` is enabled — Provider Endpoint Mapping (all stacks):**
|
||||
|
||||
When consumer-driven contract tests will be generated, build a Provider Endpoint Map during target identification. This applies to all `{detected_stack}` values — frontend, backend, and fullstack consumers all need provider scrutiny.
|
||||
|
||||
1. **Locate provider source and/or OpenAPI spec**: Scan workspace for provider project (from config, monorepo structure, or adjacent repositories). Also check for OpenAPI/Swagger spec files (`openapi.yaml`, `openapi.json`, `swagger.json`) — these document the provider's contract explicitly and can supplement or replace handler code analysis.
|
||||
2. **Map each consumer endpoint** to its provider counterpart:
|
||||
- Provider file path (route handler)
|
||||
- Route pattern (METHOD + path)
|
||||
- Validation schema location (Joi, Zod, class-validator) or OpenAPI request schema
|
||||
- Response type/DTO definition location or OpenAPI response schema
|
||||
- OpenAPI spec path (if available, e.g., `server/openapi.yaml`)
|
||||
3. **Output as "Provider Endpoint Map" table** in the coverage plan:
|
||||
|
||||
```markdown
|
||||
| Consumer Endpoint | Provider File | Route | Validation Schema | Response Type | OpenAPI Spec |
|
||||
| --------------------- | --------------------------------- | ------------------------- | ----------------------------------- | --------------- | ------------------------------------------------- |
|
||||
| GET /api/v2/users/:id | server/src/routes/userHandlers.ts | GET /api/v2/users/:userId | server/src/validation/user.ts | UserResponseDto | server/openapi.yaml#/paths/~1api~1v2~1users~1{id} |
|
||||
| POST /api/v2/users | server/src/routes/userHandlers.ts | POST /api/v2/users | server/src/validation/createUser.ts | UserResponseDto | server/openapi.yaml#/paths/~1api~1v2~1users |
|
||||
```
|
||||
|
||||
4. **If provider source not accessible**: Mark entries with `TODO — provider source not accessible` and note in coverage plan that provider scrutiny will use graceful degradation (see `contract-testing.md` Provider Scrutiny Protocol)
|
||||
|
||||
---
|
||||
|
||||
## 2. Choose Test Levels
|
||||
|
||||
Use `test-levels-framework.md` to select:
|
||||
@@ -74,6 +74,7 @@ const subagentContext = {
|
||||
detected_stack: '{detected_stack}', // "frontend" | "backend" | "fullstack"
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
provider_endpoint_map: /* from Step 2 coverage plan, if use_pactjs_utils enabled */,
|
||||
},
|
||||
timestamp: timestamp
|
||||
};
|
||||
@@ -190,6 +191,7 @@ When `use_pactjs_utils` is enabled, the API test generation subagent (step-03a)
|
||||
- **Provider verification tests**: Using `buildVerifierOptions` for one-call verifier setup
|
||||
- **Message contract tests**: Using `buildMessageVerifierOptions` if async/Kafka patterns detected
|
||||
- **Helper files**: Request filter setup with `createRequestFilter`, shared state constants
|
||||
- **Provider scrutiny**: Subagent reads provider route handlers, types, and validation schemas before generating each interaction (see `contract-testing.md` Provider Scrutiny Protocol)
|
||||
|
||||
When `pact_mcp` is `"mcp"`, the subagent can use SmartBear MCP tools to fetch existing provider states and generate tests informed by broker data.
|
||||
|
||||
@@ -97,8 +97,70 @@ test.describe('[Feature] API Tests', () => {
|
||||
- ✅ Generate request filter helpers in `pact/http/helpers/` using `createRequestFilter({ tokenGenerator: () => string })`
|
||||
- ✅ Generate shared state constants in `pact/http/helpers/states.ts`
|
||||
- ✅ If async/message patterns detected, generate message consumer tests in `pact/message/` using `buildMessageVerifierOptions`
|
||||
- ✅ **Provider endpoint comment MANDATORY** on every Pact interaction: `// Provider endpoint: <path> -> <METHOD> <route>`
|
||||
- ⚠️ **Postel's Law for matchers**: Use `like()`, `eachLike()`, `string()`, `integer()` matchers ONLY in `willRespondWith` (responses). Request bodies in `withRequest` MUST use exact values — never wrap request bodies in `like()`. The consumer controls what it sends, so contracts should be strict about request shape.
|
||||
|
||||
### 1.5 Provider Source Scrutiny (CDC Only)
|
||||
|
||||
**CRITICAL**: Before generating ANY Pact consumer interaction, perform provider source scrutiny per the **Seven-Point Scrutiny Checklist** defined in `contract-testing.md`. Do NOT generate response matchers from consumer-side types alone — this is the #1 cause of contract verification failures.
|
||||
|
||||
The seven points to verify for each interaction:
|
||||
|
||||
1. Response shape
|
||||
2. Status codes
|
||||
3. Field names
|
||||
4. Enum values
|
||||
5. Required fields
|
||||
6. Data types
|
||||
7. Nested structures
|
||||
|
||||
**Source priority**: Provider source code is most authoritative. When an OpenAPI/Swagger spec exists (`openapi.yaml`, `openapi.json`, `swagger.json`), use it as a complementary or alternative source — it documents the provider's contract explicitly and can be faster to parse than tracing through handler code. When both exist, cross-reference them; if they disagree, the source code wins. Document the discrepancy in the scrutiny evidence block (e.g., `OpenAPI shows 200 but handler returns 201; using handler behavior`) and flag it in the output JSON `summary` so it is discoverable by downstream consumers or audits.
|
||||
|
||||
**Scrutiny Sequence** (for each endpoint in the coverage plan):
|
||||
|
||||
1. **READ provider route handler and/or OpenAPI spec**: Find the handler file from `subagentContext.config.provider_endpoint_map` or by scanning the provider codebase. Also check for OpenAPI/Swagger spec files. Extract:
|
||||
- Exact status codes returned (`res.status(201)` / OpenAPI `responses` keys)
|
||||
- Response construction (`res.json({ data: ... })` / OpenAPI `schema`)
|
||||
- Error handling paths (what status codes for what conditions)
|
||||
|
||||
2. **READ provider type/model/DTO definitions**: Find the response type referenced by the handler or OpenAPI `$ref` schemas. Extract:
|
||||
- Exact field names (`transaction_id` not `transactionId`)
|
||||
- Field types (`string` ID vs `number` ID / OpenAPI `type` + `format`)
|
||||
- Optional vs required fields (OpenAPI `required` array)
|
||||
- Nested object structures (OpenAPI `$ref`, `allOf`, `oneOf`)
|
||||
|
||||
3. **READ provider validation schemas**: Find Joi/Zod/class-validator schemas or OpenAPI request body `schema.required`. Extract:
|
||||
- Required request fields and headers
|
||||
- Enum/union type allowed values (`"active" | "inactive"` / OpenAPI `enum`)
|
||||
- Request body constraints
|
||||
|
||||
4. **Cross-reference findings** against consumer expectations:
|
||||
- Does the consumer expect the same field names the provider sends?
|
||||
- Does the consumer expect the same status codes the provider returns?
|
||||
- Does the consumer expect the same nesting the provider produces?
|
||||
|
||||
5. **Document scrutiny evidence** as a block comment in the generated test:
|
||||
|
||||
```typescript
|
||||
/*
|
||||
* Provider Scrutiny Evidence:
|
||||
* - Handler: server/src/routes/userHandlers.ts:45
|
||||
* - OpenAPI: server/openapi.yaml paths./api/v2/users/{userId}.get (if available)
|
||||
* - Response type: UserResponseDto (server/src/types/user.ts:12)
|
||||
* - Status: 201 for creation (line 52), 400 for validation error (line 48)
|
||||
* - Fields: { id: number, name: string, email: string, role: "user" | "admin" }
|
||||
* - Required request headers: Authorization (Bearer token)
|
||||
*/
|
||||
```
|
||||
|
||||
6. **Graceful degradation** when provider source is not accessible (follows the canonical four-step protocol from `contract-testing.md`):
|
||||
1. **OpenAPI/Swagger spec available**: Use the spec as the source of truth for response shapes, status codes, and field names
|
||||
2. **Pact Broker available** (when `pact_mcp` is `"mcp"` in `subagentContext.config`): Use SmartBear MCP tools to fetch existing provider states and verified interactions as reference
|
||||
3. **Neither available**: Generate from consumer types but use the TODO form of the mandatory comment: `// Provider endpoint: TODO — provider source not accessible, verify manually`. Set `provider_scrutiny: "pending"` in output JSON
|
||||
4. **Never silently guess**: Document all assumptions in the scrutiny evidence block
|
||||
|
||||
> ⚠️ **Anti-pattern**: Generating response matchers from consumer-side types alone. This produces contracts that reflect what the consumer _wishes_ the provider returns, not what it _actually_ returns. Always read provider source or OpenAPI spec first.
|
||||
|
||||
### 3. Track Fixture Needs
|
||||
|
||||
Identify fixtures needed for API tests:
|
||||
@@ -145,6 +207,8 @@ Write JSON to temp file: `/tmp/tea-automate-api-tests-{{timestamp}}.json`
|
||||
],
|
||||
"fixture_needs": ["authToken", "userDataFactory", "productDataFactory"],
|
||||
"knowledge_fragments_used": ["api-request", "data-factories", "api-testing-patterns"],
|
||||
"provider_scrutiny": "completed",
|
||||
"provider_files_read": ["server/src/routes/authHandlers.ts", "server/src/routes/checkoutHandlers.ts", "server/src/types/auth.ts"],
|
||||
"test_count": 12,
|
||||
"summary": "Generated 12 API test cases covering 3 features"
|
||||
}
|
||||
@@ -185,6 +249,9 @@ Subagent completes when:
|
||||
- All API tests generated following patterns
|
||||
- JSON output valid and complete
|
||||
- No E2E/component/unit tests included (out of scope)
|
||||
- Every Pact interaction has `// Provider endpoint:` comment (if CDC enabled)
|
||||
- Provider source scrutiny completed or gracefully degraded with TODO markers (if CDC enabled)
|
||||
- Scrutiny evidence documented as block comments in test files (if CDC enabled)
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
@@ -192,3 +259,5 @@ Subagent completes when:
|
||||
- Did not follow knowledge fragment patterns
|
||||
- Invalid or missing JSON output
|
||||
- Ran tests (not subagent responsibility)
|
||||
- Pact interactions missing provider endpoint comments (if CDC enabled)
|
||||
- Response matchers generated from consumer-side types without provider scrutiny (if CDC enabled)
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-automate
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/automate
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-automate
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
name: testarch-automate
|
||||
name: bmad-testarch-automate
|
||||
description: Expand test automation coverage for codebase. Use when user says 'lets expand test coverage' or 'I want to automate tests'
|
||||
web_bundle: true
|
||||
---
|
||||
@@ -1,5 +1,5 @@
|
||||
# Test Architect workflow: automate
|
||||
name: testarch-automate
|
||||
# Test Architect workflow: bmad-testarch-automate
|
||||
name: bmad-testarch-automate
|
||||
# prettier-ignore
|
||||
description: 'Expand test automation coverage for codebase. Use when the user says "lets expand test coverage" or "I want to automate tests"'
|
||||
|
||||
@@ -13,9 +13,9 @@ document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/_bmad/tea/workflows/testarch/automate"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
installed_path: "."
|
||||
instructions: "./instructions.md"
|
||||
validation: "./checklist.md"
|
||||
template: false
|
||||
|
||||
# Variables and inputs
|
||||
6
_bmad/tea/workflows/testarch/bmad-testarch-ci/SKILL.md
Normal file
6
_bmad/tea/workflows/testarch/bmad-testarch-ci/SKILL.md
Normal file
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-testarch-ci
|
||||
description: 'Scaffold CI/CD quality pipeline with test execution. Use when the user says "lets setup CI pipeline" or "I want to create quality gates"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
# CI/CD Pipeline Setup
|
||||
|
||||
**Workflow ID**: `_bmad/tea/testarch/ci`
|
||||
**Workflow ID**: `_bmad/tea/testarch/bmad-testarch-ci`
|
||||
**Version**: 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
@@ -35,11 +35,11 @@ From `workflow.yaml`, resolve:
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/ci/steps-c/step-01-preflight.md`
|
||||
`./steps-c/step-01-preflight.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`{project-root}/_bmad/tea/workflows/testarch/ci/steps-c/step-01b-resume.md`
|
||||
`./steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user