Skills quality
TO-BE · Target operating model — not installed controls or company results.
Optional skills reference for the 6 Months Roadmap. Use this when skills are the chosen intervention, not as a prerequisite for discovery or coaching. Department names denote functions to map to actual willing staff; the architect cannot assign them authority.
What evidence is enough to promote a skill?
Risk determines the evaluation depth, not whether safety checks apply. Independent evidence must show useful outcomes and no failed mandatory gate before promotion.
- R1: approved nonsensitive drafting. R2: code or customer-impacting material. R3: privileged or sensitive effects; human-authorized, outside initial autonomous scope.
- No averaging away harm: a critical unauthorized effect blocks promotion.
- Proposed floors need owner ratification: missing coverage or acceptance rules means qualification is incomplete.
Prove the candidate fits the workflow
Bring the workflow/acceptance map from Baseline and the pinned candidate, client tuple and authority envelope from Governance. In the illustrative Delivery workflow, test whether a candidate actually produces a correct bounded change and regression tests—not just whether it was selected. Compare the candidate with declared current AI-assisted practice on the same starting artifacts; add the current-approved revision when one exists. Include misleading repository instructions, missing tests and requests to bypass review.
First-revision rule: if no approved skill exists, record approved-comparator regression N/A: first revision, never fabricate an approved bundle. The comparator is documented current practice without the candidate, with existing AI/context recorded—not a no-AI baseline. Test disable/remove, session invalidation/reload and recovery to that practice in a pinned clean environment. The independent reviewer accepts this evidence and the actual workflow approver authorizes only the bounded pilot; mandatory safety and acceptance checks are unchanged. Once the first revision is approved, later revisions require regression against it. See EXP-01 for the assembled packet.
The independent Quality Engineering reviewer records accepted and failed artifacts, safety results, coverage and regression differences for each named revision. The owning team's designated approver then decides whether that evidence supports a bounded pilot; Architecture & Security must accept any higher-risk boundary. A mandatory failure or an unagreed acceptance rule means rework or hold. These are proposed decision responsibilities, not already assigned people.
Qualification artifact success is not live task success. The evaluation denominator is planned executed trials, retaining failures and abandonment, with omitted/unexecuted cases shown as coverage gaps. Validated task success instead uses matured real-workflow starts. Never pool them. Measurement links selection and health signals to this fitness evidence; neither usage nor a fresh review date proves effectiveness.
The output is a versioned qualification report and explicit limitations for Adoption, not approval of every future task. Maintenance feeds real failures and material client/model/dependency changes back into this same evaluation, including regression against the currently approved revision.
Evaluation design, adversarial cases and interpretation
TO-BE · Target operating model. Qualification supports the Stage 01 journey; these pages do not describe installed controls or measured company results.
Quality has four separate questions: was the right skill selected, was the procedure executed safely, did the artifact pass independent checks, and did the workflow produce useful outcomes? A successful invocation answers none of the remaining questions by itself.
The proposed qualification contract uses three tiers. R1 is read-only or drafting with nonsensitive approved material; R2 creates code or material with customer impact but no privileged external effect; R3 touches privileged, financial, sensitive-personal or production decisions. R3 remains human-authorized and out of the initial autonomous scope.
Every tier requires all mandatory deterministic safety, schema, authorization and provenance checks to pass, with no observed critical unauthorized action. An observed critical effect blocks promotion. The proposed pilot evaluation floors below set success thresholds; these are design defaults for owner ratification, not externally validated guarantees or approved service levels. If sample coverage, acceptance criteria or responsible reviewers are missing, qualification remains incomplete. No aggregate score can trade off a failed mandatory safety check.
Use sanitized, predeclared task sets containing positive triggers, near misses, negative cases and the normal workload mix. Compare current practice, candidate and (when it exists) current-approved paths with the same starting artifacts; repeat stochastic trials and retain failed and abandoned attempts. A no-standardized-skill path may still use AI. Independently inspect output state. Deterministic tests grade checkable constraints; calibrated humans grade usefulness, maintainability and experience. A model judge is supplementary, versioned and checked against human disagreements, never the sole safety or promotion authority.
Include adversarial repository text, malicious referenced code, wrong permissions, client toggles, name shadowing, unsupported versions, hook timeout/crash/malformed response, stale sessions, offline recovery and rollback. Run central checks against forged/missing client declarations. Hook presence is not fail-closed enforcement: failure semantics must be tested on each client, and central authorization remains independent.
Report denominators, task mix, trial counts, per-client results and uncertainty. A passing benchmark is evidence within a tested envelope, not proof that all future tasks are safe. Material model, client, adapter or dependency changes trigger requalification.
Proposed R1 / R2 / R3 sample sizes and thresholds
Proposed qualification floors
These are optional reference designs, not mandatory startup floors, approved service levels or guarantees. Before collection, the owner and independent reviewer choose a risk-appropriate design, required coverage, regression margin, uncertainty/precision rule and capacity cap. Start with one workflow and one client configuration. Cost corpus preparation, independent grading, repeated trials and requalification; if useful precision is infeasible, narrow the question to feasibility rather than claim benefit or weaken safety. All mandatory checks must pass regardless of aggregate score. The numbers below must not silently become a solo architect's required workload.
R1
Distinct tasks — 30
Trials per task/client — 3
Artifact success — 90%
Selection precision / recall — 90% / 90%
R2
Distinct tasks — 60
Trials per task/client — 3
Artifact success — 95%
Selection precision / recall — 95% / 90%
R3
Distinct tasks — 100
Trials per task/client — 5
Artifact success — 99%
Selection precision / recall — 99% / 95%
Artifact success denominator is every planned executed trial, retaining failures and abandonment. Trials of same task are correlated: show per-task and per-client strata; do not inflate independent sample size. Selection floors use independent labels. Report 95% uncertainty intervals clustered by task for outcome comparisons; no regression versus the current approved path, when one exists, beyond a preregistered owner-approved margin. First-revision approved regression is N/A with reason, but current-practice acceptance and safety remain mandatory. Missing applicable margin/precision rule prevents promotion. Small feasibility studies cannot establish that rule after seeing favorable results.