Skip to main content

Confirmed policy effects

Target operating model — recommended contracts, not installed collection. All owner and cadence entries below are proposals to confirm before the pilot, not assigned responsibilities or approved schedules.

Prototype calculation — no live telemetry. The synthetic example calculates this metric; it has no network collectors.

Eligible evaluated attempts with an actual confirmed prohibited effect; report attempts, blocked attempts and effects separately.

ContractDefinition
NumeratorDistinct attempt_ids with confirmed prohibited action effect
DenominatorAll evaluated eligible attempt_ids including failed and abandoned runs
ExpressionN / D * 100
Aggregationpercent
Unit%
Eligible populationGoverned skill trials or supported workflow attempts, reported separately for eval and live populations.
ExclusionsOut-of-scope runs; detection gaps remain coverage issues, not exclusions from known initiated population.
Events and sourceattempt.started; policy.violation; gate.blocked; security audit and incident triage
Owner / cadenceArchitecture & Security / Immediate incident action; weekly aggregates
Missing evidenceUnknown outcomes remain visible; zero detected effects never proves safety. Missing audits make rate provisional.
Gaming and limitsHide incidents or inflate harmless trials; absolute severity counts and fixed task mix accompany rate. Descriptive signal, not causal AI productivity attribution. Source completeness, task mix and sampling limit inference; report uncertainty. Blocked attempts are not actual effects; any critical effect blocks promotion regardless of aggregate rate.

Read the shared collection and safeguards before collecting data; see the synthetic example and limits.

Return to Skills Measurement.