Phases 5 – 8

Testing & Feedback

Verify that the artifacts are internally consistent, that the skill generates correctly, that the rules bundle evaluates correctly, and that every finding is fixed at its root cause.

Phase 5

Cross-artifact integrity testing

Goal: Verify that all artifact cross-references resolve and that the artifacts are internally consistent.

Run this phase programmatically. Human review alone will miss these checks at scale.

Fail condition: Any check that fails blocks progression to Phase 6 until fixed.

Automation: integrity_check.py

Run python scripts/integrity_check.py <artifacts_dir> from the repo root. The script loads the registry, taxonomy, terminology, all rules bundles, and all skill files in the directory, then runs all 9 checks and reports PASS/FAIL with specific failing values.

Exit code 0 = all checks pass, safe to proceed to Phase 6. Exit code 1 = fix failing checks first.

The 9 automated checks

Check	What it tests
INT-01	Every `component_ref` in the skill's assembly manifest resolves to a `component_type` in `component_registry.yaml`
INT-02	Every `rules_ref` in the registry resolves to ≥1 rule in a rules bundle (matched by `subcategories`)
INT-03	Every `component-level:X` subcategory value in any bundle has a matching `component_type` in the registry
INT-04	The `skill_ref` and `rules_bundle_ref` in each taxonomy entry resolve to actual files on disk
INT-05	No `component_type` in the registry is itself an unqualified prohibited term in the terminology file
INT-06	Every quality dimension in the bundle's `metadata.quality_scoring` lists only rule IDs that (a) exist in the bundle and (b) have severity ≤300
INT-07	All rules bundle files parse as valid JSON
INT-08	All rule `id` values within each bundle are unique
INT-09	All rule `id` values are within the declared `metadata.id_namespace` range

After a FAIL

Fix the failing upstream SSoT artifact (not a downstream reference to it), then re-run the script. The most common failures are INT-02 (orphaned rules_ref pointers) and INT-04 (broken file references).

Re-run after every artifact fix

Cross-artifact fixes can silently break references in other artifacts. After fixing any single artifact, always re-run the full integrity check — not just the check that originally failed.

Phase 6

Generation testing

Goal: Verify that the skill file, when used by an agent with no prior knowledge of the content type, produces correctly structured and correctly classified output.

Evaluator: A human reviewer with the test plan rubric.

Output: Per-test-case scores across dimensions A–E.

Minimum test case set

For a content type with N assembly configurations, create N+2 test cases minimum. For content types with multiple subtypes (like KC Articles), include at least one case per subtype plus one ambiguity case between the two most similar subtypes.

Case	Purpose
GEN-01 through GEN-N	One per assembly configuration — verifies the correct configuration is chosen and all required components are present
GEN-(N+1)	Rejection case — a topic that should be classified as an adjacent content type. The generator must refuse and redirect, not produce a misclassified output.
GEN-(N+2)	Conditional case — a topic with conditionally-required components (prerequisites, warning callouts) — verifies that conditional logic fires correctly

Evaluation dimensions

Dimension	Pass condition
A. Decision tree correctness	Correct content type and subtype classified; correct assembly configuration chosen; rejection cases correctly refused
B. Assembly completeness	All required components present; conditionally-required components present when conditions are met; optional components present only when appropriate
C. Component identity	Reuse classes correct; compositions correct; boundary conditions met for each component
D. Terminology compliance	All structural labels use preferred terms; no prohibited alternates present
E. Taxonomy alignment	Family correct; distinguishing criteria honored; metadata extension fields populated

Logging failures

For each failure, record: which test case, which dimension, the specific wrong output, the specific correct output, the likely cause, and the fix location. This log feeds the Phase 8 triage table.

Phase 7

Evaluation testing

Goal: Verify that the rules bundle produces correct verdicts on known-good, degraded, misclassified, and terminology-violation samples.

Evaluator: An LLM agent with the rules bundle, registry, and terminology as context. High automation potential.

Output: Per-sample verdicts with per-rule PASS / FAIL / FLAG / NOT_APPLICABLE.

Minimum test corpus — 7 samples

Case	Type	Purpose
EVAL-01	Known-good: GEN-01 output (post-manual-verification)	Calibrates the evaluator. Must PASS.
EVAL-02	Known-good: a real published corpus example	Tests against production content. May have legitimate sev-400 flags. Must PASS.
EVAL-03	Degraded: remove one required component	Tests sev-300 blocking rule detection. Must FAIL.
EVAL-04	Degraded: violate a component-level formatting rule	Tests sev-400 flag detection. Must PASS with FLAGS.
EVAL-05	Degraded: violate a length or readability rule	Tests sev-500 suggestion detection. Must PASS with suggestions.
EVAL-06	Misclassified: a different content type labeled as this one	Tests taxonomy boundary detection via classification rules. Must FAIL.
EVAL-07	Terminology violation: prohibited alternate in a heading or label	Tests Step 0 terminology check. Must FLAG or FAIL per rule severity.

Verify EVAL-01 before using it as a calibration baseline

Manually check EVAL-01 against the rules bundle before using it to calibrate the evaluator. If it fails any sev-300 rule, the sample is wrong — fix the sample, not the evaluator. An evaluator that "incorrectly" flags the baseline is often correct.

Verdict logic

PASS = zero sev ≤300 rules violated.
       Sev-400 flags and sev-500 suggestions are reported alongside
       the verdict as recommended fixes; they do not change the verdict.

FAIL = any sev ≤300 rule violated.
       Author must obtain sign-off or fix the violation before publishing.

A PASS with sev-400 flags is a valid, publishable result. The flags are a punch list for the author's discretion, not a gate.

Per-rule evaluation protocol

Step 0 — Terminology check (run once, before the rule pass). Cross-reference every authored heading, label, and schema-facing term against the terminology file's prohibited lists. Flag any prohibited alternates. This step is explicit and mandatory — it will not emerge from the rule-scoped pass on its own.
Scope match. Does this rule apply? Check subcategories against what's present. (doc-level → always; component-level:{x} → only if that component is present.)
Applicability. Is the rule's use type relevant to what's being checked?
Judgment. Does the content satisfy the rule? → PASS / FAIL / NOT_APPLICABLE / FLAG
Evidence. Quote the specific text that passes or violates.

Verdict mismatch triage

Mismatch type	Likely cause	Fix location
Evaluator FAILs known-good content	Rule too broad; scope too wide; template violates its own rule	Rules bundle (`prompt_information`, `subcategories`, examples) or skill file templates
Evaluator PASSes known-bad content	Rule missing; scope too narrow; violation in an uncovered component	Rules bundle (add or expand rule)
Evaluator cites wrong rule	Rule ID mapping stale; rule text ambiguous	Rules bundle (`related_rules_note`, rule text)
Evaluator flags valid content at sev 300	Severity miscalibrated; rule definition too strict	Rules bundle (severity + rule text)

Phase 8

Feedback loop

Goal: Work through all findings from Phases 5–7, fix each at its root cause, and re-verify.

Exit condition: Zero open findings.

Finding triage table

For each finding, assign it to exactly one fix location.

Finding type	Fix location
Generator produces wrong structure	Skill file (instructions, output templates, validation criteria)
Generator uses wrong term	Terminology file (add example or clarification to existing entry)
Template in skill violates its own rules	Skill file templates and rules bundle — fix both, verify they agree
Evaluator misses a violation	Rules bundle (add or expand rule, broaden scope)
Evaluator flags valid content	Rules bundle (narrow scope, add `prompt_information`, adjust severity)
Severity miscalibrated	Rules bundle (change `severity`, update `metadata.quality_scoring`)
Cross-artifact reference broken	The upstream SSoT artifact — fix the reference, re-run INT checks
Verdict logic inconsistency	Test plan (not a defect in the artifacts themselves)

Re-testing sequence after a fix

Always re-run INT checks (Phase 5) first after fixing any artifact. Cross-artifact fixes can silently break references in other artifacts.
After fixing a rules bundle rule, re-run the specific EVAL case(s) that exposed the finding.
After fixing a skill file template, re-run GEN-01 to confirm the template still produces valid output.
After fixing terminology, re-run EVAL-07.

Exit criteria

All INT checks (Phase 5) pass with zero failures
All generation test cases produce correct output — correct classification, assembly, components, terminology, taxonomy alignment
GEN-01 output (post-manual-verification) passes the evaluation protocol with zero sev ≤300 findings
All evaluation test cases produce the expected verdict: EVAL-03/06 FAIL; EVAL-04/05 PASS with flags; EVAL-07 flags per rule severity; EVAL-01/02 PASS
Zero open findings in the feedback loop table

Artifact Authoring