[Feature]: Standard evaluator contract for evidence, provenance, uncertainty, and recovery

[tbitcs]

Problem Statement

Spec Kit already has a strong lifecycle and an extensible hook system, but there is no common contract for extensions that evaluate the quality or trustworthiness of artifacts between phases.

That means every evidence, provenance, factuality, security, or governance extension has to invent its own answers to questions such as:

• What did the evaluator find?
• Is the result a pass, warning, iteration request, or hard block?
• What source or artifact supports the finding?
• Is a statement observed, inferred, contradicted, or simply unsupported?
• What should the workflow do next?
• How should evaluator state survive pause/resume or context reduction?
• How do multiple independent evaluators compose?

This is closely related to #23, where the discussion moved from a single provenance integration toward a generic registerable “linter” / “evaluator” concept that could run after workflow phases and trigger iteration.

I think the missing piece is not another workflow engine. It is a small, provider-neutral evaluator result contract that existing hooks and extensions can use.

Proposed Solution

Define a standard evaluator protocol that extensions can implement while continuing to use Spec Kit's existing lifecycle and hook execution model.

The smallest useful contract would let an evaluator:

1. register for one or more lifecycle events;
2. receive the relevant resolved artifacts plus stable source/provenance references;
3. return a versioned machine-readable result;
4. distinguish observed evidence from generated assertions;
5. preserve contradictory findings instead of forcing false consensus;
6. represent insufficient evidence or unresolved uncertainty explicitly;
7. request a bounded next action such as pass, warn, iterate, clarify, gather_evidence, or block;
8. persist enough compact state to survive pause/resume;
9. compose deterministically with other evaluators;
10. remain implementation-neutral: deterministic, model-backed, local, remote, private, paid, or hybrid.

The current hook system can remain the execution mechanism. I am mainly proposing a shared evaluator result schema and lifecycle semantics, not a second orchestration layer.

Illustrative result

{
  "schema_version": "1.0",
  "evaluator": {
    "id": "epistemic",
    "version": "0.1.0"
  },
  "phase": "after_plan",
  "outcome": "iterate",
  "summary": "Two high-impact claims are unsupported and one requirement conflicts with an accepted constraint.",
  "findings": [
    {
      "id": "EPI-001",
      "severity": "high",
      "kind": "unsupported_claim",
      "subject": "REQ-014",
      "evidence_refs": [],
      "provenance_refs": ["spec.md#REQ-014"],
      "uncertainty": "insufficient_evidence",
      "recommended_action": "gather_evidence"
    }
  ],
  "next_action": {
    "kind": "iterate",
    "target_phase": "plan"
  }
}

Alternatives Considered

Existing hooks only

This is possible today and is probably the right execution substrate, but hooks alone do not define a common result format, outcome semantics, provenance model, or composition behavior.

Add the full epistemic implementation to Spec Kit core

I do not recommend this. Spec Kit should stay implementation-neutral. A generic evaluator contract should be able to support provenance engines, security checks, formal verification, factuality systems, enterprise policy, deterministic linters, and future approaches.

Use /speckit.analyze for this

/speckit.analyze addresses artifact consistency and coverage. That is complementary to questions of evidence provenance, unsupported claims, contradictory evidence, uncertainty, and recovery actions.

Keep this entirely outside Spec Kit

That works for one extension, but it prevents independently developed evaluators from composing around a shared lifecycle/result protocol.

Component

Specify CLI (initialization, commands)

AI Agent (if applicable)

All agents

Use Cases

1. Evidence-backed specification
   After /speckit.specify, identify statements presented as facts without supporting evidence and route them to clarification or research.

2. Contradiction preservation
   Preserve evidence supporting opposing propositions instead of allowing a generated summary to manufacture consensus.

3. Plan stress testing
   Evaluate important assumptions and failure modes after planning before implementation cost is incurred.

4. Observed test evidence
   Distinguish “the agent says the test passed” from “this command was observed at revision X and produced artifact Y.”

5. Safe pause/resume and context reduction
   Preserve unresolved findings, authoritative decisions, evidence references, and uncertainty without treating a generated summary as new evidence.

6. Independent evaluator composition
   Run schema, security, policy, and epistemic evaluators at the same lifecycle point using one portable result contract.

7. Private or regulated environments
   Allow organizations to register private evaluators without coupling Spec Kit to a specific model provider or service.

8. CI/headless use
   Emit evaluator results as JSON artifacts, CI annotations, reports, or release gates.

Acceptance Criteria

• A documented evaluator result schema exists with evaluator identity/version, lifecycle phase, outcome, findings, evidence/provenance references, and requested next action.
• Existing hooks/extensions remain backward compatible and can act as the execution mechanism.
• The protocol can represent at least pass, warn, iterate, clarification/evidence gathering, and block.
• Multiple evaluators at one lifecycle point have deterministic composition/precedence behavior.
• Results survive workflow pause/resume and are usable in headless/CI execution.
• The protocol does not automatically treat model-generated assertions as observed evidence.
• The protocol does not require hidden chain-of-thought; concise findings, rationale, evidence, and provenance are sufficient.
• The protocol remains provider-neutral and agent-neutral.
• Documentation includes a minimal evaluator-authoring example.
• A reference extension demonstrates the contract without making Spec Kit core depend on the reference implementation.

Additional Context

Relationship to #23

Issue #23 proposed integrating VeriTrail for hallucination detection and provenance. In the discussion, @tiagostutz suggested generalizing the idea into registerable linters/evaluators that could run after phases and trigger iteration.

This proposal is intended as a concrete continuation of that direction, not a competing concept.

• Issue: Proposal: Integrate VeriTrail for hallucination detection & provenance in SpecKit #23
• Evaluator/linter comment: Proposal: Integrate VeriTrail for hallucination detection & provenance in SpecKit #23 (comment)

Existing implementation I can contribute from

I maintain an MIT-licensed epistemic implementation in Layer1Labs Specsmith:

• https://github.com/layer1labs/specsmith/tree/main/src/epistemic
• https://github.com/layer1labs/specsmith/blob/main/docs/site/epistemic-library.md
• https://github.com/layer1labs/specsmith/blob/main/.specsmith/governance/EPISTEMIC-AXIOMS.md

The existing library includes typed belief/claim artifacts, epistemic boundaries, evidence handling, adversarial stress testing, contradiction detection, failure-mode graphs, certainty scoring, recovery proposals, persistent sessions, and hash-chain trace sealing.

I am not proposing to upstream Specsmith or make Spec Kit depend on it. I am offering it as working reference code that can be adapted into a Spec Kit extension.

Applied implementation lessons

In a separate private implementation, several requirements have proven important enough that they may be useful when defining the evaluator boundary:

• generated assertions must remain distinguishable from observed evidence;
• model self-attestation should not satisfy an evidence gate by itself;
• contradictions should be preserved rather than collapsed into a single synthesized answer;
• requirements, accepted decisions, unresolved claims, approvals, evidence references, provenance, and uncertainty need to survive context reduction and workflow resume;
• when evidence is insufficient, an evaluator should be able to request clarification, evidence gathering, abstention, escalation, iteration, or blocking rather than inventing certainty;
• deterministic checks should run before probabilistic review where appropriate;
• higher-risk work may require evaluator or verifier independence rather than allowing the same model/family to generate and certify the result.

These are examples of policies that a generic evaluator contract should be able to express. They are not proposed as mandatory Spec Kit behavior.

Contribution path

I am opening the issue before attempting a large PR because this would affect the extension/workflow contract.

If maintainers prefer to validate the idea entirely as a Community Extension first, I am happy to implement it against the current hook APIs, test it on real projects, publish the extension, and use that evidence to determine whether a small standardized evaluator contract deserves promotion into core.

I can contribute the existing MIT-licensed code, schemas, integration glue, tests, documentation, and real-project validation.

AI-assistance disclosure

This issue was drafted with ChatGPT assistance. I supplied and reviewed the existing implementation, project requirements, architecture, and contribution direction. ChatGPT helped inspect Spec Kit's current contribution guidance and extension model, locate the prior #23 discussion, and structure the proposal.

I will personally review, understand, test, and maintain any implementation I submit.