D-V-R-K Pathway
Evaluation Design shapes Uncertainty Visibility, which conditions Reliance Behavior, which propagates to Deployment Risk (EVF Paper 2, Fig. 6)
The Design - Visibility - Reliance - Risk Chain
The D-V-R-K chain (EVF Survey, Fig. 6) traces how AI evaluation design decisions propagate through uncertainty visibility and human reliance patterns to ultimately determine deployment risk. Each stage is governed by one or more of the five invariants (I1--I5), and AIJIM maps each link to concrete Audit Bundle data.
The Four Stages
Stage 1: Data (Raw Evidence)
Raw, unprocessed evidence collected from sources: documents, interviews, databases, images, or other primary material.
Examples:
- Corporate financial records or leaked datasets
- Interview transcripts with sources
- Social media posts or archived URLs
- Images, videos, or geolocation metadata
- Scientific papers or regulatory filings
Governed by: I1 (Traceability)
I1 Requirement: Complete seed data must be captured with full provenance (source, access date, method of collection, chain of custody).
- What was collected? (complete dataset or sample)
- When was it collected? (timestamp and timezone)
- How was it collected? (API, scraping, manual, etc.)
- Who had access? (authentication, authorization)
- Is it reproducible? (can someone else fetch the same data today?)
Stage 2: Verification (Analysis & Validation)
Processing: apply analysis methods (NLP, NLI, statistical tests, fact-checking) to verify claims against the data. This is where human judgment and algorithmic reasoning intersect.
Examples:
- NLI to test whether data supports a claim
- Statistical significance tests on metrics
- Timeline reconstruction to check logical consistency
- Fact-checking against multiple sources
- Expert review with documented reasoning
Governed by: I2 (Verifiability) & I4 (Configurability)
I2 Requirement: Analysis must be deterministically reproducible with identical inputs producing identical outputs.
- What model or method was used? (model name, version, checkpoint)
- What were the exact parameters? (temperature, seed, thresholds)
- Can someone else run the analysis and get the same result?
I4 Requirement: All configuration decisions must be documented and justifiable.
- Why this model and not another? (model selection rationale)
- Why these parameters? (sensitivity analysis, baseline justification)
- What would happen if parameters changed? (documented uncertainties)
Stage 3: Reporting (Artifact Completeness)
Packaging: format the verified analysis as reportable artifacts that communicate the finding and its uncertainty to readers/decision-makers.
Reportable Artifacts Include:
- Primary Finding: The main claim (e.g., "This entity is linked to X activity")
- Confidence Interval / Certainty Level: How sure are we? (numerical or categorical)
- Supporting Evidence: Specific data points and analysis results
- Decision Tree / Reasoning: How did we arrive at this conclusion?
- Limitations & Caveats: What could invalidate this finding?
- Dissenting Views: What would argue against this conclusion?
Governed by: I3 (Reportability)
I3 Requirement: Every reportable claim must include confidence intervals, decision logic, and artifact completeness validation.
- Is the claim quantified with a confidence interval or certainty level?
- Can a reader understand how the conclusion was reached?
- Are limitations and uncertainties explicitly stated?
- Could someone reconstruct the analysis from the artifacts?
Stage 4: Knowledge (Decision & Publication)
Integration: the verified, fully-reported finding becomes a publishable knowledge claim that can inform decisions, inform other research, or be embedded in public discourse.
Knowledge Outputs Include:
- Published articles with full evidence appendices
- Policy recommendations based on verified findings
- Reusable datasets and methodologies for other researchers
- Updated knowledge graphs or decision systems
- Public records and transparency commitments
Governed by: I5 (Auditability)
I5 Requirement: All decisions must be immutably logged with impact metrics, enabling independent verification and learning.
- Can external parties verify the published claim against the original data and analysis?
- Are all decisions, changes, and corrections logged?
- Can we measure the real-world impact of this knowledge claim?
- Can we learn from successes and failures to improve future investigations?
Invariant Mapping to Chain Stages
| Invariant | Primary Stage | Requirement | Verification Gate |
|---|---|---|---|
| I1: Traceability | Data | Complete seed data with provenance and chain of custody | Seed capture at ingestion |
| I2: Verifiability | Verification | Deterministic reproducibility; identical inputs → identical outputs | Reproducibility test before reporting |
| I3: Reportability | Reporting | Confidence intervals, decision logic, artifact completeness | Artifact validation before publication |
| I4: Configurability | Verification | All configuration decisions documented and justifiable | Config review during method selection |
| I5: Auditability | Knowledge | Immutable audit logs with impact metrics and decision traceability | Audit trail logging throughout pipeline |
Visual Pipeline Flow
Example: Investigative Claim
Here's how a real investigative claim flows through the D→V→R→K chain:
D — Data: "Company X received $5M wire transfer from Entity Y on 2025-03-15"
Source: Bank records (seed), access date: 2025-10-20, method: FOIA request, verified by compliance officer.
V — Verify: NLI model checks claim against offshore registry + known shell company patterns
Model: OpenAI GPT-4 (checkpoint Dec 2025), temp=0.3, seed=42. Result: 87% confidence that Entity Y is shell company. Threshold: 80%. PASS Pass.
R — Report: "Company X received wire from likely shell entity (87% confidence, CI: 81–93%)"
Decision tree: [Bank record found?] → [Shell company match?] → [Confidence > 80%?] → Report. Limitation: Cannot verify beneficial ownership without additional sources.
K — Knowledge: Published article with full reproducibility appendix; other researchers can verify using same bank records + model
Audit log: Decision recorded, impact tracked (legal action initiated? Media attention?). Future investigations can reuse this pattern.
Failure Modes & Recovery
The chain can break at any stage. AIJIM's enforcement architecture catches breaks early:
- Data Stage Failure (I1): If seed data cannot be captured (e.g., restricted source), the system marks the claim as "non-reproducible" rather than silently proceeding.
- Verification Stage Failure (I2/I4): If analysis cannot be deterministically reproduced or configuration is not justified, the claim is flagged for manual review.
- Reporting Stage Failure (I3): If confidence intervals or decision logic cannot be provided, the claim cannot be published without explicit editorial override.
- Knowledge Stage Failure (I5): If audit trail is incomplete, the claim's impact cannot be measured, limiting future learning.
The D→V→R→K chain is the backbone of AIJIM's evidence-first architecture. Every investigation flows through it, and every stage is governed by formally verified invariants.