Paper 2: EVF Survey
A systematic survey of 140 AI evaluation studies revealing massive reporting gaps in high-stakes domains -- the empirical foundation for the AIJIM invariants and the Audit Bundle transparency assessment.
The Reporting Gap Problem
When AI systems are deployed in high-stakes domains -- journalism, medicine, legal, compliance -- their outputs directly influence decisions that affect people. Yet the studies evaluating these systems routinely omit the most basic information needed to verify, reproduce, or audit their claims.
Key Findings: Four Gap Indicators
The survey analyzed 140 peer-reviewed studies across multiple AI evaluation domains. Four critical gap indicators emerged, each representing a transparency failure that AIJIM now explicitly tracks and enforces in the Audit Bundle.
Three-Tier Visibility Taxonomy
Based on the four gap indicators, the survey classifies AI evaluation transparency into three tiers. AIJIM computes this classification automatically for every Audit Bundle and displays it as the EVF Type badge.
Opaque
3--4 gap indicators absent. The AI analysis process is essentially a black box -- no seeds, no confidence intervals, no verifiable artifacts, no judge documentation.
In AIJIM: Audit Bundle shows red badge. Most EVF checks fail.
Partial Transparency
1--2 gap indicators absent. Some transparency exists (e.g., model documented, artifacts hashed) but gaps remain in seed logging or confidence intervals.
In AIJIM: Audit Bundle shows amber badge. Common state for early-stage stories.
Full Transparency
0 gap indicators absent. All four indicators present: seeds logged, confidence intervals included, all artifacts hash-verified, judge fully documented.
In AIJIM: Audit Bundle shows green badge. Achieved after complete ECAMX analysis with stability runs.
The D-V-R-K Pathway (Fig. 6)
The EVF pathway view (Fig. 6) is a non-causal interpretive lens. Evaluation Design shapes Uncertainty Visibility, which conditions Reliance Behavior, which propagates to Deployment Risk. AIJIM maps each link to concrete bundle data.
Each link is assessed independently in the Audit Bundle. The chain visualizes how evaluation design decisions (D) propagate through uncertainty visibility (V) and human reliance patterns (R) to ultimately determine deployment risk (K).
Table VI: Minimum Reporting Checklist
The survey proposes five minimum reporting requirements. AIJIM automatically checks each one and displays the results in the Audit Bundle export panel.
All seeds used in AI analysis must be recorded for reproducibility.
AI model name, version, and decision thresholds must be documented.
Every claim must link to specific evidence items with NLI labels.
All generated artifacts must carry SHA-256 hashes for independent verification.
Results must include confidence intervals or stability test artifacts.
Implementation in AIJIM
The EVF classification is computed from real bundle data, not hardcoded. The implementation lives in the Audit Bundle feature:
Gap indicators derived from: judgeConfig, artifacts[], evidence[]
Classification: computeEvfFromBundle() in features/audit-bundle/lib/evf.ts
UI: "AI Transparency Assessment" section in AuditBundleExportPanel
D-V-R-K chain: mapped to bundle fields (seeds, stability, HITL, signatures)
Table VI: 5-item checklist derived from real claim/evidence/artifact data
Connection to AIJIM Invariants