AIJIM Evidence Workspace — Dokumentation

Classification of Transparency Levels

The Visibility Taxonomy is a framework for classifying how transparent AI systems are in their reasoning and evidence practices. It ranges from completely opaque systems to fully verifiable ones, with clear intermediate categories. This taxonomy is foundational to AIJIM's mission and directly addresses the gaps identified in the Evidence Verification Framework (EVF) Survey (Paper 2).

Why This Matters: Most AI systems used in evidence evaluation fall into the "Opaque" or "Partial" categories. AIJIM enforces the "Verifiable" level by design through invariants I2 and I3.

The Five Visibility Levels

Opaque: No Artifacts

System produces a binary output (claim true/false or confidence score) with no explanation, seed data, or methodology documentation.

Characteristics:

Output only: "Claim is TRUE" or "Confidence: 82%"
No explanation of reasoning
No seed data or evidence references
No model or method documentation
Cannot be reproduced or audited

Real-World Example:

A fact-checking bot returns "CLAIM LABELED FALSE" with no explanation of which fact-checking sources were consulted, what methodology was used, or how confidence was calculated.

EVF Gap Prevalence

Studies falling into this category fail to document: seeds (91.4%), confidence intervals (97.1%), artifacts (72.9%), judge configuration (100%).

Partial: Some Reporting

System provides a primary output plus some supporting information, but critical elements are missing or incomplete.

Characteristics:

Output with brief explanation
Some evidence references (e.g., "found in 3 sources")
Partial methodology documentation
Missing seed data or reproducibility instructions
Confidence not quantified or missing error bars

Real-World Example:

A fact-checker reports "CLAIM MIXED: 2 sources support, 1 source contradicts" but does not provide: links to those sources, the search methodology, the selection criteria, or confidence intervals around the support/contradict split.

What's Still Missing?

Readers cannot independently verify because they lack: exact sources (seeds), the method for selecting sources, quantified confidence, or details about how disagreement was resolved.

Transparent: Full Artifacts

System provides complete methodology, artifact documentation, and reasoning. Humans (or other systems) can understand the logic and audit the process, but reproducibility requires significant manual effort.

Characteristics:

Output with full explanation and decision tree
Complete artifact list: all sources used
Methodology fully documented
Confidence intervals provided
Still missing: exact seed data, model version, parameter reproducibility

Real-World Example:

A published fact-check includes: all 3 sources (with URLs), the selection criteria (keywords used, date ranges), confidence (73% ± 8%), and decision logic ("2 sources more credible because peer-reviewed"). Missing: exact fetch date/time for URLs, how the model determined credibility, whether rerunning today would get the same result.

Why Not Reproducible Yet?

Without exact seed data (fetch timestamps, URLs as they existed), the sources may have changed, making bit-for-bit reproduction impossible. Humans can audit the logic, but machines cannot repeat the analysis deterministically.

Verifiable: Reproducible with Seeds & Config

AIJIM's Target Level: System provides complete seed data, configuration, and artifacts. Independent parties can reproduce analysis deterministically and verify every claim.

Characteristics:

Complete seed data with provenance (timestamps, URLs, access methods)
Full configuration documentation (model name, version, parameters, seed value)
All artifacts: sources, decision logic, confidence intervals, limitations
Decision trees and reasoning fully specified
Identical inputs + identical config → identical outputs (guaranteed)
Can be reproduced by anyone with access to the data and model

Real-World Example:

AIJIM fact-check publishes: all 3 sources (with exact URLs + fetch timestamp 2025-10-20T14:32:15Z), selection criteria, model metadata (GPT-4, checkpoint Dec 2025, temp=0.3, seed=42), decision tree, confidence (73% ± 8%), and audit trail. A researcher in 2026 can: (1) Fetch the same URLs with wayback machine, (2) Run GPT-4 with identical config, (3) Get the same 73% result, (4) Verify the reasoning, (5) Challenge or extend the analysis.

Invariants Enforced

I2 (Verifiability): Exact seeds + config guarantee reproduction
I3 (Reportability): Decision tree, confidence intervals, limitations all present

Formally Verified: Machine-Checkable Proofs

(Future roadmap) System provides machine-verifiable proofs of correctness. Beyond reproducibility: guarantees that the analysis is logically sound and bounded error.

Characteristics:

All of Level 4 (Verifiable) plus:
Formal specifications of correctness properties
Machine-checkable proofs (e.g., SAT solvers, theorem provers)
Bounded error guarantees
Can prove: "All paths through decision logic are consistent with data"

Example (Conceptual):

"Formal proof that any claim with > 80% confidence is supported by at least 2 of 3 sources, using decision logic D and data S, for all possible values in parameter range P." This is beyond current practice but represents the ultimate transparency goal.

Visibility Levels Compared

Level	Output	Seeds?	Config?	Artifacts?	Reproducible?
1: Opaque	Score only	FAIL No	FAIL No	FAIL No	FAIL No
2: Partial	Brief explanation	WARN Partial	WARN Partial	WARN Some	FAIL No
3: Transparent	Full explanation	PASS Yes	WARN Partial	PASS Yes	WARN Manual
4: Verifiable	Complete documentation	PASS Yes	PASS Yes	PASS Yes	PASS Automatic
5: Formally Verified	Proof + documentation	PASS Yes	PASS Yes	PASS Yes	PASS Proven

Mapping to EVF Survey Gaps

The EVF Survey (Paper 2) identified four critical gaps in current evidence evaluation practices. The Visibility Taxonomy explains why these gaps exist and how AIJIM's enforcement architecture addresses them:

Gap 1: Seeds Missing (91.4%)

Most studies (91.4%) do not publish the exact data used for analysis. This locks them into the "Opaque" or "Partial" levels. AIJIM enforces Level 4 through I1 (Traceability): all claims must include complete seed data with provenance.

Gap 2: Confidence Intervals Missing (97.1%)

Only 2.9% of studies quantify uncertainty with confidence intervals. Most remain at "Opaque" or "Transparent" (humans can read the explanation, but no bounds). AIJIM enforces Level 4 through I3 (Reportability): all claims must include quantified confidence and error bounds.

Gap 3: Artifacts Incomplete (72.9%)

27.1% of studies document their full methodology and artifacts; 72.9% do not. This limits most to "Partial" or "Transparent" (explanation exists, but reproducibility is not guaranteed). AIJIM enforces Level 4 through I3 (Reportability): all artifacts must be complete and verifiable.

Gap 4: Judge Configuration Undocumented (100%)

None of the 140 studies fully documented model/judge selection, parameters, or sensitivity analysis. This prevents any level of reproducibility. AIJIM enforces Level 4 through I4 (Configurability): judge selection and configuration must be documented with justification.

AIJIM's Vision: Move all evidence claims from Levels 1–2 (Opaque/Partial, current industry baseline) to Level 4 (Verifiable) by enforcing invariants I1–I5 at runtime. This dramatically improves transparency, reproducibility, and public trust.

Practical Implications for Newsrooms

Understanding the Visibility Taxonomy helps newsrooms make informed choices about evidence practices:

Opaque/Partial Systems (Levels 1–2): Risk: readers cannot verify claims. Benefit: faster publishing. Best for: preliminary reporting, breaking news where urgency outweighs verification needs.
Transparent Systems (Level 3): Humans can audit logic, but reproducibility is limited. Good for: published articles where explanation is important but exact reproducibility is not critical.
Verifiable Systems (Level 4 / AIJIM): Full reproducibility; external parties can independently verify. Best for: investigative journalism, regulatory compliance, high-stakes claims, academic publication, digital preservation.

The Visibility Taxonomy is not a judgment of "good" vs "bad"—it is a framework for transparency. Different contexts may call for different levels. AIJIM's innovation is making Level 4 (Verifiable) achievable in production newsroom workflows.