Evidence Investigator

Evidence Investigator: Building an Auditable Multiagent Research System

An engineering experiment in scientific evidence retrieval, agent coordination, decision safeguards and human review.

Lab: Agentic AI and AI Reliability Project: Evidence Investigator Version tested: v1.3.3 RC Status: Implemented and tested locally with a synthetic dataset Environment: AlmaLinux 10 on WSL2

The research question

Large language models can produce convincing explanations of scientific claims. A convincing explanation, however, is not the same as a verified conclusion. The model may overlook contradictory passages, misinterpret a source or report confidence that has never been calibrated against real outcomes.

Evidence Investigator explores a specific question: Can several AI agents investigate a claim together while a separate decision layer, deterministic safeguards and a human reviewer retain control over what the system is allowed to conclude?

The goal is not to automate scientific truth. The goal is to make an experimental research workflow inspectable, bounded and testable.

The hypothesis

A useful research assistant should separate evidence discovery from decision making. It should preserve the original source passages, record agent actions and prevent a language model from silently changing a decision while writing the final report.

My working hypothesis is that bounded agent collaboration combined with source verification and explicit decision checks can produce a more auditable process than asking a single model to answer a claim in one step.

That hypothesis still needs comparative evaluation. A successful demonstration alone cannot establish that the approach is more accurate.

System architecture

Evidence Investigator is implemented in Go with no external Go module dependencies. Provider integrations use native HTTP and JSON rather than an AI SDK. A lightweight web interface exposes investigations, evidence passages and case review.

The system has five agent roles.

Planner determines how to approach the claim and delegates work.

Researcher searches a bounded local document corpus and investigates candidate passages.

Critic challenges the evidence and looks for missing context or weak support.

Analyst prepares verified evidence for the decision stage.

Narrator converts the provisional decision into a readable report without permission to change its verdict.

These roles do not form a rigid sequence. They exchange typed messages through a Go event router. A Narrator can request additional research and a Critic can send the investigation back to another role. The dispatcher processes events serially so that evidence revisions and recorded actions remain ordered. This is an event driven runtime, not a distributed swarm of concurrent agents.

The router treats every model proposed action as untrusted. Go checks the requested operation, recipient, source reference and role permissions before allowing it to run. The available research tools are limited to reading and searching the configured corpus.

The accompanying technical architecture diagram documents both the logical components and the investigation lifecycle. It can be published alongside this article as a separate figure.

What happens during an investigation?

Claim intake and retrieval

A user submits a claim through the web interface or REST API. The retrieval engine searches a bounded corpus using BM25 and returns candidate passages. Each passage includes a source reference, sentence position and SHA 256 fingerprint for provenance checks.

A lexical relevance score is useful for finding text. It does not establish whether a passage logically supports the claim.

Agent collaboration

The Planner initiates research. Other agents can search for passages, inspect sources, challenge findings and delegate followup work. Every tool action is validated before execution. The runtime also tracks the current evidence revision.

When new verified evidence is added, the revision changes. A previous decision cannot simply be reused for the new evidence set. The Analyst must prepare a fresh evaluation before another report can be issued.

Decision and policy checks

The Analyst submits structured evidence through a separate decision adapter. The application then applies deterministic policy and consistency checks. The Narrator receives the resulting provisional decision and must preserve it while explaining the evidence.

For the local integration described here, the language model calls used Mistral API and the decision request used Inception Mercury Decide through OpenRouter. The intended TypeSafe JEV integration has not yet been verified against the real TypeSafe service. The system's decision related labels should not be mistaken for proof that TypeSafe JEV was the provider in this test.

Case storage and human review

The report is stored as an investigation case with provenance and review state. A separate reviewer credential is required to record an authorized review disposition. A provisional SUPPORTED assessment does not automatically become an approved scientific finding.

Experimental setup

The first successful live integration test ran on AlmaLinux 10 under Windows Subsystem for Linux 2. The application used the bundled synthetic demo corpus with five documents.

The test claim was:

In a simulated greenhouse, lower temperatures reduced tomato fruit yield.

The corpus included a fictional greenhouse experiment describing tomato plants grown at 12°C and 22°C. This controlled synthetic example was selected to exercise the complete software workflow. It was not intended to test real scientific generalization.

The experiment used the Autonomous multiagent + decision model execution mode. The application retrieved evidence, coordinated agents, invoked the configured decision provider and generated a report requiring human review.

Reproduction notes

The source requires Go 1.23 or newer. Within the project checkout, go test ./... and go vet ./... run basic development checks. The local demo uses the bundled synthetic corpus. Live agent execution requires separately configured LLM and decision provider credentials.

The observed run used the autonomous investigation mode with the stated claim. Provider availability, model output and request timing may vary between runs. Repeating the same input should not be assumed to reproduce an identical agent trace or verdict.

Observed results

The investigation completed with a provisional SUPPORTED assessment. The report cited synthetic passages indicating lower tomato fruit yield at the lower temperature. The application recorded the result without treating it as an independently verified scientific fact.

Reported execution time: 15.547 seconds LLM calls: 15 Decision model calls: 1 Audited agent actions: 17 Passages returned in the final report: 4 Termination reason: budget_or_convergence Review state at report generation: Pending human review

The decision provider returned a confidence value of 1.000. This value is not an independently calibrated probability of correctness. It must not be presented as a 100% accuracy result.

One retrieved passage concerned urban tree canopy rather than the greenhouse claim. That is a useful reminder that citation integrity and evidence relevance are separate requirements. The system found relevant evidence but its final passage selection still needs improvement.

This was one successful integration run on synthetic data. It is not a benchmark result for the broader scientific claim verification task.

Verification performed

The development checks for v1.3.3 passed Go unit and integration tests, the race detector, go vet and an offline build. Regression tests also covered invalid agent actions and queue exhaustion during agent coordination.

An earlier synthetic retrieval sanity test found the expected document in all five demo cases at a retrieval depth of five. Because the examples were constructed for the demo, this result must not be interpreted as a real world retrieval accuracy estimate.

The runtime regression tests use mock language model and decision responses. Separately, the local WSL integration exercised real external providers and produced the report described above. These are different forms of verification and should not be conflated.

A full SciFact evaluation, measured verdict accuracy, confidence calibration, broader provider compatibility testing and Docker or VPS deployment verification are still outstanding.

Engineering failures that shaped the design

A valid JSON response can still request an invalid action

In an earlier build, the Critic agent issued an invalid search request. The output could be parsed but the requested tool operation did not satisfy the runtime rules. Initially that error stopped the entire investigation.

The fix was to validate the proposed action separately from its JSON structure. The runtime now rejects the unsafe or malformed action, records the event and allows a bounded correction attempt. If correction fails, it can continue using already verified evidence when permitted by policy.

Agent conversations do not always converge naturally

Another test stopped after nine agent turns and two evidence revisions without ever reaching a decision call. The router had no pending messages and its deduplication logic suppressed a needed followup to the Analyst.

Version 1.3.3 introduced bounded finalization. When ordinary message routing is exhausted, the runtime can request a guarded Analyst evaluation and Narrator report without bypassing evidence checks or the decision policy. A regression test reproduced the prior failure and passed with the fix.

Provider behavior is part of system reliability

During local integration, a model rejected a structured output request and another encountered upstream HTTP 429 rate limiting despite unused account quota. These failures exposed the difference between having a compatible API shape and having a reliable provider at runtime.

Retries, compatibility checks, error visibility and budgets matter as much as prompting when several agent turns depend on external services.

Limitations

The current evidence corpus used in the demonstration is synthetic. The reported outcome is therefore a functional integration result rather than a scientific validation result.

The retrieval engine uses lexical ranking, which can return irrelevant but overlapping passages. Source hashes protect passage provenance but cannot prove that the source is scientifically sound. Structured output validation can enforce the shape of an answer without ensuring that its meaning is correct.

The runtime is bounded but a single investigation can still generate many LLM calls. Its cost, latency and convergence behavior need systematic measurement across more claims and providers.

The current local result uses Mercury Decide through OpenRouter. A production claim about TypeSafe JEV would require a separate verified integration and corresponding evaluation. The project has not been verified as deployed to the Runtime Garden VPS.

All conclusions remain provisional until reviewed by an authorized human.

Next experiments

The next phase focuses on measurement rather than adding more agent roles.

SciFact evaluation. Run retrieval and verdict benchmarks against a public research dataset with documented splits and repeatable settings.

Ablation studies. Compare a direct LLM assessment with a separate decision model and with the full agent runtime. Measure answer quality as well as the additional token usage and latency.

Evidence quality. Improve relevance filtering, citation selection and handling of conflicting evidence.

Provider validation. Verify the TypeSafe JEV integration explicitly and improve diagnostics for failed API calls.

Human review and observability. Expose agent traces, evidence revisions, policy checks and actionable review errors in the interface.

What this experiment tells us so far

Evidence Investigator now has a locally demonstrated path from claim intake to retrieved evidence, agent collaboration, a provisional decision and a report waiting for human review. The most useful lessons came from the failures between those stages.

The successful run establishes that the components can work together in one controlled setting. It does not yet establish that multiple agents outperform simpler approaches or that a confidence score should be trusted.

That distinction is the point of publishing this work as a Lab experiment. The goal is to build a system, test its assumptions, document the limits and share what the evidence actually supports.

Join the discussion

How would you evaluate whether agent collaboration improves scientific claim assessment over a simpler baseline? What would convince you that a separate decision model adds measurable value?

I'd be interested in feedback on evaluation methodology, evidence quality and bounded agent coordination.

Evidence Investigator is an experimental Runtime Garden project.

Where ideas grow into systems.

Build. Experiment. Share. Grow.