VisHarness: A Lightweight Visual Agent Harness for VLMs to Reason, Act, and Think with Visual Tools

As models move from passive perception to long-horizon visual action, the harness becomes part of the reasoning system. The central design question is how to preserve the evidence that connects a visual observation to the next decision.

This framing leads directly to VisHarness: a lightweight visual agent harness whose Tool packs let VLMs reason over evidence, act through typed tools, and think across state changes without changing the contract that the evaluation sees.

A lightweight visual agent harness for VLMs to reason, act, and think with visual tools. VisHarness gives a model a stable way to observe, transform, retrieve, remember, and act while preserving the provenance of every visible result.

Tool packs are mechanism choices, not model claims. Their purpose is to make the evidence available to the model inspectable, repeatable, and comparable across a control arm.

One evaluation turn joins observation, visual-tool reasoning, environment action, and persistent thinking in a VLM loop whose evidence can be checked after the model finishes.

{
  "tool_id": "image.crop",
  "coordinates": "pixel-space",
  "deterministic": true,
  "side_effects": "none",
  "evidence": "parent + source map",
  "readiness": "runtime-verified"
}

What changes when the harness is present?

The harness does not hide the model’s decision process behind a single helper call. It records the input artifact, the tool contract, the backend/readiness state, the returned evidence, and the final answer. That makes a tool failure, a missing model weight, and a model mistake separable events.

Matched control arms use fixed item identities and explicit denominators. A higher score is interpreted together with coverage, tool usage, and evidence status rather than as a standalone optimization target.

ARC-AGI-3 turns visual reasoning into an interactive control problem: an agent receives an unfamiliar world, infers mechanics and goals from observations, and must complete levels while keeping environment actions efficient.

The official metric is Relative Human Action Efficiency (RHAE): completion is combined with action efficiency relative to first-time human references. Internal reasoning, visual inspection, and tool calls are not environment actions unless they trigger the game. The Competition-Mode scorecard is the source of record for the result and its replay links.

The scorecard, 25 official replay links, action comparison, and harness-side counters are presented together. The available efficiency proxy is 40.11 actions per level (42.8% of the published first-time-human action total). The VISTA-shaped schematic is explanatory, while the linked ARC pages are the recorded interactive replays.

ActiveVision, ZeroBench-main, and PerceptionBench probe different limits of a visual agent: active inspection, zero-shot visual reasoning, and fine-grained perception. All three are deliberately difficult—on 85- or 100-item benchmarks, even a single-digit-point change means solving several additional frontier-level problems; on a 3,000-item perception suite, two points correspond to roughly 60 additional judged-correct answers.

Source of record: model-benchmark-leaderboard.md and the release README benchmark results. Values below preserve the published scores, denominators, effort labels, and comparability boundaries.

ActiveVision · interactive visual search and comparison

ZeroBench-main · frontier zero-shot visual reasoning

PerceptionBench · 3,000-item fine-grained perception

GPT-5.5 PerceptionBench is retained as a 53-item pilot in the release records and is omitted from this full-cohort table; it does not provide a 3,000-item model comparison.

* Grok ActiveVision is aggregate evidence; the public record reports the aggregate result. † Selection condition, effort, or run protocol differs, so the positive delta is a descriptive capability advantage rather than a matched causal estimate. ‡ ZeroBench official w/o rows use published five-sample pass@1 references, while local w/ rows are single-sample runs. Blue protocol labels identify the scorer or metric, model effort, sampling, and source. Cross-effort negative arithmetic differences are not reported as deltas.

Tool usage is the mechanism view of the release: it shows which evidence operations were available, adopted, and reliable across model × benchmark cohorts. Call count is a diagnostic signal, not a quality target.

Source of record: harness-usage-leaderboard.md. The snapshot reports prediction/trace coverage separately from judge coverage.

Tool-use leaderboard

Aggregate operation mix

Manifest boundary. image.code is a library-only / historical operation in this accounting and is outside the current default 8-pack / 29-tool manifest. The aggregate table describes observed traces; it does not redefine the release manifest.

Release links and citation

VisHarness repository ↗Tool packs ↗Model results ↗Usage results ↗VISTA reference ↗

@misc{visharness2026,
  author = {Chenhao Zhang},
  title  = {VisHarness: A Lightweight Visual Agent Harness for VLMs to Reason, Act, and Think with Visual Tools},
  year   = {2026},
  url    = {https://github.com/MING-ZCH/VisHarness}
}