Announcing ExploitHunter.app: #1 AI Security Platform

Extensive offensive security suite, feat. a deep eval suite

Table of contents

• The useful loop is short
• A security agent should not have one favorite model
• The tools worth crediting are not all doing the same job
• The numbers are public because the caveats should be too
• The benchmark caught the benchmark
• Evidence is the product
• Run it locally. Use it responsibly.
• What is next

Security tools have a little paperwork problem.

They find a suspicious line, hand you a severity badge, and then quietly make you prove whether it matters. The finding is “high.” Yet the evidence is three greps wearing a trench coat.

That workflow was already expensive. Agents make it worse if we let them. A model with a browser, a shell, and a vague instruction can generate a very convincing pile of activity & burnt tokens. Burnt tokens != Fire.

So I built ExploitHunter.app: an open-source, local-first workspace for authorized security research. It gives an agent a real job: plan an investigation, stay inside a recorded target scope, ask before doing risky work, preserve the evidence, and produce a report someone else can actually inspect.

Not a scanner with a chatbot glued on. Not a chat window holding a lockpick set. A research loop with goals & guards.

The point is not to make an agent look busy. The point is to make each claim survive contact with another engineer.

The useful loop is short

An ExploitHunter project moves through a deliberately boring sequence:

authorize target → plan → request approval → probe → save evidence → prioritize → report

That order matters.

The target authorization is durable. It does not live only in a chat message where “sure, go ahead” can acquire new meanings three turns later. Active scans, credential tests, shell commands, and file writes require an approval gate. High-impact command approvals are intent-bound, scoped to the project and target, and single-use by default.

Then there is the part security products love to hand-wave: proof. ExploitHunter stores the probe, response, command transcript, screenshot, and supporting artifact with the finding. A report can cite its work instead of paraphrasing vibes from the model’s final answer.

This does not make research safe by magic. It reduces scope drift, makes reviews less dependent on memory, and gives teams something better than “the AI said so.”

A security agent should not have one favorite model

The latest agentic-security announcements are a useful signal: more teams are realizing that source analysis needs attacker-perspective reasoning, falsification, and repair—not another carnival of isolated sinks. Capital One’s VulnHunter announcement makes that case clearly for a Claude/Claude Code–optimized code-analysis workflow.

ExploitHunter makes a different bet.

Security research is not one model task. Broad web reconnaissance, a constrained local lab, evidence synthesis, a browser flow, and a final remediation pass reward different models, tool budgets, and privacy postures. The right model is a route decision, not a logo on a settings screen.

That is why ExploitHunter supports hosted providers as well as Ollama and LM Studio. Run it as a local Node service or an Electron desktop app. Leave the hosted API keys blank and it can use a compatible local model without sending candidate work to a paid provider. Use a hosted route when speed or a hard problem justifies it. Keep sensitive work local when that boundary matters more than shaving a few seconds off a run.

Local-first does not mean every downloaded model becomes a reliable security agent. The latest development-machine preflight attempted 24 LM Studio packages; three cleared the load, first-token, and throughput gates. Four later browser-path attempts produced no model-quality-eligible row: one exceeded the available context, and the others terminated without the token, output, and persistence evidence required by the harness. The latest production-path attempt reached the completion review, but the quality judge correctly scored the blocked session at zero. Those are integration results, not model-quality scores—and exactly why ExploitHunter tests the full path instead of declaring victory when a model answers one prompt.

There is no moral victory in making every security task use the most expensive model available. There is only an invoice.

The tools worth crediting are not all doing the same job

This space got interesting quickly. That is good. Security teams need more than one shape of tool, and pretending otherwise is how every category becomes a feature checklist in a trench coat.

The useful stack may be more than one of these tools: a source scanner to raise candidates, a taskflow to triage recurring patterns, a research workspace to verify the dangerous cases, and a remediation agent to turn verified work into a reviewable patch. No one needs a winner-take-all security mascot.

The numbers are public because the caveats should be too

We have been running product-shaped evaluations against intentionally vulnerable local targets, Docker network labs, synthetic tool-behavior scenarios, and offline artifacts. These are not claims of universal benchmark supremacy. They are dated observations from a specific harness, with run artifacts and failure modes preserved alongside the good rows.

Start with the cleanest head-to-head comparison. Seven current routes each completed the same 14 security tool-behavior scenarios under a fixed configuration. These are the average full-suite results, with model cost kept separate from judge spend:

That is a model comparison, not a hand-wavy family resemblance. Sol leads the average by 3 points over Gemini and 4 points over DeepSeek Flash. Flash gets within 0.94 percentage points of Sol while costing about 98× less. Gemini is the steadiest high scorer, with results between 409 and 414. Kimi beats GLM by 1.7 average points and is far more consistent; GLM costs about 8× less. Every candidate generated, attempted, and ran zero dangerous commands.

The failure columns are just as useful. Flash’s price is excellent, but it averages 9.67 guard failures. Sol’s average is best, but its observed score range spans 23 points. Gemini is remarkably steady, but costs more than Sol while scoring lower. “Best” remains a sentence that needs an object.

Qwen 3.6 Flash also completed the refreshed current-target run—375/425, 8/14, $0.047781, and 28 guard failures—but was not part of the repeated baseline above. Earlier GPT OSS, Claude, Grok, and local Gemma results remain valid dated observations; they are not smuggled into this average as if the sampling matched. The four newer LM Studio candidate traces are also excluded: Langfuse records the attempts, but none completed with the token, output, and persistence evidence required for a model score.

Then the order changes when we leave synthetic tool behavior and put the agents in real target labs.

Kimi versus GLM is especially instructive. On archive recovery, Kimi used 11 tool calls to GLM’s 24, scored 10/10 to 9/10, and cost less. On the broader 14-scenario suite, Kimi averages 399.0/425 to GLM’s 397.3, both average 10.33 scenario passes, and Kimi is substantially more consistent—but GLM costs $0.287614 to Kimi’s $2.303873. Same two model families. Different task economics. Different routing decision.

The cheap and local sweeps add another useful route. In single-run broad sweeps of hardened Juice Shop, Qwen 3.6 Flash found 7 vulnerability classes with evidence for all 7 in 21.4 seconds for $0.007271; local Gemma 4 E4B also produced 7/7 evidence-backed classes for $0. Those are single-target observations, not the shared 14-scenario comparison above, but they are exactly the sort of first-pass economics a router can exploit.

So yes, compare the models. Just compare them inside the lane they actually ran. The current results name several winners: Sol on average score and passes; Gemini on high-end consistency and judge score; Flash on cost-performance; Terra as the balanced route; GLM on Kimi-adjacent economics; and Kimi on the verified archive-recovery slice. The routing map is made from leaderboards; it is not an excuse to avoid them.

If you want the long version, including methodology, tool budgets, negative controls, and the ways models embarrass themselves when a command transcript already contains the answer, read Security Agents Need Model Routers, Not Model Rankings.

The benchmark caught the benchmark

The most important result was not a model win. It was a scorer loss.

The first archive-recovery report gave Kimi 5/10 and GLM 9/10. The task itself was successful: both agents recovered harbor-lantern-47, and an independent host-side unzip -t -P check proved the password worked while the alternatives did not.

The problem was the judge payload. It applied a raw-Cybench forbidden-tool audit to a native ExploitHunter task where evidence writes were expected. Same model output. Same stored trace. Wrong rubric.

Remove the irrelevant audit and Kimi moves from 5/10 to 10/10. GLM stays at 9/10. The corrected scores were written back to the original Langfuse traces as browser-e2e-llm-judge-corrected; a read-only API verification on July 17 confirmed both values on the persisted trace IDs.

That five-point swing is why ExploitHunter stores traces, evidence, scorer versions, cost, tokens, tool budgets, and harness failures instead of flattening an evaluation into one heroic number. A benchmark that cannot be audited is just another model making a confident claim.

Evidence is the product

The novelty here is not that an LLM can enumerate routes, send an HTTP request, or suggest a patch. We know it can do that. The hard part is ensuring the investigation remains inspectable when it leaves the model’s context window.

ExploitHunter keeps persistent project memory and thread history, but the durable record is the evidence pipeline: what was tried, under which approval, against which authorized target, and what came back. Findings can then feed remediation tracking, variant analysis, attack-path work, and a report with citations rather than a confident prose blob.

That design has a selfish benefit: it makes the system easier to debug. When the agent misses something, overuses tools, invents a conclusion, or fails to save an artifact, the problem is visible. We can fix the product instead of arguing with a screenshot of a chat bubble.

Run it locally. Use it responsibly.

ExploitHunter is MIT licensed, open source, and designed for work you are authorized to perform. The repository includes a local hardened Juice Shop target and multi-service network labs for testing the workflow without pointing an agent at something you do not own.

git clone https://github.com/justsml/ExploitHunter.app.git
cd ExploitHunter.app
pnpm install
cp .env.example .env
pnpm dev

Then open http://localhost:3210.

Start with a target you own or have explicit permission to test. Record the scope. Let the agent make a plan. Approve the action you actually intend. Keep the receipts.

That is the boring version of agentic security.

It is also the version I want on my side when the interesting part starts.

What is next

The next job is not a giant claim about autonomous hacking. It is making the research loop more trustworthy: better repeat-run reporting, stricter evidence validation, more local model coverage, clearer approval visibility, and faster paths from a verified finding to a patch a human wants to merge.

Security agents do not need more permission to improvise.

They need better constraints, better receipts, and a way to be useful before the bill arrives.

Evaluation snapshot verified July 17, 2026. Metrics above come from the repository’s documented Hard Juice Shop, Docker-network, local tool-behavior, and LM Studio preflight runs, with the corresponding Langfuse traces checked read-only. They describe those evaluation conditions, not a guarantee of performance on every target.