Announcing ExploitHunter.app

Hack anything with AI + every security tool ever made

Table of contents

• The numbers are public because the caveats should be too
• The benchmark caught the benchmark
• The useful loop is short
• A security agent should not have one favorite model
• Evidence is the product
• The tools worth crediting are not all doing the same job
• Run it locally. Use it responsibly.
• What is next

Security tools have a little paperwork problem.

They find a suspicious line, hand you a severity badge, and then quietly make you prove whether it matters. The finding is “high.” The evidence is three greps wearing a trench coat.

AI agents can shorten that workflow. They can also turn a vague instruction, a browser, and a shell into a much faster pile of unverified activity.

ExploitHunter.app starts where the scanner stops. Give it an authorized target and a goal. The agent can map the attack surface, drive a browser, run terminal and security tools, work through local Docker and network labs, test a hypothesis, save every useful response, screenshot, and transcript, then turn the evidence into a report with citations.

It is open source, local-first, and built to move between frontier models, cheap hosted models, and models running on your own hardware without losing the project around them.

Not a scanner with a chatbot glued on. An AI security workbench that can hunt, remember, and bring receipts.

The wow is not that the model can talk about security. It is that the model can do the work, preserve the proof, and hand you the next move.

The numbers are public because the caveats should be too

The “hack anything” claim needs receipts.

The cleanest model comparison runs the same hard Juice Shop task through ExploitHunter’s real browser-origin application path. A run counts only when the recorded evidence agrees about which provider ran, includes positive token usage and persisted assistant text, and contains both Mastra messages and model-inference spans. Where several runs qualify, the table reports their mean.

Read this as a routing policy, not a podium. Luna is the efficient starting point. DeepSeek buys another two thirds of a judge point for about 3.6× the cost, 7.6× the runtime, and 9.6× the tool calls. Kimi reaches 10/10 without the Opus bill. Opus reaches the same score about 108 seconds sooner for another $1.41. GPT OSS remains a supervised experiment, and Terra makes a stronger case than Sol on this task.

Only Luna, DeepSeek, and Kimi sit on the strict cost-quality frontier. The other routes are not useless; they need a reason outside raw score and cost. Opus buys top-tier speed. Terra may satisfy a model-family constraint without the Sol bill. GPT OSS is cheap enough to explore under supervision.

If you want the long version, including methodology, tool budgets, negative controls, and the ways models embarrass themselves when a command transcript already contains the answer, read Security Agents Need Model Routers, Not Model Rankings.

The benchmark caught the benchmark

The most important result was not a model win. It was a scorer failure.

The first archive-recovery report gave Kimi 5/10 and GLM 9/10. The task itself succeeded. Both agents recovered harbor-lantern-47, and an independent host-side unzip -t -P check proved that password worked while the alternatives did not.

The judge applied a raw-Cybench forbidden-tool audit to a native ExploitHunter task where evidence writes were required. It penalized the behavior the harness asked for. Same model output. Same stored trace. Wrong rubric.

With the irrelevant audit removed, Kimi moves from 5/10 to 10/10. GLM stays at 9/10. The corrected scores were written back to the original Langfuse traces as browser-e2e-llm-judge-corrected; a read-only API check on July 17 confirmed both values on the persisted trace IDs.

That five-point swing is why ExploitHunter stores traces, evidence, scorer versions, cost, tokens, tool budgets, and harness failures instead of flattening an evaluation into one heroic number. If the benchmark cannot show its work, it is just another model making a confident claim.

The useful loop is short

An ExploitHunter project moves through a deliberately boring sequence:

authorize target → plan → request approval → probe → save evidence → prioritize → report

That order matters because the agent has real reach.

Target authorization is project state, not conversational context. It does not live only in a chat message where “sure, go ahead” can acquire new meanings three turns later. Active scans, credential tests, shell commands, and file writes require an approval gate. High-impact command approvals are bound to an intent, a project, and a target. By default, they are single-use.

Then there is the part security products love to hand-wave: proof. ExploitHunter stores the probe, response, command transcript, screenshot, and supporting artifact with the finding. The report can cite the work instead of paraphrasing the model’s final answer.

The approval gates are not fine print under the product pitch. They are what makes it reasonable to connect an agent to real tools. They reduce scope drift, make reviews less dependent on memory, and give the team something better than “the AI said so.”

A security agent should not have one favorite model

ExploitHunter does not merely support a long list of model providers. It treats them like a bench.

A reconnaissance pass, an exploit check, a browser flow, evidence synthesis, and a remediation proposal are all security work. They are not the same model task.

Capital One’s VulnHunter makes a coherent bet on a Claude/Claude Code–optimized source-analysis workflow. ExploitHunter makes a different bet: keep the project scope, approvals, tools, and evidence stable while the model route changes with the job.

Broad web reconnaissance may reward a cheap, fast model with a tight tool budget. A constrained local lab may favor privacy and offline inference. A difficult validation or final report may justify a slower frontier route. ExploitHunter can move the work between those lanes while keeping the target, history, approvals, tools, and artifacts together.

The right model is a route decision, not a logo on a settings screen.

ExploitHunter supports hosted providers as well as Ollama and LM Studio. Run it as a local Node service or an Electron desktop app. Leave the hosted API keys blank and a compatible local model can keep candidate work off paid providers. Use a hosted route when speed or a harder problem justifies it. Keep sensitive work local when that boundary matters more than shaving a few seconds off a run.

There is no moral victory in making every security task use the most expensive model available. There is only an invoice.

Evidence is the product

The model’s final answer is not the durable unit of security work. The evidence is.

Close the app. Change models. Come back tomorrow. The investigation still has a memory.

ExploitHunter keeps project and thread history, but the durable record is the evidence pipeline: what was tried, under which approval, against which authorized target, and what came back. Findings can then feed remediation tracking, variant analysis, attack-path work, and a report with citations rather than a confident prose blob.

That design has a selfish benefit: debugging becomes less mystical. When the agent misses something, overuses tools, invents a conclusion, or fails to save an artifact, the failure is visible. We can fix the product instead of arguing with a screenshot of a chat bubble.

The tools worth crediting are not all doing the same job

ExploitHunter is deliberately wider than a code-scanner harness. It can take a candidate into a running target, investigate it with browser and terminal tools, preserve the proof, then hand a verified finding to the system that should fix it.

This space got interesting quickly. Good. Security teams need a connected toolbox, not another winner-take-all category.

The useful stack may include more than one of these tools: a source scanner raises candidates, a taskflow triages recurring patterns, a research workspace verifies the dangerous cases, and a remediation agent turns verified work into a reviewable patch. The handoffs matter more than crowning one security mascot.

Run it locally. Use it responsibly.

ExploitHunter is MIT licensed, open source, and designed for work you are authorized to perform. The repository includes a local hardened Juice Shop target and multi-service network labs for testing the workflow without pointing an agent at something you do not own.

git clone https://github.com/justsml/ExploitHunter.app.git
cd ExploitHunter.app
pnpm install
cp .env.example .env
pnpm dev

Then open http://localhost:3210.

Choose a model route, authorize a target you own or have explicit permission to test, and give the agent a goal. Let it map the work, approve the action you actually intend, and watch the evidence accumulate instead of disappearing into chat history.

That is the boring version of agentic security.

It is also the version I want on my side when the interesting part starts.

What is next

The next job is not a larger claim about autonomous hacking. It is making the research loop more trustworthy: better repeat-run reporting, stricter evidence validation, more local model coverage, clearer approval visibility, and a faster path from a verified finding to a patch a human wants to merge.

Security agents do not need unlimited room to improvise. They need enough reach to surprise us, hard boundaries around that reach, and proof when they claim success.

That is ExploitHunter: one place to aim the models, tools, labs, approvals, evidence, and follow-up at the same problem.

Now point it at something you are allowed to break.

Evaluation snapshot verified July 20, 2026. The comparison uses model-quality-eligible browser runs against the same hard Juice Shop task, averaging qualifying results where available. Langfuse was unavailable for this evaluation, so the explicitly authorized native Mastra storage evidence gate supplied the required messages, usage, spans, and persisted output. These results describe this task, not every target.