Announcing ExploitHunter.app

An open-source security workbench that keeps scope, approvals, tools, and evidence in one project.

Table of contents

• The loop is deliberately boring
• Evidence is the product
• Eight models, one target
• The benchmark caught the benchmark
• Route models by task
• Where it sits next to deepsec, VulnHunter, and friends
• Run it locally. Use it responsibly.
• What is next

Security tools have a paperwork problem.

They find a suspicious line, hand you a severity badge, and quietly make you prove whether it matters. The finding is “high.” The evidence is three greps wearing a trench coat.

AI agents can shorten that workflow. They can also turn a vague instruction, a browser, and a shell into a much faster pile of unverified activity.

ExploitHunter.app starts where the scanner stops. Give it an authorized target and a goal. The agent maps reachable routes, drives a browser, runs terminal and security tools, tests a hypothesis, and saves the responses, screenshots, and transcripts that support the finding. Then it turns that evidence into a report with citations.

It is open source and local-first. You can move between hosted and local models without splitting the target, approvals, history, and evidence across separate projects.

It is not a scanner with a chatbot glued on. It is a workbench that remembers what the agent tried and what came back.

Talking about security is cheap. The useful part is running the authorized check, preserving the proof, and handing you the next move.

The loop is deliberately boring

An ExploitHunter project moves through the same sequence every time:

authorize target → plan → request approval → probe → save evidence → prioritize → report

The order matters because the agent has real reach.

Target authorization is project state, not conversational context. It does not live in a chat message where “sure, go ahead” can acquire new meanings three turns later. Active scans, credential tests, shell commands, and file writes all sit behind an approval gate. High-impact command approvals are bound to an intent, a project, and a target. By default they are single-use.

The approval gates are not fine print under the product pitch. They are what makes it reasonable to connect an agent to real tools. They limit scope drift, make reviews less dependent on memory, and give the team something better than “the AI said so.”

Evidence is the product

The model’s final answer is not the durable unit of security work. The evidence is.

Close the app. Change models. Come back tomorrow. The investigation still has a memory.

ExploitHunter keeps project and thread history, but the durable record is the evidence pipeline: what was tried, under which approval, against which authorized target, and what came back. The probe, the response, the command transcript, the screenshot, and the supporting artifact are stored with the finding. The report cites the work instead of paraphrasing the model’s closing paragraph.

Findings can then feed remediation tracking, variant analysis, attack-path work, and a report a reviewer can check line by line.

There is a selfish benefit too: debugging stops being mystical. When the agent misses something, overuses tools, invents a conclusion, or fails to save an artifact, the failure is visible in the record. We fix the product instead of arguing with a screenshot of a chat bubble.

Eight models, one target

Because every run leaves a record, comparing models stops being a vibes exercise.

I gave eight model routes the same hard Juice Shop task and ran them through ExploitHunter’s real application path. Same target. Same tools. Same evidence contract. If a run could not prove which model ran or preserve its output, it did not get a row. Benchmarks may enjoy ambiguity. Invoices rarely do.

| Model route | Judge | Cost | Runtime | Tool calls | The read | |---|---:|---:|---:|---:|---| | Kimi K3 | 10.0/10 | $0.220184 | 223.4s | 8.0 | Highest quality at the lower price of the two perfect scorers | | Claude Opus 4.8 | 10.0/10 | $1.633301 | 115.9s | 8.0 | Same quality as Kimi, nearly twice as fast, 7.4× the cost | | DeepSeek V4 Flash | 9.33/10 | $0.058695 | 395.5s | 32.0 | Best score below the two perfect routes | | GPT-5.6 Luna | 8.67/10 | $0.016304 | 52.2s | 3.3 | Best cost-and-speed result | | GPT-5.6 Terra | 8.0/10 | $0.124046 | 107.5s | 6.0 | Beats Sol on price and speed at the same score | | GPT-5.6 Sol | 8.0/10 | $0.368514 | 229.6s | 10.0 | Capable, but dominated by Terra in this slice | | Qwen 3.6 Flash | 5.5/10 | $0.085678 | 96.9s | 16.5 | Cheapest setting improved efficiency, but quality still trails | | GPT OSS 120B | 5.0/10 | $0.062529 | 36.6s | 4.3 | Fast and inexpensive, but too volatile for the default route |

Read this as a routing policy, not a podium.

Luna is the efficient starting point. DeepSeek buys another two thirds of a judge point for about 3.6× the cost, 7.6× the runtime, and 9.6× the tool calls. Kimi reaches 10/10 without the Opus bill. Opus reaches the same score about 108 seconds sooner for another $1.41. GPT OSS stays a supervised experiment, and Terra makes a stronger case than Sol on this task.

Only Luna, DeepSeek, and Kimi sit on that frontier. The other routes are not useless. They need a reason outside raw score and cost: Opus buys speed, Terra satisfies a model-family constraint without the Sol bill, and GPT OSS is cheap enough to explore under supervision.

The long version, with methodology, tool budgets, negative controls, and the ways models embarrass themselves when a command transcript already contains the answer, is in Security Agents Need Model Routers, Not Model Rankings.

The benchmark caught the benchmark

The most important result was not a model win. It was a scorer failure.

The first archive-recovery report gave Kimi 5/10 and GLM 9/10. The task itself succeeded. Both agents recovered harbor-lantern-47, and an independent host-side unzip -t -P check proved that password worked while the alternatives did not.

The judge had applied a raw-Cybench forbidden-tool audit to a native ExploitHunter task where evidence writes were required. It penalized the exact behavior the harness asked for. Same model output. Same stored trace. Wrong rubric.

With the irrelevant audit removed, Kimi moves from 5/10 to 10/10. GLM stays at 9/10. The corrected scores were written back to the original Langfuse traces as browser-e2e-llm-judge-corrected, and a read-only API check on July 17 confirmed both values on the persisted trace IDs.

That five-point swing is why ExploitHunter stores traces, evidence, scorer versions, cost, tokens, tool budgets, and harness failures instead of flattening an evaluation into one heroic number. If the benchmark cannot show its work, it is just another model making a confident claim.

Route models by task

ExploitHunter does not merely support a long list of model providers. It treats them like a bench.

A reconnaissance pass, an exploit check, a browser flow, evidence synthesis, and a remediation proposal are all security work. They are not the same model task.

Capital One’s VulnHunter makes a coherent bet on a Claude/Claude Code-optimized source-analysis workflow. ExploitHunter makes a different bet: keep the project scope, approvals, tools, and evidence stable while the model route changes with the job.

Broad web reconnaissance may reward a cheap, fast model with a tight tool budget. A constrained local lab may favor privacy and offline inference. A difficult validation or final report may justify a slower frontier route. ExploitHunter moves the work between those lanes while the target, history, approvals, tools, and artifacts stay put.

The right model is a route decision, not a logo on a settings screen.

ExploitHunter supports hosted providers as well as Ollama and LM Studio. Run it as a local Node service or an Electron desktop app. Leave the hosted API keys blank and a compatible local model keeps candidate work off paid providers. Use a hosted route when speed or a harder problem justifies it. Keep sensitive work local when that boundary matters more than shaving seconds off a run.

There is no moral victory in sending every security task to the most expensive model available. There is only an invoice.

Where it sits next to deepsec, VulnHunter, and friends

ExploitHunter is deliberately wider than a code-scanner harness. It takes a candidate into a running target, investigates it with browser and terminal tools, preserves the proof, then hands a verified finding to the system that should fix it.

Several projects now cover different parts of that workflow. Good. Security teams need tools that hand work to one another, not another winner-take-all category.

| Tool | Where it is strong | Where ExploitHunter is different | |---|---|---| | Vercel deepsec | A codebase-first harness: static candidate discovery, coding-agent investigation, revalidation, enrichment, and optional large-scale sandbox fanout. | Deepsec fits repo analysis and PR-oriented follow-up. ExploitHunter centers an authorized research project that may include a running app, browser, network lab, terminal, persistent evidence, and explicit operator approvals. | | Capital One VulnHunter | Attacker-first source analysis, structured falsification of findings, and focused code-remediation proposals. | The overlap is real: evidence and false-positive reduction should be table stakes. ExploitHunter is less tied to a coding harness or one model path, and more focused on coordinating the investigation before a code change is proposed. | | GitHub Security Lab Taskflow Agent | Declarative, MCP-enabled taskflows, especially CodeQL-alert triage and variant analysis. GitHub reports it has helped find roughly 30 real-world vulnerabilities. | The right foundation when the input is a repeatable code-scanning workflow. ExploitHunter is the workbench for exploratory, tool-using research where scope, approvals, and evidence need to survive a longer investigation. | | OpenHands Vulnerability Fixer | Turning scanner output from Trivy or other tools into prioritized fixes, tests, and pull requests. | A remediation factory. ExploitHunter is earlier in the loop: establish that the finding is real, record why, and hand a well-supported problem to the fixing system. | | Assay | Offline policy enforcement, deterministic replay, and cryptographic evidence bundles for agent tool calls. | Complementary, not a competitor. It is the kind of deny-by-default runtime control that agentic research workspaces should be able to use beneath their own approval layer. |

A useful stack probably includes more than one of these: a source scanner raises candidates, a taskflow triages recurring patterns, a research workspace verifies the dangerous cases, and a remediation agent turns verified work into a reviewable patch. The handoffs matter more than crowning a security mascot.

Run it locally. Use it responsibly.

ExploitHunter is MIT licensed, open source, and built for work you are authorized to perform. The repository ships with a local hardened Juice Shop target and multi-service network labs, so you can exercise the whole workflow without pointing an agent at something you do not own.

git clone https://github.com/justsml/ExploitHunter.app.git
cd ExploitHunter.app
pnpm install
cp .env.example .env
pnpm dev

Then open http://localhost:3210.

Choose a model route, authorize a target you own or have explicit permission to test, and give the agent a goal. Let it map the work, approve the actions you actually intend, and watch the evidence accumulate instead of disappearing into chat history.

That is the boring version of agentic security.

It is also the version I want on my side when the interesting part starts.

What is next

The next job is not a larger claim about autonomous hacking. It is making the research loop more trustworthy: better repeat-run reporting, stricter evidence validation, more local model coverage, clearer approval visibility, and a faster path from a verified finding to a patch a human wants to merge.

Security agents do not need unlimited room to improvise. They need enough reach to surprise us, hard boundaries around that reach, and proof when they claim success.

That is ExploitHunter: one place to aim the models, tools, labs, approvals, evidence, and follow-up at the same problem.

Now point it at something you are allowed to break.