When someone buys a tool to detect vulnerabilities (a scanner, a code analyzer, or an agent that simulates a real attacker) they almost always ask the same thing: how good is it at finding problems? It's the obvious question. It's like buying a car and asking about the engine's horsepower.
But anyone who has driven in winter knows the engine isn't the only thing that matters. The brakes matter, so do traction control, airbags, and the windshield wipers. All that engineering that never shows up in the commercial is what makes a car trustworthy, not just fast.
With security tools, it's exactly the same. The engine is the part that looks for vulnerabilities. And then there's everything else, what in English is called the harness, which is what turns that engine into something a company can actually trust. This post is about that invisible part, and why it's usually what decides whether the tool is useful or not.
An analogy to start
Imagine you hire a brilliant detective. They're amazing at solving cases. But they have three problems:
- Sometimes they solve the case and sometimes they get distracted; you never know which one you got today.
- When they accuse someone, they don't keep the evidence. They just ask you to take their word for it.
- They accuse so many people that nobody listens anymore when they point at the real culprit.
That detective is a detection engine with no harness. They're talented, but they're not reliable, not defensible, and in the end, not useful.
The harness is everything that turns that brilliant detective into a professional investigator: the protocol they always follow the same way, the chain of custody for the evidence, the filter that prevents baseless accusations, and the case file that shows how they reached each conclusion.
The engine answers a single question: can it find the problem? The harness answers the ones that really matter when your money and reputation are on the line: does it find it every time?, is it going to flood me with false alarms?, can it prove to me what it found?, will I know if it stops working tomorrow?
Why this weighs so much in security
There are products where an occasional failure is tolerated. A security tool isn't one of them, for three very concrete reasons.
1. Getting it wrong is expensive, in two different ways
There are two ways to fail, and both hurt.
The alarm that never went off, a false negative. A real vulnerability exists and the tool didn't see it. It's the worst scenario, because the company believed it was protected. It's like a smoke detector that doesn't go off during the fire: not only does it fail to help, it gave a false sense of security.
The alarm that went off for no reason, a false positive. The tool screams vulnerability about something that really isn't one. It seems harmless, but it isn't. Imagine a smoke detector that trips every time you make toast. What do people do? They take the battery out. And the day there's a real fire, nobody finds out.
In business terms: every false positive consumes the most expensive resource a security team has, its time and attention. A security engineer costs a lot of money per hour. If they spend half their day chasing alarms that were nothing, you're paying that salary to generate frustration. And worse: with enough false alarms, the team stops trusting the tool and starts ignoring every alert, including the real ones.
The harness is precisely the layer that attacks both errors at once: it widens the search so nothing slips through, and it adds verification filters so the alarm doesn't go off for nothing.
2. A finding with no proof is worth nothing
Let's take an example. The tool says it found a backdoor in your payments application. What does the team do with that? If it doesn't come with proof, they have to start from scratch to investigate whether it's true. And meanwhile, nobody knows whether to panic or ignore it.
Now compare it with this: I found a backdoor in your payments application, here's exactly the request that triggers it, here's the server response that confirms it, and here are the steps to reproduce it. That's actionable. The team verifies it in five minutes and fixes it.
In business terms: the difference between those two scenarios is the difference between a tool that generates work and one that generates results. The first one says: you have a problem, go figure it out. The second says: you have this exact problem, here's the evidence, fix it. That evidence isn't produced by the engine on its own; it's captured and packaged by the harness.
3. These tools don't always give the same result
This surprises a lot of people. You'd expect a tool to be like a calculator: put in the same thing, get the same thing out. But modern security tools, especially the ones that use artificial intelligence or explore adaptively like a human attacker would, don't work that way. The same tool, against the same target, can take different paths on two different days.
It's closer to a human auditor than to a calculator: two competent auditors reviewing the same company may find slightly different things, and that's normal.
In business terms: this makes the phrase we tested it once and it worked dangerously insufficient. It's like judging a football team by a single match. The harness is what imposes discipline on that variability: it bounds it, measures it, and makes it comparable, so you can state with data that the tool works consistently, not that it worked once.
The pieces of a good harness
Let's go through the concrete parts worth considering. For each one, an everyday example.
Preparing the target well before you start
Before analyzing anything, you have to know exactly what's going to be analyzed and have permission to do it.
Simple example: to a person, shop.com, the https version, and the www version are obviously the same shop. To a machine, they can look like three different things. If the tool doesn't recognize them as the same, it ends up reporting the same problem three times or, worse, reviewing one version and leaving the others untouched.
It's like a mail system that doesn't understand that 5 Main Street, 5 Main St., and Main 5 are the same address. You'd end up sending three letters or none.
Setting clear limits
Every review needs rules: how long it can run, how far it can reach, and how much it can cost.
Example: an analysis that never ends isn't an analysis, it's an endless expense. And a tool that steps outside the authorized scope, that starts touching systems it shouldn't, isn't a minor bug: it's a legal incident.
Think of it like hiring a building inspector. You give them a key that opens only the building you asked them to review, a schedule, and a budget. You don't hand them the keys to the whole block just in case.
A subtle but important point: what happens when something gets interrupted halfway through. If a review is cut short, by time, by a failure, or because someone canceled it, a good harness doesn't throw away everything it had already found up to that point. It saves the partial work.
It's the difference between a word processor that autosaves every few minutes and one that loses everything if the power goes out. The second one technically works, until the day it doesn't.
Capturing the evidence in the moment
Everything the tool sees is momentary. If it isn't saved instantly, it's lost forever.
Example: it's like a security camera. There's no point in having the best camera in the world if it isn't recording when the robbery happens. The harness is what makes sure everything gets recorded, organized, and recoverable later, not in a pile of loose notes that are impossible to read.
Not reporting the same thing five times
In practice, the tool detects the same problem from several angles. If it isn't smart about it, it shows you five problems where there's really just one.
Example: imagine three different departments in your company reporting the same unpaid invoice. If your system doesn't realize it's the same invoice, you think you have a 15,000 dollar problem when it's really 5,000, and you send three people to solve the same thing.
There's a delicate balance here: group too much and you hide real problems; group too little and you generate noise. Getting that balance right is fine harness work, and it makes a huge difference in how the tool feels to use.
Verifying before raising the alarm
This is perhaps the piece that most separates a serious tool from an alarm-generating machine. Before bothering anyone, does the tool confirm that what it found is real?
Example: it's the difference between a neighbor who calls the police every time they see a shadow and one who first looks out the window to confirm there really is someone there. The second one calls less often, but when they call, people believe them.
In business terms: this verification layer is what lets you raise the quality of the results without touching the engine. It's a design decision that pays dividends every day in the form of less wasted time.
Measuring whether it's actually improving
If you can't measure whether your tool is improving, you're working blind.
Example: to know whether a student is learning, you give them exams with known answers and see how many they get right over time. Same here: you run the tool against practice targets where we already know which vulnerabilities exist, and you measure how many it finds, how many it invents, and whether it's consistent.
And since these tools don't always give the same result, one exam isn't enough: you have to run them several times and look at the average, just like you wouldn't judge a student by a single question on a bad day.
In business terms: without this measurement, saying we improved the tool is an act of faith. With it, each new version can be defended with numbers: this version finds 12% more and is wrong 30% less. That's something a customer understands and an investor trusts.
Knowing when the harness itself fails
The system doing the watching can also fail, and you need to find out clearly.
Example of an error that's expensive: a system that, when something breaks, invents an explanation that sounds plausible but that nobody checked. It's like a car dashboard that turns on the check engine light when the real problem was a sensor. You send the car to the shop, pay for the engine inspection, and the problem was still there. A good harness prefers to say something failed and show the data rather than lie with confidence.
What the business gains by investing in the harness
Summed up in terms of results:
- Trust. Findings come with proof, so the customer fixes them instead of doubting them.
- Less noise, less cost. Fewer false alarms means your expensive people work on what matters, not chasing ghosts.
- Demonstrable improvement. You can prove with numbers that the product gets better, version after version.
- Resilience. When something goes wrong, and it always does eventually, work isn't lost and data isn't corrupted.
- Predictable costs. Every review has a bounded, known cost, with no surprises on the bill.
- Faster development. It sounds contradictory, but a good harness speeds you up: it lets you improve the engine with a safety net, because if something breaks, you find out on your own.
Common mistakes worth avoiding
- Confusing it worked in the demo with it works. A successful demo is like a single good play: it doesn't prove the team is good.
- Trusting tailor-made tests. If whoever builds the tool also writes the exam, the exam tends to be designed to pass. The real test is against reality, not against your own expectations.
- Ignoring what happens when things go wrong. Most of the value is lost in the runs that don't end as expected. Designing only for the sunny day is designing for the minority of days.
- Letting the engine decide on its own what the customer sees. Without an independent verification filter, product quality depends on the engine's mood that day.
- Not measuring. Without continuous measurement, you don't know whether you're moving forward or backward.
Harness engineering isn't glamorous. It doesn't show up in the headline or in the product screenshot. But it's the difference between a tool that sometimes finds problems and one a team can lean on for real business decisions.
Back to the car analogy: nobody buys a car for its airbags, but nobody who understands cars would buy one without them. In security, where getting it wrong is expensive and conclusions have to be defensible, the harness isn't the accessory around the product. To a large extent, the harness is the product.
If you're building or buying a tool in this space, the best advice we can give is simple: don't judge the engine alone. Ask about the brakes.
About Krill Security
At Krill Security, we build a continuous offensive security platform driven by autonomous agents, and we treat harness engineering as part of the product, not as an accessory. Every finding comes with real exploitation evidence, security posture is measured over time, and the platform is designed to be consistent, not to work well just once.
If you're evaluating an autonomous security tool, ours included, ask it the hard questions: how it verifies its findings, what happens when a run is interrupted, and how it proves it's improving version after version. Don't judge the engine alone: ask about the brakes.