IDSLab: What Does Another IDS Stage Actually Add?

TL;DR: the tool is at hashtagrr.github.io/IDSLab. The rest of this is how it came to exist.

I was reading research papers on machine learning and deep learning based intrusion detection systems when I started noticing a pattern.

Several proposed architectures combined existing classifiers in different arrangements. A binary classifier at the beginning, another classifier to check selected flows, perhaps an anomaly detector towards the end. The resulting system would then be evaluated against other approaches, often with an improvement in accuracy, F1-score or detection rate.

There are good reasons to explore these combinations. Different classifiers may make different mistakes, and a second stage might be able to correct some of them.

But I started wondering how many IDS architectures we could build just by rearranging the algorithms we already have.

Take Random Forest, XGBoost, SVM, CNNs and a handful of anomaly detectors. We could put them in different orders, add or remove stages, change which flows get forwarded and try several thresholds. Even with a small selection of models, there would be plenty of configurations to test.

Some would perform better than others. That much seemed likely. I was more interested in why.

If a three-stage IDS improves detection, what did the second stage contribute? Can a later classifier recover attacks that an earlier classifier missed? Would a different threshold on the first classifier achieve the same result? How much extra processing did those stages introduce?

I don't think an improved metric tells us all that. A new arrangement of models may be worth studying, but I wanted to investigate the behaviour of the arrangement itself.

Then came a more practical question. How much of a researcher's time goes into writing the pipelines needed to test each new design, before they can start answering any of those questions?

That train of thought eventually became IDSLab.

The questions behind the stages

Consider a binary classifier that labels network flows as malicious or benign. Now imagine adding another classifier to examine some of its decisions.

Whether that second classifier can help depends on more than the algorithm we choose. It depends on which flows can reach it.

If the first classifier sends benign predictions straight to the final output, later stages cannot recover attacks hidden among those predictions. They never see them. A later model could still be useful for filtering false alarms among flows classified as attacks, but it would be answering a different question.

Change the routing, and the purpose of the second stage changes too. We could forward uncertain flows, let another model reconsider the decision, or use a later anomaly detector to check a particular subset of traffic.

So it becomes interesting to look at the cascade one stage at a time. Which errors can a stage realistically correct? How often does it receive traffic? What happens if we remove it? Is another stage worth the additional computation?

Those were the kinds of experiments I wanted to make easier to run.

From the questions to IDSLab

I built IDSLab as an open-source framework for constructing and evaluating staged network intrusion detection systems.

The researcher describes an architecture as a graph: the stages, the models at each stage, their decision rules, and the routes that carry flows between them. IDSLab takes care of fitting the models, running the configured cascade and producing the evaluation outputs.

A stage can be a binary classifier, an anomaly detector or an attack-type classifier. The framework supports common Python ML libraries, PyTorch models, custom adapters and externally generated scores. Configurations are saved as YAML, so the design can be inspected, modified and rerun.

I also added a visual builder. It lets you put stages onto a canvas, connect their outputs and configure the models without writing the entire YAML file by hand.

The aim is to take care of as much of the recurring experimental setup as possible. Researchers still need to choose sensible models, routing policies and datasets. But they can spend more time exploring the decisions behind an architecture and less time implementing the same supporting code again.

Did the extra stages actually help?

This question also shaped the evaluation protocol.

Suppose a cascade outperforms a separately trained single classifier. That is a useful result to examine, but it doesn't tell us whether the improvement came from the extra stages, a different threshold or some difference in training and selection.

IDSLab therefore compares a cascade against controls built from its own fitted first-stage model, including threshold-adjusted comparisons. It also reports what changes when stages are disabled and records the path each test flow took through the graph.

A diagnostic called the frontier gap compares the cascade's detection rate with the best the first-stage model can reach at the same false positive rate on the TEST sample. Its headline value uses the upper convex hull of the first stage's ROC curve, which also allows a randomised mixture of two thresholds. The report includes a separate step value for a single-threshold comparison. This helps investigate whether the later stages offer something beyond adjusting the first classifier's operating point.

The framework also keeps fitting and selection separate from final testing. Models are fitted on training data, calibration uses its designated partition where needed, and model or threshold choices are made on validation data. The resulting configuration is frozen before it is evaluated on TEST.

I wanted these rules to be part of the tool, rather than something every new experiment had to re-implement.

One example where more stages did worse

The repository includes a synthetic experiment called r1-router-gate, with a binary classifier, a router and an anomaly detection gate. In seed 0, the results looked like this:

Validation suggested that each additional stage helped. On TEST, the first classifier alone did better than the complete cascade.

Looking at the individual flows made the result easier to understand. The router forwarded 73 actual attacks to the anomaly gate. That gate failed to detect any of them and incorrectly flagged two benign flows. Removing it actually improved TEST accuracy by 0.07 percentage points in that configuration.

The other seeds also make an interesting comparison. In seed 1, removing the router or the gate slightly improved TEST accuracy. In seed 2, selection switched both later stages off. Averaged across the three seeds, the cascade achieved 83.90% TEST accuracy against 84.11% for its first stage alone. None of the per-seed paired bootstrap intervals for the difference excluded zero.

These numbers come from a synthetic demonstration, not a claim about how cascades generally perform on real traffic. I like the example because it shows why inspecting the stages matters. The final accuracy alone would have told us much less about what happened.

Working with the results

I also wanted to make the experiments manageable without having to move between configuration files and result folders all the time.

The redesigned dashboard lists experiments alongside their next step. A new experiment can start from a template, an existing YAML file or a blank design. Setup then follows four steps: Data, Cascade, Evaluation, and Review & run. The visual builder sits within that flow, and the YAML view edits the same draft while preserving comments.

When a run starts, the Jobs page shows its progress. Afterwards, the Results page puts the cascade alongside its first-stage controls, with stage effects and per-seed paired bootstrap comparisons. The report and per-flow decisions are also available for closer inspection.

I wanted those results to lead to follow-up questions. If one architecture improves recall, what happens to false positives? If a later stage helps, how many flows actually reach it? Does the result hold across seeds?

IDSLab also reports stage-level scoring time per 1,000 flows and the fraction of traffic reaching each stage. Those are useful for comparing experimental designs, but they should not be mistaken for live network latency or deployment throughput. Real-world performance would need a separate benchmark under realistic traffic and hardware conditions.

There are limits to the selection method as well. The default approach selects stages progressively, so it may miss a combination where two stages only become useful together. The framework includes a joint-selection check for examining that possibility, and the documentation describes other limitations of the protocol.

Trying it out

The project is available at github.com/hashtagRR/IDSLab. It requires Python 3.10 or later, and Git for the direct installation method below. IDSLab isn't on PyPI yet, so it needs to be installed from GitHub.

On GNU/Linux or macOS, these commands create a virtual environment, install IDSLab and run the reference example without requiring activation:

python3 -m venv .venv
.venv/bin/python -m pip install "git+https://github.com/hashtagRR/IDSLab.git"
.venv/bin/idslab example demo
.venv/bin/idslab run demo/r1_router_gate.yaml --out demo/runs
.venv/bin/idslab dashboard --workspace demo --open

On Windows, the equivalent PowerShell commands use the virtual environment's Scripts directory:

py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install "git+https://github.com/hashtagRR/IDSLab.git"
.\.venv\Scripts\idslab.exe example demo
.\.venv\Scripts\idslab.exe run demo\r1_router_gate.yaml --out demo\runs
.\.venv\Scripts\idslab.exe dashboard --workspace demo --open

The installation guide also covers installing from the downloaded source code if you don't have Git. The project documentation explains model integration, configurations and the evaluation protocol. IDSLab is released under AGPL-3.0-or-later and is designed for offline research evaluation rather than capturing live network traffic as an IDS sensor.

A few questions worth keeping open

There are probably many more useful combinations of classifiers left to try. Some may benefit from additional stages. Others may work just as well with a single well-tuned model.

I'd like to understand both outcomes.

That was the thinking behind IDSLab. If someone wants to test a new cascade, I hope they can get to the interesting questions a little sooner: what improved, which stage made the difference, and what did that improvement cost?

And if an experiment shows that an extra classifier wasn't doing much, I'd consider that a useful finding too.

Source: GitHub
Documentation: IDSLab
Licence: AGPL-3.0-or-later