New Logs tab​

The Session Details view has a new Logs tab. Studio reads the logs of the session and shows them in the browser. You no longer download the artifacts archive and unpack it on your machine.

A console-style viewer. The selected file opens in a log console with line numbers, timestamps, and a color-coded level badge (Verbose, Debug, Information, Warning, Error, Fatal). A stack trace stays attached to the log entry that produced it. It does not break into separate unparsed lines.

Session Details: new navigation​

The biggest visual change is the navigation of the Session Details view. We replaced the old sidebar with a resizable tree of your scenarios and their steps.

• When you select a scenario or a step in the tree, the tabs show the data of this selection only. This applies to the Summary, Status codes, Charts, Failures, and Custom Metrics tabs.
• You can collapse the tree, or drag its border to change the width.

The Logs tab uses the same control for its list of log files. The navigation is therefore consistent across the whole view.

We also corrected two problems of the old layout. The page no longer jumps when you switch tabs, and the navbar stays visible.

New Data Transfer chart​

The Charts tab has a new Data Transfer chart. It shows the network throughput over time. The chart gives the amount of data per second for the successful (Ok) requests, the failed (Fail) requests, and all (Total) requests.

The data transfer metrics are also part of the Summary table now.

A metric table under every chart​

The Throughput, Latency, and Data Transfer charts each have a table below the chart. One row is one scenario or one step. Each row has a color mark that matches the line on the chart, so you can map the numbers to the lines. You do not need the legend.

Every metric has two values:

• final - the value for the whole run. For an active session, it is the most recent data point.
• peak - the highest value of the run.

Tooltips everywhere​

An info icon near each chart title explains what the chart shows. Each column header of a metric table explains its metric. The .NET process metrics have the same tooltips: CPU usage, memory working set, GC heap size, GC LOH size, time in GC, thread pool queue length, thread count, and DNS lookups.

Better chart scale​

The charts now calculate the maximum of the Y axis from the peak value of the series. Before this change, a custom metric with large values made the main lines flat at the bottom of the chart.

📦 Docker image & Helm chart​

NBomber Studio is available as a Docker image and a Helm chart:

• Docker Hub: nbomberdocker/nbomber-studio
• Helm chart: nbomber-studio (source)

• We reworked how NBomber calculates the scenario-level stats.
• We added the calculation of the data transfer throughput.
• We improved HTML report with a new data transfer chart.
• We made the console, TXT, and Markdown reports consistent.
• We made NBomber Cluster much easier to troubleshoot.
• We improved the logging.

HTML report improvements​

We reworked the layout, the charts, and the tables to match the reporting of NBomber Studio.

The main change is the navigation. The session page now has a resizable tree of your scenarios and their steps on the left side. You select a scenario or step. Then the summary table, the status codes, and the charts show the data of this selection only. You can collapse the tree, or drag its border to change the width.

The other changes:

• A new Data Transfer chart. It shows the network throughput over time, for the total, the ok, and the fail requests. The chart selects the byte unit (KB, MB) from the peak value.

• A metric table under every chart. The Throughput, Latency, and Data Transfer charts each have a table below the chart. One row is one scenario or one step, with a color mark that matches the line on the chart. Each metric has a final value for the whole run and a peak value for the highest point of the run.

• Tooltips. An info icon near each chart title explains the chart. Each column header of a metric table explains its metric.
• A better chart scale. The charts calculate the maximum of the Y axis from the peak value. A custom metric with large values no longer flattens the main lines.

We also corrected several defects. The NBomber icon is correct now. The tooltips no longer stay on the screen after you switch tabs. The sticky table headers keep their position. The report also keeps the scroll position when you move between scenarios.

Improved reports formatting​

The console, TXT, and Markdown reports now use one consistent shape for the counters. The request counts and the RPS are on separate rows. Each row shows total, ok, and fail side by side:

|           scenario name |test_scenario                                                            |
|                requests |total = 6000, ok = 5980, fail = 20                                       |
|           RPS (req/sec) |total = 200, ok = 199.3, fail = 0.7                                      |
|            latency (ms) |min = 1.2, mean = 4.8, max = 120.5, StdDev = 6.1                         |
| latency percentile (ms) |p50 = 4.1, p75 = 5.9, p95 = 12.4, p99 = 31.7                             |
|      data transfer (KB) |min = 1.2, mean = 4.5, max = 9.1, throughput = 850.3 KB/s, total = 42 MB |

These changes are small, but they remove the guesswork when you scan a report.

Reworked ScenarioStats​

This is the most important change in this release. It changes the numbers that you see for a scenario that contains steps.

Before this release, NBomber built the scenario-level stats (ScenarioStats.Ok and ScenarioStats.Fail, printed as the global information row) from the scenario iteration measurements only. NBomber did not aggregate the step data. If your scenario had steps, all the work inside these steps was invisible at the scenario level:

• The scenario RPS counted the iterations, not the step requests.
• The scenario latency was the duration of the whole scenario iteration.

From version 6.6.0, when a scenario contains steps, the scenario-level stats aggregate all step measurements:

• scnStats.Ok.Request.RPS is the sum of the RPS of all steps in the scenario.
• scnStats.Ok.Request.Count and scnStats.Fail.Request.Count are the sums of the ok and the fail step requests.
• NBomber calculates the scenario latency and the percentiles from the step latencies, not from the scenario iteration duration.
• NBomber merges the scenario status codes from the step status codes. When the scenario has steps, NBomber ignores the status code of the scenario body.
• The scenario data transfer is the sum of the data transfer of all steps.

This basic example shows the difference. The scenario has 2 steps. The load simulation represents a constant rate: 10 requests per second, for a duration of 10 seconds.

var scenario = Scenario.Create("test_scenario", async ctx =>
{
    await Step.Run("step_1", ctx, async () =>
    {
        await Task.Delay(1000);
        return Response.Ok();     
    });
    
    await Step.Run("step_2", ctx, async () =>
    {
        await Task.Delay(1000);
        return Response.Ok();     
    });

    return Response.Ok();
})
.WithLoadSimulations(
    Simulation.Inject(10, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(10))
);

The previous NBomber versions print this result. The scenario reports 10 RPS, and the latency is the duration of one scenario iteration: p99 = 2030.59.

┌─────────────────────────┬─────────────────────────────────────────────────────────────┐
│      scenario and steps │ ok stats                                                    │
├─────────────────────────┼─────────────────────────────────────────────────────────────┤
│           scenario name │ test_scenario                                               │
│                requests │ total = 100, ok = 100, fail = 0                             │
│           RPS (req/sec) │ total = 10/s, ok = 10/s, fail = 0/s                         │
│            latency (ms) │ min = 1999.67, mean = 2014.44, max = 2029.89, StdDev = 8.76 │
│ latency percentile (ms) │ p50 = 2015.23, p75 = 2019.33, p95 = 2029.57, p99 = 2030.59  │
│                         │                                                             │
│               step name │ step_1                                                      │
│                requests │ total = 100, ok = 100, fail = 0                             │
│           RPS (req/sec) │ total = 10/s, ok = 10/s, fail = 0/s                         │
│            latency (ms) │ min = 999.2, mean = 1006.94, max = 1015.79, StdDev = 6.1    │
│ latency percentile (ms) │ p50 = 1004.54, p75 = 1014.27, p95 = 1015.3, p99 = 1015.81   │
│                         │                                                             │
│               step name │ step_2                                                      │
│                requests │ total = 100, ok = 100, fail = 0                             │
│           RPS (req/sec) │ total = 10/s, ok = 10/s, fail = 0/s                         │
│            latency (ms) │ min = 999.39, mean = 1007.37, max = 1015.7, StdDev = 6.4    │
│ latency percentile (ms) │ p50 = 1004.03, p75 = 1014.27, p95 = 1015.3, p99 = 1015.81   │
└─────────────────────────┴─────────────────────────────────────────────────────────────┘

NBomber 6.6.0 prints a different result for the scenario stats.

┌─────────────────────────┬────────────────────────────────────────────────────────────┐
│      scenario and steps │ ok stats                                                   │
├─────────────────────────┼────────────────────────────────────────────────────────────┤
│           scenario name │ test_scenario                                              │
│                requests │ total = 200, ok = 200, fail = 0                            │
│           RPS (req/sec) │ total = 20/s, ok = 20/s, fail = 0/s                        │
│            latency (ms) │ min = 999.3, mean = 1006.8, max = 1016.35, StdDev = 6.61   │
│ latency percentile (ms) │ p50 = 1003.01, p75 = 1014.27, p95 = 1015.3, p99 = 1016.32  │
│                         │                                                            │
│               step name │ step_1                                                     │
│                requests │ total = 100, ok = 100, fail = 0                            │
│           RPS (req/sec) │ total = 10/s, ok = 10/s, fail = 0/s                        │
│            latency (ms) │ min = 999.33, mean = 1005.97, max = 1016.15, StdDev = 6.44 │
│ latency percentile (ms) │ p50 = 1001.98, p75 = 1014.27, p95 = 1015.3, p99 = 1015.3   │
│                         │                                                            │
│               step name │ step_2                                                     │
│                requests │ total = 100, ok = 100, fail = 0                            │
│           RPS (req/sec) │ total = 10/s, ok = 10/s, fail = 0/s                        │
│            latency (ms) │ min = 999.3, mean = 1007.62, max = 1016.35, StdDev = 6.67  │
│ latency percentile (ms) │ p50 = 1004.03, p75 = 1014.27, p95 = 1015.3, p99 = 1016.32  │
└─────────────────────────┴────────────────────────────────────────────────────────────┘

The scenario now reports 20 RPS (the aggregated RPS of the 2 steps) instead of 10 RPS. This data is a true summary of the load that you produce, and the step stats add up to it.

If you want the previous behavior, you can enable it for each scenario separately:

Scenario.Create("test_scenario", async ctx => ...)
        .WithStatsBasedOnIterations(true);

You can also set this option in the JSON config:

{
    "GlobalSettings": {

        "ScenariosSettings": [
            {
                "ScenarioName": "test_scenario",

                "StatsBasedOnIterations": true
            }
        ]

    }
}

Read the documentation of this option for more information.

Data transfer throughput​

We added BytesPerSecond to DataTransferStats (#992). Until now, NBomber reported only the total transferred size. To get the network throughput, you had to divide this size by the duration yourself.

This property is available on the DataTransfer stats:

var throughput = stats.ScenarioStats[0].Ok.DataTransfer.BytesPerSecond;

NBomber also prints this value in the live status table and in the final reports:

data transfer (KB): min = 1.2, mean = 4.5, max = 9.1, throughput = 850.3 KB/s, total = 42 MB

The CSV report has two new columns: data_transfer_bytes_per_sec and fail_data_transfer_bytes_per_sec.

Cluster improvements​

• Agent IDs are now numbers. The Coordinator assigns a short numeric ID to each Agent, instead of a GUID. The log files, the folders, and the cluster reports are now much easier to read and to correlate.
• Fixed the early stop in cluster mode. When a session stopped before the planned duration, NBomber calculated the final RPS and throughput of an Agent against the planned duration. As a result, these values were too low. Now NBomber calculates them against the duration that the Agent executed.
• The Coordinator can now gather every Agent's log files into one place. Logs land in reportFolder/agents_logs/agentId

Improved logging​

We reworked logging around a new internal NBomberLogger:

• Before the session starts, NBomber holds the log events in memory (configuration parsing, cluster connection, validation). NBomber then writes these events to the session log file. Before this release, these events went to a temporary session folder that you had to find separately.
• The temporary session IDs and the temporary log folders are gone.
• Each session writes one log file with a meaningful name: nbomber-log-<timestamp>.txt for a single node, coordinator-log-<timestamp>.txt for the Coordinator, and agent-<id>-log-<timestamp>.txt for an Agent. NBomber no longer rolls the log file every hour in one session.

If you supply your own Serilog configuration, nothing changes for you. NBomber only enriches your logger with the session properties.

Bug fixes​

• Fixed: the data transfer was lost for the failed scenario iterations, and it leaked into the next iteration. An iteration can end with an error, a timeout, or a restart. NBomber now attributes the measured bytes to that iteration.
• Pinned Serilog.Sinks.SpectreConsole to 0.4.0. A transitive update of this package broke the console rendering.

What's next​

• Follow our Roadmap to see what we plan for the next release.

Fixes and improvements​

Air-gapped / offline deployment support​

NBomber Studio's UI used to pull a few assets from public CDNs at runtime: the theme stylesheet, jQuery, jQuery Slimscroll, and the Monaco Editor (used for script editing). That made it impossible to run Studio in fully air-gapped or restricted-network environments.

All of these dependencies are now bundled locally into the build:

• theme.css is vendored directly into the UI assets instead of being linked from jsDelivr.
• jQuery / Slimscroll <script> tags removed from index.html.
• Monaco Editor and its language workers (JSON, CSS, HTML, TypeScript) are now bundled via Vite (monaco-editor + local worker imports) instead of being loaded from a CDN path.

Impact: Studio now works in networks with no outbound internet access.

Active Sessions table not clearing​

Fixed a bug where the "Active Sessions" table would keep showing stale rows after every running session finished — the table never cleared once all sessions left the Active state. The polling logic in ActiveSessionsTable.vue and the backend active/lastupdate endpoint were adjusted so an empty result now correctly refreshes the table instead of being ignored.

ScriptValidator hang on large build output​

The C# script validator could hang indefinitely when a dotnet build produced a large amount of stdout/stderr output — synchronous ReadToEnd() calls on the process streams could deadlock once the OS pipe buffer filled up. Switched to reading stdout/stderr asynchronously (ReadToEndAsync) in parallel with waiting for process exit, eliminating the deadlock.

📦 Docker image & Helm chart​

NBomber Studio is available as a Docker image and a Helm chart for easy deployment:

• Docker Hub: nbomberdocker/nbomber-studio
• Helm chart: nbomber-studio (source)

What's next​

• Follow our Roadmap to see what's coming in the next release.

Load Test Schedules​

Load Test Schedules let you configure a load test to run automatically at specific times, so you no longer need to kick it off manually.

Container Spec Settings​

We've also extended the Container Spec settings for running load tests in Kubernetes. The following standard options are now available: volumeMounts, volumes, envFrom, and tolerations. Here's an example:

containers:
  resources:
    requests:
      cpu: "2"
      memory: "256Mi"
    
    limits:
      cpu: "4"
      memory: "1Gi"
  
  volumeMounts:
    - name: secret-volume
      mountPath: "/etc/secrets"
      readOnly: true

  envFrom:
    - prefix: APP_
      configMapRef:
        name: app-config
        optional: true
    - secretRef:
        name: app-secret
        optional: true

nodeSelector:
  nodepool: loadtest
  disktype: ssd

tolerations:
  - key: dedicated
    operator: Equal
    value: loadtest
    effect: NoSchedule

volumes:
  - name: secret-volume
    secret:
      secretName: app-secret

Auth Improvements​

We added more flexibility to auth configuration. Starting from this version, you can enable or disable a specific auth provider via the Enabled: bool property in config.json.

"Auth": {
    "Enabled": true,    

    "JwtSecret": "YOUR_SECRET_KEY", 

    "StaticUserAuth": {
      "Enabled": true,
      "Users": [
        {
          "Email": "user_email@test",
          "Hash": "$2a$11$7MTzvj3mTFWVhOryGcDARe9ir0lFc/RPUThoCYBtJWKNff29TPWFC", // password is 'password'
          "UserName": "user"
        }
      ]
    },

    "OpenIDConnect": {
      "Enabled": true,
      "Authority": "https://accounts.google.com",
      "ClientId": "YOUR_CLIENT_ID",
      "ClientSecret": "YOUR_CLIENT_SECRET",
      "CallbackPath": "/signin-oidc"
    }
  }

UI/UX Improvements​

Auto refresh toggle​

Auto refresh toogle is now available on all views, giving you full control over real-time updates.

New LoadTest tabs​

We added a few new tabs to the LoadTest view: History, Trends, and Schedule. They provide focused navigation and features specific to the selected load test.

License info tooltip​

We added a handy license info tooltip that shows your license details, such as company name and expiration date.

What's next​

• F# Script support
• Follow our Roadmap to see what's coming in the next release.

Load Tests in Kubernetes give you a control plane for managing, deploying, and running load tests. With this feature, you can schedule and execute load tests in Kubernetes directly from NBomber Studio. It's a game changer that brings a SaaS-like experience while letting you run your tests inside your own private K8s cluster.

Load Tests in Kubernetes​

Starting with this release, NBomber Studio lets you deploy and run tests in K8s. When you start a test, Studio spins up everything it needs — the Pods that run your load test, along with a NATS message broker for cluster mode. It also propagates your license key, the connection string for TimescaleDB, your JSON configuration, the container spec, and any CLI arguments you want to pass into your test. Once the test finishes, Studio cleans up the environment by destroying every resource it created during the session.

Let's take a look at the new capabilities. To create and run your test, we've added support for two project types:

• C# Script — best suited for simple load tests, demos, and dry runs where you want to test a few endpoints or a single microservice and get quick results. Its main benefit is that you can change the code on the fly without recompiling your scenario, which makes it ideal for trying out NBomber and iterating quickly.

• Docker — designed for production use cases. You provide a Docker image with your NBomber test, and Studio deploys it across as many Pods as you configure. You can also pass and dynamically edit the JSON configuration of your test scenarios.

C# Script​

Docker​

For more details, please check our documentation.

What's next​

• F# Script support.
• We're going to release a calendar scheduler that lets you run load tests at a specific time or on a recurring schedule (daily, weekly, etc.).
• Please follow our Roadmap page for the upcoming release.

With these additions, NBomber Studio is evolving beyond being just a UI for NBomber metrics into a full-fledged load testing platform. We are also working on the upcoming Load Testing in Kubernetes feature, which we expect to release in April.

Authentication enabled by default​

Starting with this release, NBomber Studio enables authentication by default. You can configure authentication settings to add new members, disable authentication entirely, or enable OIDC for external identity providers.

For more info please refer to this page.

Compare Runs​

Compare runs lets you select up to five load test sessions and compare them across multiple dimensions — latency, throughput, status codes, and more. You can quickly identify differences through percentage comparisons or explore detailed comparison charts for deeper analysis.

Trend Analysis​

Trend analysis is particularly useful for teams that want to track load test performance over longer periods, such as several months or even a full year. You'll see a degradation curve based on load test metrics summarized per day and plotted across the entire period.

Stop Session​

Sometimes you may start a load test and later decide to stop it to adjust settings or investigate an issue. Now you can simply select a session and click Stop.

Download HTML Report​

Sharing load test results previously required storing generated HTML reports separately. Now you can directly download an HTML report from NBomber Studio using the Download HTML Report option for any completed session.

Helm Charts​

We’ve published two Helm charts to simplify installation in Kubernetes:

• TimescaleDB Chart for running a single-node PostgreSQL (TimescaleDB OSS) used by NBomber Studio
• NBomber Studio Chart for deploying a single-node instance of NBomber Studio

These charts make it much easier to set up NBomber Studio in Kubernetes environments. You can find more info about installation on this page.

What's next​

Load Testing in Kubernetes​

We are putting the finishing touches on the feature: Load Testing in K8s. This will allow you to create and deploy load tests directly to Kubernetes from NBomber Studio, without requiring additional CI/CD jobs. We expect this to be released in April.

• HTML Report improvements
• NBomber Cluster improvements
• OpenTelemetry Reporting Sink
• What's next

HTML Report improvements​

One of the invisible but important changes in the new HTML Report is that it has been completely rewritten in TypeScript. For a long time, we used vanilla JavaScript with Vue, but over time it became difficult to manage complexity, reason about the code, and maintain correctness. Moving to TypeScript gives us type safety and makes future development more sustainable.

Now, let’s look at the visible improvements 🙂

Status code chart colors​

Previously, the colors for the status code chart were automatically selected by the charting library. This made it difficult to clearly distinguish successful responses from errors. In this release, we fixed that by introducing two distinct color groups—green for successful responses and red for errors—with slightly different shades.

This change significantly improves the clarity and readability of the status code chart.

Before: status codes chart

After: status codes chart

Advanced options​

This feature allows you to include custom metrics and plot them on the Throughput chart.

Metrics filter​

The metrics filter provides a simple text box that allows you to quickly find and focus on the metrics you are interested in.

Chart size​

With View mode, you can adjust the size of charts to make analysis more convenient and readable.

Full-screen mode​

Full-screen mode allows you to open any chart in full-screen view for more detailed analysis.

Failures tab​

To make failures more visible during test analysis, we added a red badge on the Failures tab that displays the aggregated failure count.

Aggregated failure counter​

In some cases, it’s useful not only to see the Fail RPS, but also the cumulative number of failures over time. Using the Advanced Options feature, you can now include the aggregated failure count directly on your chart.

NBomber Cluster improvements​

As you may already know, NBomber supports cluster mode, which allows you to run load tests in a distributed fashion. In this release, we introduced two major improvements focused on simplicity and flexibility.

Cluster via CLI args​

Previously, creating a cluster required defining and loading a JSON Config file that described the cluster topology. For example, the following configuration defines a cluster with one Coordinator and two Agents, all running the test_scenario.

"ClusterSettings": {

    "AutoCluster": {
        "ClusterId": "default",
        "NATSServerURL": "nats://localhost",

        "Coordinator": {
            "TargetScenarios": ["test_scenario"]
        },

        "Agent": {
            "TargetScenarios": ["test_scenario"],
            "AgentsCount": 2
        }
    }
}

You then had to load this configuration on each NBomber node to form a cluster (in our example, three-node cluster: 1 Coordinator + 2 Agents):

# start node 1
dotnet my-nbomber-test.dll --config=config.json

# start node 2
dotnet my-nbomber-test.dll --config=config.json

# start node 3
dotnet my-nbomber-test.dll --config=config.json

In v6.2.0, you can now configure the cluster via CLI arguments, without creating or loading a JSON configuration file. This is especially useful when you're preparing a CI/CD job and want to run your regular single-node NBomber test in cluster mode on K8s. No JSON config is required—just pass CLI arguments and you’re ready to go:

# --target sets TargetScenarios for both Coordinator and Agents

# to start node 1
dotnet my-nbomber-test.dll --target=test_scenario --cluster-id=default --cluster-agents-count=2 --cluster-nats-url=nats://localhost

# to start node 2
dotnet my-nbomber-test.dll --target=test_scenario --cluster-id=default --cluster-agents-count=2 --cluster-nats-url=nats://localhost

# to start node 3
dotnet my-nbomber-test.dll --target=test_scenario --cluster-id=default --cluster-agents-count=2 --cluster-nats-url=nats://localhost

All of the above commands can also be used to override settings loaded from config.json.

dotnet my-nbomber-test.dll --config=config.json --target=test_scenario --cluster-agents-count=5

If you want to run different scenarios on the Coordinator and Agents, this can also be done via CLI arguments. In the following example, "scenario_1" runs on the Coordinator and "scenario_2" runs on the Agents:

dotnet my-nbomber-test.dll \
    --cluster-coordinator-target=scenario_1 \
    --cluster-agent-target=scenario_2 \
    --cluster-id=default \
    --cluster-agents-count=5 \
    --cluster-nats-url=nats://localhost

A full list of available CLI options can be found at the following link.

Simplification of Manual cluster mode​

For ManualCluster mode — which provides more flexibility than AutoCluster by allowing explicit control over scenario placement — we introduced a small but important simplification.

AgentGroups are now optional. If you don’t need fine-grained placement, you can simply specify TargetScenarios, making ManualCluster nearly as simple to configure as AutoCluster.

Here is the updated type definition for ManualCluster configuration:

type ManualAgentSettings = {
    AgentGroups: (AgentGroupSettings list) option
    TargetScenarios: (string list) option
    AgentsCount: int
}

type ManualClusterSettings = {
    ClusterId: string
    NATSServerURL: string
    Coordinator: ManualCordSettings
    Agent: ManualAgentSettings
}

ManualCluster using only TargetScenarios​

This configuration is similar to AutoCluster but gives you manual control over which node runs as Coordinator and which runs as an Agent. In AutoCluster, this role assignment is handled automatically via leader election.

"ClusterSettings": {

    "ManualCluster": {
        "ClusterId": "test_cluster",
        "NATSServerURL": "nats://localhost",

        "Coordinator": {
            "TargetScenarios": ["test_scenario_1"]
        },

        "Agent": {
            "TargetScenarios": ["test_scenario_2"],
            "AgentsCount": 1
        }
    }
}

# start Coordinator node
dotnet my-nbomber-test.dll --config=config.json --cluster-node-type=coordinator

# start Agent node
dotnet my-nbomber-test.dll --config=config.json --cluster-node-type=agent

ManualCluster with AgentGroups​

If you need more flexible scenario placement, AgentGroups are still fully supported.

"ClusterSettings": {

    "ManualCluster": {
        "ClusterId": "test_cluster",
        "NATSServerURL": "nats://localhost",

        "Coordinator": {
            "TargetScenarios": ["publish_kafka_scenario"]
        },

        "Agent": {
            "AgentGroups": [
                { "AgentGroup": "readers", "TargetScenarios": ["read_db_scenario"] },
                { "AgentGroup": "writers", "TargetScenarios": ["write_dbScenario"] }
            ],
            "AgentsCount": 4
        }
    }
}

# start Coordinator node
dotnet my-nbomber-test.dll --config=config.json --cluster-node-type=coordinator

# start 2 Agent nodes under "readers" AgentGroup
dotnet my-nbomber-test.dll --config=config.json --cluster-node-type=agent --cluster-agent-group=reads
dotnet my-nbomber-test.dll --config=config.json --cluster-node-type=agent --cluster-agent-group=reads

# start 2 Agent nodes under "writers" AgentGroup
dotnet my-nbomber-test.dll --config=config.json --cluster-node-type=agent --cluster-agent-group=writes
dotnet my-nbomber-test.dll --config=config.json --cluster-node-type=agent --cluster-agent-group=writes

OpenTelemetry Reporting Sink​

We released a new OpenTelemetry Reporting Sink that allows you to stream real-time NBomber metrics to any observability platform that supports the OpenTelemetry (OTel) protocol.

This effectively enables integration with the vast majority of modern observability platforms. Previously, we had to implement a dedicated reporting sink for each platform individually. Now, if you want to store NBomber metrics in a system like Azure Monitor (or many others), you can simply use the OpenTelemetry sink.

What's next​

1. We are planning to release a new version of NBomber Studio soon—hopefully in February. This release will include several exciting features such as Compare Runs and Trends.

2. In parallel, we are finishing work on a highly anticipated feature for NBomber Studio: Load Testing in K8s. This will allow you to create and deploy load tests directly to Kubernetes from NBomber Studio, without requiring additional CI/CD jobs. We expect this to be released in late February or early March.

3. We have also started working on NBomber.YCSB, a port of the Yahoo Cloud Serving Benchmark (YCSB), which will allow you to generate popular synthetic workloads against your database and evaluate how well it handles load. Feel free to join and contribute! We have many exciting ideas ahead, and we believe this project will bring great value to the .NET community and beyond.

• Release Notes for v6.2.0
• Please follow our Roadmap page for upcoming releases.

Custom Metrics​

Metrics are numerical measurements reported over time. By default, NBomber automatically collects built-in metrics: CPU, RAM (usage), Data (sent/receive), and more. Besides built-ins, now you can define your custom metrics.

NBomber provides different types of custom metrics that you can use, depending on what you want to measure. These include:

• Counter: a cumulative count of events, such as the number of successful or failed requests.

var counter = Metric.CreateCounter("my-counter", unitOfMeasure: "MB");
counter.Add(2); // tracks a value that may increase or decrease over time
counter.Add(-1);

• Gauge: measures the last value recorded, such as memory usage or the number of active users at a given time.

var gauge = Metric.CreateGauge("my-gauge", unitOfMeasure: "KB");
gauge.Set(6.5);
gauge.Set(7.3); // set the current value of the metric

You can use your custom metrics with thresholds.

var scenario = Scenario.Create("scenario", async context =>
{
    ...
})
.WithThresholds(
    Threshold.Create(metric => metric.Counters.Get("my-counter").Value < 1000),
    Threshold.Create(metric => metric.Gauges.Get("my-gauge").Value >= 6.5)
);

Or define thresholds in the JSON config file:

"ThresholdSettings": [
    { "Metric": ["my-counter", "value < 1000"] },
    { "Metric": ["my-gauge", "value >= 6.5"] }
]

All these capabilities open up new ways to validate specific conditions. After running a load test, you will be able to view and check custom metric values in the HTML report.

You can also see them in the Console UI.

For more information about this feature, please refer to our documentation.

HTML Report: New Metrics Tabs​

We've slightly rearranged the tabs in the HTML report to provide more convenient navigation based on the scenario context.

HTML Report: Printout Threshold code body​

In the previous v5.8 release, NBomber introduced a new feature called Runtime Thresholds. This feature allows you to define thresholds using C#/F# code or JSON expressions. Defining thresholds in JSON was a bit more complete as you could see the JSON expression text in your HTML report. The downside of defining a threshold using C#/F# code was that checkExpression wasn't displayed in the final HTML report.

Well, we've fixed that! In the new release, we've added functionality that iterates over the code body and prints it.

GetScenarioTimerTime​

We've added a feature that allows you to get the current time of the scenario timer. This can be useful for scenarios that depend on time-based criteria within the scenario.

var scenario = Scenario.Create("hello_world_scenario", async context =>
{
    var currentScenarioTime = context.GetScenarioTimerTime();
    
    context.Logger.Information($"Current scenario time: {currentScenarioTime}");

    return Response.Ok();
})

You can find the complete example by this link.

Custom Latency​

We've added an option to pass custom latency, which can be useful for streaming-based scenarios where you need to measure the latency between several independent processes that publish and consume data independently. In the future, we will describe this use case in our documentation and provide guidance on how to handle it.

Response.Ok(customLatencyMs: 120)

New System Metrics​

We've expanded our standard list of system metrics by adding a few important ones:

• dns-lookups-duration - The average time taken for a DNS lookup
• dns-lookups-requested - The number of DNS lookups requested since the process started
• gc:loh-size - The number of bytes for the large object heap
• gc:time-in-gc - The percent of time in GC since the last GC

CSV Report: Failures Stats​

Previously, NBomber's CSV reports didn't include statistics on failures. The default way to get failure stats was by getting them from the final NodeStats record or by using IReportingSink and subscribing to real-time metrics. However, some of our clients still rely on CSV for extracting data to process further. To maintain consistency with other reports, we've now added failure statistics to the CSV report as well.

Logs folder change for NBomber Cluster​

When running NBomber Cluster, the Coordinator folder and Agent(s) folders are created at the same level. This can be inconvenient if you're storing logs in blob storage. We've addressed this by nesting the Agent logs under the same session folder path. The folder structure now looks like this:

• sessionId/reports - for Coordinator
• sessionId/reports/agentId - for Agent

XML documentation link​

Since day one, NBomber has provided documentation comments for IDE IntelliSense. While this works well, there were cases where a more detailed documentation page would be helpful. To address this, we've added a link to the Documentation page that describes specific features. Now, you can click the Documentation link in the pop-up window and navigate to the full page documentation.

Performance improvements​

At NBomber, we are always striving to optimize performance, aiming to make our framework the fastest and most resource-efficient. This release is no exception.

Mean and StdDev​

We made an optimization related to the calculation of Mean and StdDev. For real-time reporting, we need to calculate these values periodically (each 5 sec). Previously, we had to iterate through all items to calculate the Mean. However, we discovered that we can completely skip this step by tracking the Mean as we go. This optimization also contributed to the performance of StdDev, which depends on the Mean calculation.

JSON serialization​

We found that for long-running scenarios, especially those lasting over 1-2 hours, generating the HTML report can be quite heavy. The main contributor to this is extra memory allocations that occur when serializing the metrics history in JSON for the HTML report. Historically, we used the F# JSON serializer, which works well with F# types and also plays an important role in schema validation for JSON config parsing. F# JSON validates the JSON based on the types you pass, which is great. However, the downside is that this library was written a long time ago and isn't very memory-efficient. In the new release, we've limited the use of F# JSON, using it only for parsing the JSON config. For all other cases, we've switched to the standard .NET System.Text.Json, which delivers much better performance. In addtition, we conducted a benchmark using one hour of statistics for five concurrent scenarios.

Statistics compression​

We've started compressing statistics, which has reduced memory usage. While we hadn't encountered any issues with this in the past, we still decided to optimize it.

Previously, storing one hour of stats for a single scenario required ~5MB of RAM. With the new optimizations, we've reduced it to just 360KB. In absolute numbers, the impact is minimal, but optimizations like this help reduce GC pressure, making them worthwhile.

NBomber Studio​

Alongside NBomber, we released a new version of NBomber Studio. This update includes bug fixes related to frontend session handling and improvements to the UI. We've removed manual steps, such as having to click the refresh or search button to get updates.

In the previous version, UI auto-updates only worked for an open session, but we've now expanded this functionality further. The search button has been removed from filtering, and now, when you change something—such as the date range or test name—requests are sent automatically. Additionally, we’ve added auto-updating for the History tab, as it's a useful feature.

What is next​

• Release Notes for v6.0.1
• Please follow our Roadmap page for the upcoming release.

• Scenario Weight
• Native extensions for probability distributions
• Runtime Thresholds

Scenario Weight​

Previously, NBomber allowed introducing randomness, but primarily within a single Scenario or Step. There was no native way to introduce randomness across multiple scenarios. In the new version we’ve unlocked a highly requested feature - Scenario Weight. The Weight value allows you to control the relative frequency or probability of Scenario's execution. Scenarios with higher weights will run more frequently during the test, while those with lower weights will execute less often, providing flexible control over load distribution. NBomber uses the weights to determine the probability of each Scenario.

Example: In this case, the "read_scenario" probability is 80%, and "write_scenario" is 20%.

var readScenario = Scenario
    .Create("read_scenario", ...)
    .WithWeight(80); // sets 80%

var writeScenario = Scenario
    .Create("write_scenario", ...)
    .WithWeight(20); // sets 20%

NBomberRunner
    .RegisterScenarios(readScenario, writeScenario)
    .Run();

For more information, please refer to our documentation.

Native extensions for probability distributions​

In our documentation, we referenced several popular probability distributions: Uniform, Zipfian, and Multinomial. To use them, we noted that you need to install the MathNet.Numerics package.

To start working with probabilities, you should install MathNet.Numerics package. It's a free package that contains many useful probabilies.

In the new version of NBomber, we decided it would be beneficial to have these distributions available out of the box. Since they will be included with NBomber, we aimed to ensure they are highly performant compared to MathNet.Numerics. One downside of MathNet.Numerics is that its API caused extra allocations, which we wanted to avoid in our implementation.

• We added native support for Zipf distribution, which allows us to choose an item according to the Zipfian distribution. For example, when choosing a record, some records will be extremely popular (the head) while most records will be unpopular (the tail). The Zipf distribution is a mathematical concept that describes the frequency of items ranked from most common to least common in a dataset, like word frequencies in a book or city populations. Also, this distribution can be used to describe hot keys/items (skewed key distributions) in database or popular tweets in Tweeter web service.

var scenario = Scenario.Create("zipfian_distribution", async context =>
{    
    var stepNumber = context.Random.Zipf(n: 5, s: 1.3);

    switch (stepNumber)
    {
        case 1: // run step 1
            Step.Run("step_1", ...);
            break;

        case 2: // run step 2
            Step.Run("step_2", ...);
            break;
        
        ...
    }
});

• We added native support for Multinomial distribution, which allows us to specify probabilities for each item. For example, assigning a probability of 0.95 to the Read operation, 0.05 to the Update operation, and 0 to Delete and Insert. This setup creates a read-heavy workload. To simulate Multinomial distribution we can use extension method Choice for Random object.

// 70% for read, 20% for write, 10% for delete
var items = new[] { ("read", 70), ("write", 20), ("delete", 10) };

var scenario = Scenario.Create("multinomial_distribution", async context =>
{    
    var randomItem = context.Random.Choice(items);

    switch (randomItem)
    {
        case "read": // 70% for read
            Step.Run("read", ...);            
            break;

        case "write": // 20% for write
            Step.Run("write", ...);
            break;

        case "delete": // 10% for delete
            Step.Run("delete", ...);
            break;
    }
});

For more information, please refer to our documentation.

Runtime Thresholds​

We're excited to announce that this feature has finally arrived for NBomber fans. Assertions and thresholds have been supported from day one. Typically, thresholds with assertions are applied after the active load test has finished, once all the available statistics have been provided. Here’s an example:

var scenario = Scenario.Create("http-scenario", async context => ...);

var result = NBomberRunner
    .RegisterScenarios(scenario)
    .Run();

// the test is finished, and we can now retrieve the final stats 
// to check against our thresholds.
var scnStats = result.ScenarioStats.Get("http-scenario");

Assert.True(scnStats.Fail.Request.Percent <= 5);  

However, there may be cases where you want to define thresholds that are active throughout the entire test run — we refer to these as Runtime Thresholds. Here is an example:

var scenario = Scenario.Create("http_scenario", async context =>
{
    ...
})
.WithThresholds(  
    Threshold.Create(scenarioStats => scenarioStats.Fail.Request.Percent < 10),

    Threshold.Create("step_1", stepStats => stepStats.Fail.Request.Percent < 10),
)

The key feature of Runtime Thresholds is that they can be defined in the JSON Config. Generally, Runtime Thresholds offer flexibility in terms of configuration: you can set them to run without causing early termination of the test or delay the start of the threshold check.

"ThresholdSettings": [

    { "OkRequest": "RPS >= 80" },

    { "FailRequest": "Percent < 10" },

    { "OkLatency": "max < 100" },

    { "StatusCode": ["500", "Percent < 5"] },
    { "StatusCode": ["400", "Percent < 10"], "AbortWhenErrorCount": 5 }
    { "StatusCode": ["200", "Percent >= 90"], "StartCheckAfter": "00:00:20" }
]

After completing a test that includes Runtime Thresholds, the results of the threshold checks will be available in the HTML report. This allows you to easily review and analyze how the system performed against the defined thresholds throughout the test run.

For more information, please refer to our documentation.

Get/Find/Exist extensions for Stats data​

The Stats Extensions include the useful helper methods: Get, Find, Exists, which are used to work with ScenarioStats, StepStats, StatusCodes, and other similar data.

var result = NBomberRunner
                .RegisterScenarios(scenario1, scenario2)
                .Run();

// gets scenario stats: 
// it throws exception if "scenario_1" is not found 
var scnStats1 = result.ScenarioStats.Get("scenario_1");

// finds scenario stats: 
// it returns null if "scenario_2" is not found 
var scnStats2 = result.ScenarioStats.Find("scenario_2");

// check that step "login" exist in the list
bool isLoginExist = scnStats1.StepStats.Exists("login");

// gets "login" step stats
// it throws exception if "login" is not found 
var loginStats = scnStats1.StepStats.Get("login");

// check that status code "503" exist in the list
bool isExist = scnStats1.Fail.StatusCodes.Exists("503");

// get stats of status code "503"
// in case of status not found, the exception will be thrown
var statusCode = scnStats1.Fail.StatusCodes.Get("503");

HTML report improvements​

We’ve improved the HTML report by updating the logic behind the latency indicators panel. Previously, this panel displayed cumulative results across all scenarios. Based on user feedback, we adjusted it to show data specific to the selected items. For instance, if you select a specific step to analyze, the panel will re-render to display data filtered specifically for that step or scenario.

Example: Scenario latency indicators panel.

Example: Step latency indicators panel.

Disable rendering of Metrics Table in Console​

In certain cases, some of you may have noticed that using the console's stdout for debugging can be tricky, as the NBomber Metrics Table may overwrite your printouts. If you want to use the console for debugging purposes, you can now disable the rendering of the Metrics Table in the console.

In code:

NBomberRunner
    .RegisterScenarios(scenario)
    .DisplayConsoleMetrics(false)

Also, via CLI args:

dotnet MyLoadTest.dll --display-console-metrics=false

Also, via JSON Config:

{
  "GlobalSettings": {        
    "DisplayConsoleMetrics": false
  }
}

LoadConfig improvements​

We have a method that loads JSON configuration. Previously, this method only supported loading data from a file or URL. We've extended it to allow loading JSON content as a string.

NBomberRunner
    .RegisterScenarios(scenario) 
    .LoadConfig("{ YOUR JSON }")

What is next​

• Release Notes for v5.8.
• Please follow our Roadmap page for the upcoming release.

TimescaleDB support​

TimescaleDB is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL and packaged as a PostgreSQL extension, providing automatic partitioning across time and space (partitioning key), as well as full SQL support. Using Reporting Sinks for TimescaleDB, you can store NBomber metrics into TimescaleDB and analyze your performance results with NBomber.MetricsUI (in development) or Grafana.

Example:

var timescaleDbSink = new TimescaleDbSink();

var scenario = Scenario.Create("scenario", async context => { ... });

NBomberRunner
    .RegisterScenarios(scenario)    
    .WithReportingInterval(TimeSpan.FromSeconds(5))
    .WithReportingSinks(timescaleDbSink)
    .LoadInfraConfig("infra-config.json");

{
  "TimescaleDbSink": {
    "ConnectionString": "Host=localhost;Port=5432;Database=metricsdb;Username=timescaledb;Password=timescaledb;Pooling=true;Maximum Pool Size=300;"
  }
}

During the load test run, NBomber's metrics will appear in TimescaleDB, and you can query them.

For more information, please refer to our documentation.

WebBrowser testing​

The primary use case for the browser testing is to test performance at the browser level. Browser-level testing allows you to measure user experience and identify issues that are challenging to detect at the protocol level. You can interact with the browser to script tests much closer to the actual website experience. You'll get insight into what your user sees on a browser level and catch issues skipped from the protocol level. At the same time, you can adopt a hybrid approach to performance testing by leveraging the existing functionalities of NBomber and performing backend load testing on a protocol level, all in the same scenario. This gives you a full picture of how the user experience is under different load scenarios.

NBomber.WebBrowser package adds a few useful extensions to the Playwright and Puppeteer libraries. Playwright is a library which provides a high-level API to control Chrome or Firefox. Playwright was created specifically to accommodate the needs of end-to-end testing. Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox. Test on Windows, Linux, and macOS, locally or on CI, headless or headed with native mobile emulation.

// downloading the Chrome
var installedBrowser = await new BrowserFetcher(SupportedBrowser.Chrome).DownloadAsync(BrowserTag.Stable);
var browserPath = installedBrowser.GetExecutablePath();

using var playwright = await Playwright.CreateAsync();

await using var browser = await playwright.Chromium.LaunchAsync(
    new BrowserTypeLaunchOptions
    {
        Headless = true,
        ExecutablePath = browserPath
    }
);

var scenario = Scenario.Create("playwright_scenario", async context =>
{
    var page = await browser.NewPageAsync();

    await Step.Run("open nbomber", context, async () =>
    {
        var pageResponse = await page.GotoAsync("https://nbomber.com/");

        var html = await page.ContentAsync();
        var totalSize = await page.GetDataTransferSize();

        return Response.Ok(sizeBytes: totalSize);
    });

    await Step.Run("open bing", context, async () =>
    {
        var pageResponse = await page.GotoAsync("https://www.bing.com/maps");

        await page.WaitForSelectorAsync(".searchbox input");
        await page.FocusAsync(".searchbox input");
        await page.Keyboard.TypeAsync("CN Tower, Toronto, Ontario, Canada");

        await page.Keyboard.PressAsync("Enter");
        await page.WaitForLoadStateAsync(LoadState.Load);

        var totalSize = await page.GetDataTransferSize();
        return Response.Ok(sizeBytes: totalSize);
    });

    await page.CloseAsync();

    return Response.Ok();
})
.WithWarmUpDuration(TimeSpan.FromSeconds(3))
.WithLoadSimulations(
    Simulation.KeepConstant(1, TimeSpan.FromSeconds(30))
);

For more information, please refer to our documentation.

NBomber improvements​

In this release, we added a list of imrovements.

SessionId override​

For specific scenarios, it can be helpful to control the session-id fully. Now you can pass your custom session-id via F#/C# API or CLI args.

NBomberRunner.RegisterScenarios(...)
             .WithSessionId("my-session-123")
             .Run()

CLI args:

--session-id=my-session-123

Manipulate data for reports​

Another feature that we added is the ability to manipulate data for reports. There could be cases when you need to filter out some of the metrics data from your report. One example can be a "pause step" or "hidden step" that you can create and use but don't want to see in final reports. For this, you can use a new method:

Example:

NBomberRunner
    .RegisterScenarios(scenario)
    .WithReportFinalizer(data =>
    {
        var stepsStats = data.GetScenarioStats("mongo-read-write")
            .StepStats
            .Where(x => x.StepName != "pause")
            .ToArray();

        data.GetScenarioStats("mongo-read-write").StepStats = stepsStats;

        return data;
    })
    .Run();

Cluster info for HTML report​

Also, we extended our HTML report by adding NBomberConfig and Cluster Info with a list of agents who participated in the cluster.

What is next​

Please follow our Roadmap page for the upcoming release.

This article will demonstrate how to load test HTTP API using C# and the NBomber framework.

To read the full article please follow this link.

• Scheduler timer calibration
• Fixed stopping process for scenarios
• Add API method NBomberRunner.EnableStopTestForcibly
• Fixed. Incorrect handling of stop scenarios earlier blocks the console
• Fixed. Under the high load the console timer is desynchronize
• Added Log.CloseAndFlush
• Refactored functionality to stop Scenario actors
• Refactored measuring time to use time buckets
• Changed the default ReportingInterval to 5 sec
• Adapted NBomber for int64 changes in NBomber.Contracts

NBomber.Cluster​

• [Fixed] Added auto reconnection for NATS clients on initialization phase
• [Fixed] Coordinator should wait forever when MinAgentsCount is specified but not satisfied
• [Fixed] Coordiantor fails in case when only EmptyScenario defined in TargetScenarios

New Contributors​

• @candritzky made their first contribution in https://github.com/PragmaticFlow/NBomber/pull/566

Full Changelog: https://github.com/PragmaticFlow/NBomber/compare/v4.1.0...v4.1.2

In this release, we focused on improving engine performance, fixing RAM consumption issues, improving UI/UX, including reporting, extending API to provide flexibility but keep it SIMPLE as before.

Console​

In the previous version, we always had a desire to improve the console output. The main pain point for us was the progress bar, which did not scale well while resizing the console, and sometimes it literally broke down when you are writing log messages to the console. The second most important problem for us was the table rendering, which was also displayed crookedly while resizing windows, and we wanted to fix this. We started considering Spectre.Console project for a long time as a good candidate for replacement and then we started a smooth transition. It was good with tables, but we still had some problems with the progress bar. Since for logs, we use Serilog, we figure out that we need to develop a proper integration between Serilog and Spectre.Console to fix our issue. After all, we developed Serilog.Sinks.SpectreConsole that nicely integrates Serilog with Spectre.Console. It's open-source, and you can use it for your integrations too. Now our console UI is much more stable, and I hope more beautiful :)

HTML report​

We use Vue.js, which amazed us with its simplicity and minimalism for rendering HTML. Compared to the previous version, we have significantly expanded the functionality by adding: fail stats, status codes, hints analyzer results. You can take a look at the new HTML report here Also, we switched to Google Charts for rendering charts since it's free and under Apache 2.0 license.

Runtime Improvements

In this section, we will discuss some internals of NBomber that improve the performance and stability of the system.

Stats collecting​

We refactored our stats collecting module to use the actor model via F# MailboxProcessor and to have one StatsScenarioActor per scenario. Also, we send metrics by batches to eliminate the overload of one actor by many concurrent ScenarioActors that execute steps and send metrics.

Minimizing RAM usage​

Compression of stats results​

We refactored our statistics module to fix the issue with memory growing footprint for long-running tests. Now you can run tests that will work for years and have a stable (usually small) memory footprint. This is achieved essentially due to one important optimization: all results are located in memory but decomposed into buckets with the same or very similar results. Each such bucket has its own counter, which is monotonically growing. Due to this optimization, we got quite effective compression. Aside from this, such optimizations are often used in time-series databases where many metrics of the same value or very similar can arrive at one point in time.

Default step timeout​

Another important thing was introducing default timeout for steps. NBomber v1 did not support any timeouts out of the box, and this task shifted to the shoulders of the developers. For example, the HTTP plugin added its own timeout, which was quite sane. But on the other hand, everyone needs such a basic thing as a timeout, and forcing everyone to implement it is not a good idea. Also, newbies often weren't aware of this, and afterward, it led to the thread pool starvation problem. It especially was visible when testing very slow API using a big request rate, say 2-4K requests per sec from 1 node. It leads to many tasks (.NET Task) was activated, consumed RAM, and never finished but growing. In the new version, each step by default contains a timeout of 1 second and can be changed by the user.

Removed uneccesary memory allocations​

We refactored a few places where we did extra memory allocations on every step invocation. For example, StepContext was previously created on every invocation.

Task computation expression​

Internally, we use FsToolkit.ErrorHandling library that provides handy F# computation expressions and corresponding extensions. With the latest updates to this library for work with Task, we have automatically switched to using Ply library, which provides a low overhead Task computation expression.

Low-level optimizations​

GC tuning​

Also, we tuned GC a bit. At runtime, we set:

GCSettings.LatencyMode <- GCLatencyMode.SustainedLowLatency

And also, for the project settings, we apply:

<ServerGarbageCollection>true</ServerGarbageCollection>
<ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>

New Statistics

One of the major changes we had been working on is the statistics module. Initially, we didn't really want to change it, but using NBomber v1 in production, we often ran into bottlenecks or questions regarding providing advanced statistics.

Ok and Fail stats​

An important addition to the previous statistics was that we started tracking the fail stats with the full scope, including data transfer for fails, latency, and percentiles.

type OkStepStats = {
    Request: RequestStats
    Latency: LatencyStats
    DataTransfer: DataTransferStats
    StatusCodes: StatusCodeStats[]
}

type FailStepStats = {
    Request: RequestStats
    Latency: LatencyStats
    DataTransfer: DataTransferStats
    StatusCodes: StatusCodeStats[]
}

type StepStats = {
    StepName: string
    Ok: OkStepStats
    Fail: FailStepStats
}

Now you can write your assertions for fails stats too. Also, you will see them in reporting, including real-time reporting too.

// here is an example of assertion
let step1 = scenarioStats.StepStats.[0]

// ok stats
test <@ step1.Ok.Request.Count >= 4 @>
test <@ step1.Ok.Latency.MinMs <= 503.0 @>
test <@ step1.Ok.Latency.Percent50 <= 505.0 @>
test <@ step1.Ok.DataTransfer.MinBytes = 100 @>

// fail stats
test <@ step1.Fail.Request.Count = 0 @>
test <@ step1.Fail.Latency.MinMs = 0.0 @>
test <@ step1.Fail.DataTransfer.MinBytes = 0 @>

Unit of measure​

In the new version, we changed basic data types to represent latency and data transfer.

Latency​

We started using float instead of integer to expand the range of possible results for latency, for example: [0.2 ms, 1.5 ms, 9.3 ms]. We were pushed to this by the case of testing a high-speed in-memory database that had a latency of less than 1 ms, and in our reports, we saw it as 0 ms instead of 0.1 - 0.5 ms. Just for the record, most load test tools represent latency as integer, meaning using them, you cannot cover such cases since your results will be 0 ms.

type LatencyStats = {
    MinMs: float
    MeanMs: float
    MaxMs: float
    Percent50: float
    Percent75: float
    Percent95: float
    Percent99: float
    StdDev: float
    LatencyCount: LatencyCount
}

Data transfer​

We switched to using Byte instead of KB to keep parity with standard tools for data transfer stats.

type DataTransferStats = {
    MinBytes: int
    MeanBytes: int
    MaxBytes: int
    Percent50: int
    Percent75: int
    Percent95: int
    Percent99: int
    StdDev: float
    AllBytes: int64
}

In addition, we provide helper functions for converting Byte to KB and MB. This can be useful when you want to assert on KB or MB.

test <@ kb (step1.Ok.DataTransfer.MinBytes) <= 50.0 @>
test <@ mb (step1.Ok.DataTransfer.MinBytes) <= 50.0 @>

Hint Analyzer​

In this release, we have integrated a new HintAnalyzer feature that allows to analise the results of statistics and, based on this, displays hints. HintAnalyzer may refer not only to the statistics data but also to the use of certain plugins. Any NBomber plugin can add new analyzers. For example, PingPlugin pings the target host before starting any test, and after that, it analyzes received results and prints hints.

plugin stats: NBomber.Plugins.Network.PingPlugin

Here's a hint by PingPlugin.

Also, HintAnalyzer can suggest some other hints related to the usage of NBomber.

You can disable HintAnalyzer if you want

Scenario.create "simple_http" [step]
|> NBomberRunner.registerScenario
|> NBomberRunner.disableHintsAnalyzer

New API​

In the new version, we have added a couple of additions that improve the convenience of writing test scripts without adding new abstractions to NBomber itself. We are very scrupulous about adding any new features to NBomber as they can introduce additional abstractions and hence additional complexity. Our initial goal was to build an easy-to-use framework with as few abstractions as possible. Therefore, any new feature that we consider should be easy to understand and harmoniously fit into NBomber.

Status code​

We have added the ability to specify any status codes you want or your protocol, or your API service returns.

Response.ok(statusCode = 100)

Also, we provide statistics on status codes.

type StatusCodeStats = {
    StatusCode: int    
    Message: string
    Count: int
}

Step invocation count​

For certain tests, it may be necessary to understand the current step invocation count. For example, you want to change the behavior of your step execution when the step's invocation counter is reached 100 invocations. For such cases, you need to start a counter and manually increment it. NBomber now supports this out of the box.

let step = Step.create("step", fun context -> task {    
    
    // invocation count of the current step 
    // (will be incremented on each invocation)
    context.InvocationCount // int
    
    return Response.ok()
})

Step timeout​

In the new version, each step by default contains a timeout of 1 second and can be changed by the user.

let step = Step.create("step", 
                       timeout = seconds 0.5,
                       execute = fun context -> task {    
    
    return Response.ok()
})

Dynamic step order​

We decided to provide the ability to change the order of the steps at runtime dynamically. Also, it allows changing the number of steps per scenario iteration. This feature opens up a new horizon of possibilities for writing load tests. We came up with this idea from a real case when we tested the database, and we needed to introduce a certain randomity into our tests. For a better understanding, I suggest looking at an example:

// by default these steps will run sequentially
Scenario.create "test_redis" [step1; step2]

// in this case, only step2 will be invoked
|> Scenario.withDyncamicStepOrder(fun () -> [| 1 |]) 

// in this case, we reversed the order of steps
|> Scenario.withDyncamicStepOrder(fun () -> [| 1; 0 |]) 

// in this case, we introduce randomity
|> Scenario.withDyncamicStepOrder(fun () -> 
    let index1 = random.Next(0, 1)
    let index2 = random.Next(0, 1)
    [| index1; index2 |]
) 

Basically, with this feature, you can introduce convenient load distribution. For example, you can define a scenario where you will test Redis database with the following request distribution: 30% for write requests and 70% for read requests.

Scenario.create "test_redis" [write_redis; read_redis]
|> Scenario.withDyncamicStepOrder(fun () -> 
// using a randomator you can specify:
//  - 30% will be write_redis
//  - 70% will be read_redis
)

Scenario info​

It's a context property that contains info about the current Scenario. The main use case is to use ScenarioInfo.ThreadId as correlation id for your requests. Another one is to use ScenarioInfo.ThreadNumber for partition your requests. For example, you have a database, and you want to split all your requests into 4 partitions.

let step = Step.create("step", fun context -> task {    
    // gets the current scenario thread id
    // you can use it as correlation id
    context.ScenarioInfo.ThreadId
    context.ScenarioInfo.ThreadNumber
    context.ScenarioInfo.ScenarioName
    context.ScenarioInfo.ScenarioDuration

    return Response.ok()
)

TimeSpan extensions​

We added TimeSpan extensions for F# and C# to have a more expressive API.

milliseconds 500
seconds 2
minutes 1

Init-only scenarios​

NBomber had the ability to provide Scenario initialization via Scenario.init in the very first version. It had a mandatory restriction to create a Scenario you should provide at least one Step.

// you should provide at least one step
Scenario.create "write_scenario" [write_step] 
|> Scenario.init populateKafka

This was convenient until the moment when several scenarios had a dependency on the same initialization. In this case, only one initializer should be executed, and it's getting tricky since NBomber v1 didn't provide any functionality for this.

// both scenarios depend on populateKafka
Scenario.create "write_scenario" [write_step] 
|> Scenario.init populateKafka

Scenario.create "read_scenario" [read_step] 
|> Scenario.init populateKafka

In NBomber v2, we can define Scenario without steps and have only the init or clean function.

let initScenario = 
    Scenario.create "init_scenario" [] // no steps
    |> Scenario.init populateKafka
    |> Scenario.clean cleanKafka

let writeScenario = Scenario.create "write_scenario" [write_step] 
let readScenario = Scenario.create "read_scenario" [read_step] 

// now all scenarios will be executed
NBomberRunner.registerScenarios [initScenario; writeScenario; readScenario] 
|> NBomberRunner.run

// now you can simply chnage scenarios
NBomberRunner.registerScenarios [initScenario; writeScenario] 
|> NBomberRunner.run

// or
NBomberRunner.registerScenarios [initScenario; readScenario] 
|> NBomberRunner.run

LoadSimulation inject random​

Injects a random number of scenario copies (threads) per 1 sec during a given duration. Use it when you want to maintain a random rate of requests without being affected by the target system's performance under test.

Scenario.create "scenario_2" [read_step]
|> Scenario.withLoadSimulation [
    InjectPerSecRandom(minRate = 10, maxRate = 50, during = seconds 10)
] 

Logs​

In the new version, we write a log file to the current session folder.

/{current_session_id}/nbomber-log-{date}.txt
/{current_session_id}/reports

HTTP plugin​

NBomber.Http has been slightly modified to make it easier to be used. In the previous version, this plugin provided an HttpStep wrapper over the standard Step that hid many details and added extra magic. In the new version, we have reworked the approach with DSL plugins to use standard NBomber abstractions instead of creating new ones. Now, the HTTP plugin contains HttpClientFactory and has an HTTP module with functions for building HTTP requests.

let httpFactory = HttpClientFactory.create()

let step = Step.create("simple step", 
                       clientFactory = httpFactory,
                       execute = fun context ->

    Http.createRequest "GET" "https://nbomber.com"
    |> Http.withHeader "Accept" "text/html"
    |> Http.withBody(new StringContent("{ some JSON }"))
    |> Http.withCheck(fun response -> task {        
        return if response.IsSuccessStatusCode then Response.ok()
                else Response.fail()
    })
    |> Http.send context
)

InfluxDb plugin​

InfluxDb plugin is supplemented with tracking fail stats and load simulation value. In Grafana, you can render your load simulation timeline and compare it with what your target system can handle.