Capacity Testing Guide: Load Limits & Scalability

Overview

A product launch we supported last year was expected to triple daily traffic. Load testing passed at the target peak, but on launch day the checkout API began failing at 2.3x the expected load. The database connection pool, not the application servers, was the first bottleneck. Capacity testing would have caught it because it deliberately pushes past the expected peak to find the actual breaking point.

Capacity testing answers one question: how much load can your system handle before it degrades or fails? Unlike load testing, which validates behavior at expected load, capacity testing increases load until a resource saturates. It reveals whether your database connection pool exhausts at 500 concurrent users, whether your cache tier saturates at 10,000 requests per second, or whether your application servers run out of memory under sustained peak load.

If you are comparing load, stress and spike testing side by side, see Stress Testing vs Load Testing vs Spike Testing. This guide focuses on finding your system's actual breaking point and turning it into a capacity plan — not on comparing test types.

The flow below shows the capacity testing process from load profile definition to CI integration — each step feeds the next, and the loop between bottleneck identification and scalability testing is where most capacity issues get fixed:

The results feed directly into capacity planning: how many servers you need, when to scale, and what bottlenecks to fix before the next traffic surge.

Related reading: Performance Testing Guide, API Load Testing with Postman & k6 and Failover Testing Guide.

When to Use

• Before major traffic events such as Black Friday, product launches or viral campaigns.
• After infrastructure changes such as new hardware, cloud instance types or architectural changes.
• For capacity planning when finance and infrastructure teams need data to budget for growth.
• When onboarding enterprise customers who need proof that the platform can handle their projected load.
• For compliance with SLAs or government tenders that require documented capacity tests.
• After major releases that change data access patterns, caching or service boundaries.

Core Concepts

# Capacity Testing flow — applicable to k6 v0.50, JMeter 5.6, Gatling 3.10
Capacity Testing
├── Load Profile Definition
│   ├── Baseline (normal traffic)
│   ├── Expected Peak (planned events)
│   └── Stress (beyond expected limits)
├── Resource Monitoring
│   ├── CPU, Memory, Disk, Network
│   ├── Database Connections, Thread Pools
│   └── Queue Depths, Cache Hit Rates
├── Bottleneck Identification
│   ├── First Resource to Saturate
│   ├── Cascading Effects
│   └── Recovery Behavior
└── Scalability Validation
    ├── Horizontal Scaling (add nodes)
    ├── Vertical Scaling (bigger nodes)
    └── Auto-Scaling Response Time

Load Profiles

Capacity tests are meaningless without realistic load. Define three profiles:

Profile	Users/Requests	Duration	Purpose
Baseline	Normal daily peak	1 hour	Establish current capacity
Expected Peak	3x baseline	30 minutes	Validate headroom for growth
Stress	5-10x baseline	Until failure	Find the breaking point

The key parameters to vary are concurrent users or requests per second, the request mix (read vs. write, heavy vs. light endpoints), payload size (small JSON vs. multi-MB file uploads), and data volume (empty database vs. production-sized dataset). I once ran a capacity test with a 90/10 read/write mix and missed a write-path bottleneck that only appeared at 60/40 — the mix matters as much as the load.

Incremental Load Testing

Increase load in steps, holding each level for a fixed duration. This reveals exactly where degradation begins.

// k6 v0.50 incremental load test
import http from 'k6/http';

export const options = {
  stages: [
    { duration: '5m', target: 100 },  // baseline
    { duration: '5m', target: 300 },  // expected peak
    { duration: '5m', target: 500 },  // stress
    { duration: '5m', target: 700 },  // push further
    { duration: '5m', target: 0 },    // recovery
  ],
};

export default function () {
  http.get('https://api.qapractices.internal/v1/health');
}

Capacity Test with k6 Thresholds

Add pass/fail criteria directly in the test so CI can fail when capacity drops below acceptable limits.

// k6 v0.50 capacity test with thresholds
export const options = {
  stages: [
    { duration: '5m', target: 100 },
    { duration: '5m', target: 300 },
    { duration: '5m', target: 500 },
    { duration: '5m', target: 700 },
    { duration: '5m', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.05'],
    http_req_duration: ['p(95)<500', 'p(99)<1000'],
  },
};

At each step, record four things: response time percentiles (P50, P95, P99), error rate and error types, throughput in requests per second, and resource utilization across CPU, memory, disk I/O and network. I log these to a time-series database so I can correlate the exact moment a resource saturates with the exact moment latency spikes — without that correlation, you are guessing.

Resource Exhaustion Testing

Identify which resource runs out first by monitoring all layers. The resource that saturates first is your real bottleneck — and it is rarely the one you predicted. I have run capacity tests where the team was convinced the database was the bottleneck, but the real limit was the load balancer's connection table.

CPU

CPU saturation is the easiest to spot: usage climbs toward 100% and response times start to degrade. Increase the request rate until CPU plateaus, then correlate the plateau with P99 latency. If P99 climbs before CPU hits 100%, the bottleneck is somewhere else — usually a lock or a queue.

Memory

Memory exhaustion is sneakier. The heap or RSS grows until the process is killed by the OS (OOM killer on Linux, jetsam on iOS) or starts swapping to disk. Run sustained load and watch for steady growth — if memory never plateaus, you have a leak. I once found a memory leak at the 4-hour mark that looked fine for the first 3 hours.

Database connections

Pool exhaustion shows up as connection errors and request queueing. Increase concurrent users while monitoring active_connections — when the pool hits its max, new requests wait. The symptom is rising latency without rising CPU, because requests are blocked waiting for a connection, not doing work.

Thread pool

When all threads are busy, requests wait in queue. Track pool utilization and queue depth under increasing load. Thread pool exhaustion looks similar to database pool exhaustion, but the queue is in your application, not the database.

Disk I/O

High iowait, slow queries and timeouts point to disk saturation. Run heavy write loads or large file uploads and monitor disk latency. Disk I/O is often the hidden bottleneck on cloud instances with shared storage — I have seen AWS gp2 volumes become the limit before the application servers did.

Network bandwidth

Throughput caps and packet loss indicate network saturation. Saturate the network with large payloads or many concurrent transfers. This is rare on modern cloud infrastructure but common in hybrid setups with VPN tunnels or cross-region traffic.

Cache hit rate

When cache miss rate climbs, backend load increases. Increase read diversity until the cache no longer absorbs load. A cache that handles 90% of reads at 100 RPS might handle only 60% at 1,000 RPS because the working set no longer fits — the backend takes the hit, and the bottleneck shifts.

Scalability Testing

Verify that adding resources actually increases capacity. This sounds obvious, but I have seen teams double their server count and get only a 30% capacity increase because the database was the shared bottleneck.

Start with a baseline: 2 servers handle 1,000 RPS with P99 under 200 ms. Double to 4 servers and confirm capacity increases to approximately 2,000 RPS. Double again to 8 servers and check for approximately 4,000 RPS. If capacity does not scale linearly, the bottleneck is something shared — usually the database, a shared cache layer, or the network. I once traced a scaling plateau to a single Redis instance that was handling all cache reads; adding more application servers just moved the queue from the app to Redis.

Measure the cost per request at each scale point. Scaling linearly is useless if each new node costs more than the revenue it enables — I track cost per request alongside capacity to make sure the scaling curve is economically viable.

Recovery Testing

After hitting the breaking point, measure how quickly the system recovers. This is the test that tells you whether a traffic surge will cause a permanent outage or a temporary degradation.

Stop the load generator and measure the time for response times to return to baseline. On one test, I found that the system recovered in 30 seconds on paper but took 8 minutes in practice because the auto-scaler had spun up extra nodes that took time to drain and terminate. Verify no residual errors such as stuck threads or leaked connections — I once found a connection pool that never released connections after a saturation event, which meant the system was fine until the next traffic spike.

Check that auto-scaling down-scales when load drops, and confirm that data consistency is maintained. A capacity test that corrupts data is worse than no test at all — I always run a consistency check on the database after recovery testing.

Capacity Planning Outputs

After testing, produce a capacity report with the following example values from a hypothetical e-commerce checkout service:

Metric	Value	Notes
Baseline capacity	500 concurrent users / 250 RPS	Current normal load at 40% CPU
Safe operating limit	1,000 concurrent users / 500 RPS	Headroom before alerts fire
Hard limit	2,500 concurrent users / 1,200 RPS	Database connection pool exhausts here
First bottleneck	Database connection pool	Primary constraint observed at 2,500 users
Scaling factor	1.8x per added node	Not perfectly linear due to shared cache
Recovery time	2 minutes	After load stops

These numbers are illustrative. Your actual values depend on the application, infrastructure and dataset.

Best Practices

1. Test with production-like data. I once ran a capacity test against an empty database and got 5,000 RPS. The same test against a production-sized dataset dropped to 800 RPS because the query planner chose a different execution plan. Empty databases and small payloads give optimistic results — use realistic data volumes and distributions.
2. Monitor all layers, not just the app. On one test, the application metrics looked fine but the database server was at 100% CPU. The bottleneck was a missing index that only surfaced under load. Monitor the database, cache, load balancer, network and storage to find the true bottleneck — application metrics alone will lie to you.
3. Test the full request path. A test that hits only the API gateway misses bottlenecks in downstream services, message queues and third-party integrations. I caught a payment provider timeout at 300 concurrent users that would have caused a 30% checkout failure on launch day — the gateway was fine, the downstream call was the problem.
4. Document and share results. Capacity test results should be accessible to engineering, product and finance teams. I publish a one-page summary with the hard limit, the first bottleneck and the scaling factor — that is what finance needs for budgeting and what product needs for SLA commitments.
5. Re-test after major changes. A new feature, dependency upgrade or architectural change can shift the bottleneck. I re-run capacity tests quarterly or after notable releases — on one occasion a seemingly harmless caching change moved the bottleneck from the database to the network, and we only found out because we re-tested.
6. Use pass/fail thresholds in the test script. I add k6 v0.50 thresholds to every capacity test so CI can reject builds where P99 latency or error rate exceeds the capacity target. Thresholds turn a one-time test into a continuous guard.
7. Include a sustained soak phase. Some issues, such as memory leaks or connection pool exhaustion, only appear after hours of continuous load. I once found a connection leak at the 4-hour mark that would have caused a production outage during a sustained traffic surge — the first 3 hours looked clean.

Common Mistakes

1. Testing in a smaller environment. I learned this the hard way: a single-node staging environment predicted 3,000 RPS, but the multi-node production cluster only reached 1,800 RPS because of database lock contention that only appears with concurrent writers. Network, database and caching behavior change at scale — always test in an environment that mirrors production topology.
2. Using unrealistic request mixes. A test that is 100% reads will hide write-path bottlenecks. I once ran a capacity test with a 90/10 read/write ratio and missed a write contention issue that caused a 40% drop in capacity when the real production mix was 60/40. Match the production read/write ratio.
3. Ignoring warm-up time. Cold caches, JIT compilation and connection pool initialization cause initial slowness. I saw a team reject a build because P99 was 2 seconds during the first 30 seconds of the test — it dropped to 200ms after warm-up. Run a warm-up phase before measuring.
4. Stopping at the first error. The first error is rarely the root cause. On one test, the first error was a timeout at 400 concurrent users, but the real limit was the database connection pool at 600 users. If I had stopped at the first error, I would have reported the wrong bottleneck.
5. Not testing sustained load. Some issues, such as memory leaks, log disk fill and connection pool exhaustion, only appear after hours of continuous load. I found a log disk fill at the 6-hour mark that would have caused a production outage during a sustained traffic surge — the first 5 hours looked clean.
6. Forgetting to scale the load generator. If the generator itself is the bottleneck, you will underestimate system capacity. I once spent two days investigating a "system bottleneck" that was actually the load generator running out of CPU — always monitor the generator's resource usage too.

Frequently Asked Questions

What is the difference between load testing and capacity testing?

Load testing validates that the system behaves correctly at expected load — "does it work at 1,000 users?" Capacity testing pushes beyond expected load to find the breaking point — "at how many users does it stop working, and what fails first?" I run load tests before every release and capacity tests before every major traffic event.

How do I know when the test has reached the real limit?

Watch for sustained error rates above your threshold, P99 latency that keeps growing, throughput that stops increasing, or a resource such as CPU or database connections reaching 100% utilization. The real limit is not the first error — it is the point where adding more load does not increase throughput. I learned to look for throughput plateau, not error rate, because some systems degrade gracefully before they fail.

Should I run capacity tests in production?

Usually no. Run them in a staging environment that mirrors production hardware, data volume and network topology. For very large systems, a small production canary test may be possible, but it requires careful traffic steering and rollback plans. I once ran a canary capacity test in production on a Sunday at 3 AM — it worked, but I would not recommend it unless you have no other option.

What tools should I use for capacity testing?

k6 v0.50, JMeter 5.6 and Gatling 3.10 are the most common open-source options. k6 is scriptable in JavaScript and CI-friendly, JMeter 5.6 has a mature GUI and protocol support, and Gatling 3.10 is strong for high-concurrency simulations. I use k6 for most capacity tests because the thresholds integrate directly with CI — JMeter is better when I need to test obscure protocols.

How often should I repeat capacity tests?

Re-run after major releases, infrastructure changes or at least quarterly. Traffic patterns, data volumes and code change, so capacity limits change too. I once skipped capacity testing for two quarters and discovered the hard limit had dropped by 35% because of a new feature that added an unindexed query — the test caught it before the next traffic surge.

Summary and Next Steps

Capacity testing is not a one-time exercise. The breaking point you find today will shift with the next release, the next infrastructure change, or the next traffic surge. Here is what I do after every capacity test:

1. Write down the first bottleneck and the hard limit. These two numbers are the ones you will reference in every capacity conversation with finance and engineering. Everything else is detail.
2. Add the k6 v0.50 threshold script to CI. If the capacity drops below the threshold on a future build, CI fails before the regression reaches production. I treat capacity thresholds the same way I treat unit test thresholds — they are gates, not suggestions.
3. File tickets for the bottlenecks you found. A bottleneck you found but did not fix will bite you on the next traffic surge. File the ticket while the evidence is fresh.
4. Schedule the next test. Quarterly or after major releases — whichever comes first. Capacity decays as code and data grow.
5. Share the capacity report with finance and product. They need the numbers to budget for infrastructure and to promise SLAs they can keep.

Related Resources

• Performance Testing Guide
• API Load Testing with Postman & k6
• Failover Testing Guide
• Test Exit Report Template
• Performance Testing Topic

References

• Gatling Enterprise Documentation
• k6 Documentation
• Apache JMeter
• Google SRE Book — Capacity Planning