For quite a while, my VPS worked exactly the way I wanted it to.

It handled real production traffic. Nginx behaved normally. The backend behaved normally. I was not watching the server every five minutes or looking for reasons to blame the hosting provider.

Then requests started slowing down.

Some eventually timed out completely.

The strange part was that the VPS did not look dead.

It was online.

Nginx was running.

The application was running.

Ping worked.

Sometimes I could SSH into the machine, check a few basic metrics, and see nothing dramatic.

Then I managed to catch the server during one of the actual failures.

That changed everything.

CPU iowait:       97-100%
I/O PSI full:     ~95-98%
read latency:     up to 18.7 seconds
flush latency:    up to 53.6 seconds
I/O queue depth:  128+

Those are seconds, not milliseconds.

At the worst point I observed ordinary reads taking up to 18.7 seconds and flushes taking more than 53 seconds.

The VPS was technically online.

The workload was barely moving.

This article is about how I traced that failure from the application layer down through Nginx, EXT4, Linux pressure metrics, D-state processes and the virtual block device.

It is also about where the investigation had to stop, because inside a VPS there is a point beyond which the guest simply cannot see.

And yes, eventually I cancelled the server.

FDCServers refunded me.

But this is not an article claiming that every FDCServers VPS behaves like this.

It is the story of one production VPS and one extremely interesting intermittent storage problem.

The first problem: the server looked healthy when it wasn't

Like most people, I started with the obvious checks:

top
free -h
df -h
systemctl status nginx

And this is where intermittent failures become dangerous.

During healthy periods, the machine genuinely looked healthy.

For example:

D-state processes: 0
CPU iowait:        ~0%
disk latency:      ~2-4 ms

I/O PSI some:      0.04
I/O PSI full:      0.04

Nothing there screams disaster.

If support had inspected the VPS at exactly that moment, they could honestly have told me:

Everything looks normal.

And they would have been right.

At that moment.

That became one of the most important lessons from the whole incident:

Never investigate an intermittent performance problem only after it has disappeared.

The useful evidence exists while the failure is happening.

Not after a reboot.

Not ten minutes later.

Not after load disappears.

During the failure.

The iostat output that made this impossible to ignore

Eventually I captured this:

r_await = 18744 ms
f_await = 53561 ms
aqu-sz  = 128.54
util    ≈ 100%
read    = 88 KB/s

That was the moment the investigation stopped being vague.

Roughly translated:

read latency    ≈ 18.7 seconds
flush latency   ≈ 53.6 seconds
queue depth     > 128
utilization     ≈ 100%
read throughput ≈ 88 KB/s

A device being 100% utilized does not automatically mean there is a problem.

A healthy storage device can be busy.

But the rest of the picture mattered:

• enormous latency

• queue depth above 128

• almost no useful read throughput

• blocked processes

• failing HTTP requests

The virtual block device was extremely busy while accomplishing very little.

That is a very different situation from simply having a busy disk.

Why I didn't stop at iowait

During the worst periods I also saw:

blocked processes: 4-9
CPU iowait:        97-100%
CPU idle:          0%

It would be easy to look at that and say:

The CPU is spending 100% of its time waiting for the disk.

But I do not like using iowait as a complete diagnosis.

Linux CPU accounting is more nuanced than that.

iowait is useful evidence, but it is still evidence from one angle.

So instead of treating it as the answer, I started looking for independent signals.

The strongest one turned out to be PSI.

PSI showed that the workload itself was being stalled

Linux exposes Pressure Stall Information through /proc/pressure.

For I/O:

cat /proc/pressure/io

During one of the severe incidents:

some avg10=99.14
full avg10=95.55

Later, full approached 98%.

This was one of the clearest signals in the entire investigation.

In simplified terms:

• some means at least some non-idle work is stalled on I/O

• full means all non-idle tasks are stalled at the same time

When full approaches 100%, the system may still technically be alive, but almost every opportunity for useful workload progress is being lost to I/O stalls.

That described my VPS almost perfectly.

The machine existed.

The processes existed.

But useful work was barely being completed.

D-state showed that this was bigger than one process

Next I looked for tasks stuck in uninterruptible sleep:

ps -eo state,pid,ppid,etime,wchan:50,comm,args \
  | awk 'NR==1 || $1 ~ /^D/'

During the failures I saw several unrelated components entering D-state:

jbd2
systemd-journald
Nginx workers
Nginx cache processes
other filesystem activity

One process entering D-state briefly is not enough to prove much.

But multiple unrelated processes blocking at the same time is different.

At that point, blaming one application worker no longer made much sense.

The affected processes shared something lower in the stack.

The filesystem and storage path became the obvious place to look next.

The kernel stacks showed exactly where Nginx was waiting

The blocked paths were revealing.

For jbd2 I saw paths involving:

wait_on_buffer
jbd2_log_wait_commit
jbd2_journal_commit_transaction

For Nginx:

folio_wait_bit_common
filemap_read
generic_file_read_iter
ext4_file_read_iter
vfs_read
pread64

Then the kernel produced this:

INFO: task nginx blocked for more than 122 seconds.

At the same time:

systemctl status nginx

still said:

active (running)

Both were true.

That distinction is worth remembering.

active (running) tells you the service has not exited.

It does not tell you the service is capable of completing useful work.

A process can exist.

A service manager can consider it healthy enough to remain active.

And users can still be staring at timeouts.

The most useful test took about three milliseconds

The cleanest experiment was not complicated.

I tested the normal public path through Nginx.

It failed:

HTTP=000
SSL connection timeout

Then I bypassed Nginx and called the backend directly on the same VPS.

The result:

connect = 0.000423 s
TTFB    = 0.003063 s
total   = 0.003139 s

About three milliseconds.

That was extremely useful.

The backend could:

• accept the connection

• execute the request

• generate a response

• return it almost immediately

At the same time, Nginx workers were blocked in filesystem reads.

So the failure boundary started looking like this:

application execution       -> progressing normally
filesystem-backed web path  -> not progressing normally

That did not reveal the physical root cause.

But it made the application itself a much less convincing suspect.

The application was doing its job.

Something lower in the filesystem and virtual storage path was not.

The VPS could go from healthy to unusable in about 30 seconds

The intermittent nature of the problem made it particularly annoying.

At one point I captured the VPS in this state:

HTTP:          200
D-state:       0
CPU iowait:    3%
r_await:       ~1.18 ms
I/O PSI full:  ~2.95%

Nothing alarming there.

Then roughly 30 seconds later:

D-state:       4
CPU iowait:    91%
CPU idle:      0%
I/O PSI some:  86.11%
I/O PSI full:  78.02%
r_await:       236.50 ms
HTTP:          000

And later:

CPU iowait:    96-100%
I/O PSI full:  ~98%
HTTPS queue:   512
HTTP:          000

Then the workload disappeared.

The machine recovered:

D-state:      0
CPU iowait:   6%
r_await:      ~0.98 ms
queue depth:  ~0.07
HTTP:         200

This explains why intermittent infrastructure failures can become so frustrating between a customer and a hosting provider.

I could show a VPS that was almost completely stalled.

Someone else could inspect the same VPS ten minutes later and see sub-millisecond storage latency.

Both observations could be completely accurate.

They were simply observing two different states.

Sometimes r_await = 0 tells you almost nothing

One strange moment in the investigation was seeing:

r_await = 0

while the VPS was obviously unhealthy.

At the same time there was:

• extreme iowait

• D-state processes

• outstanding I/O

• failing requests

• almost no useful read throughput

So had the storage suddenly become perfect?

No.

The problem was that almost nothing was completing.

Metrics based on completed operations become less informative when the system reaches a state where operations barely complete at all.

That is why I stopped trusting any individual storage metric.

I wanted all of them together:

latency
IOPS
throughput
queue depth
in-flight I/O
D-state tasks
PSI
request completion

One metric can mislead you.

A group of independent metrics telling the same story is much harder to ignore.

The first major incident happened during a backup

This was where FDCServers entered the investigation more directly.

One of the first severe failures overlapped with a scheduled backup.

FDCServers confirmed that the backup was running.

That was a reasonable explanation candidate.

If the sequence had been:

backup begins
I/O stalls
backup ends
problem disappears forever

I probably would not be writing this article.

But the same class of storage stall later appeared outside that original backup window.

And it returned on multiple days.

So the backup could have explained one event.

It could not explain the full pattern I was observing.

One incident lasted more than four hours

During one incident, my own service logs later showed the VPS unavailable for:

4 hours, 41 minutes and 15 seconds

I need to be careful with that number.

I cannot prove that the storage stall caused that exact VPS state.

To prove that, I would need access to host-side information that I simply did not have.

And that gets to the main limitation of diagnosing infrastructure from inside a virtual machine.

My visibility looked roughly like this:

application
    ↓
Linux VFS
    ↓
EXT4
    ↓
virtual block device
    ↓
?

Everything behind that question mark belonged to infrastructure outside the guest.

Possibilities could include:

• virtualization

• host-level I/O queues

• storage networking

• distributed storage

• physical media

• host scheduling

• other storage systems invisible to the VM

I could see what Linux inside the VPS was waiting for.

I could see where useful work stopped.

But I could not see the provider's entire storage architecture.

So I am not going to invent a root cause.

I cannot honestly say:

A physical SSD failed.

I cannot honestly say:

A specific storage node was broken.

I did not have that evidence.

FDCServers migrated my VPS to another node

Support continued investigating.

Eventually FDCServers migrated the VPS to another node.

That was a meaningful step.

If the problem had been isolated to something on the original host, moving the VM could have solved it.

At that point I hoped the investigation was finished.

Unfortunately, after the migration I captured another severe guest-side storage stall.

That does not prove that every FDCServers node had the same problem.

It does not prove that every event had an identical physical root cause.

And it certainly does not prove that the entire FDCServers infrastructure was unhealthy.

The conclusion I could support was much narrower:

From my perspective, the production problem affecting this VPS had not been eliminated.

That was enough for me.

At some point, debugging stops being fun and becomes an operational decision

I genuinely enjoyed this investigation.

That may sound strange considering it involved production problems, but this is exactly the kind of thing I like about Linux.

Linux turned:

My website randomly stops responding.

into something much more precise:

The application backend is healthy, while filesystem-dependent processes are simultaneously blocked below VFS/EXT4, the virtual block device is heavily queued, and almost the entire workload is stalled on I/O.

That is a useful technical result.

But a useful technical result does not automatically make production safe again.

I did not control:

• the hypervisor

• the physical host

• the storage backend

• the infrastructure behind the virtual disk

• any of the provider-side systems below my VPS

And after the same class of failure returned even after migration, continuing the investigation stopped making operational sense.

Production is not supposed to be an endless forensic experiment.

So I asked FDCServers to cancel the VPS.

They refunded me.

I think that part matters too.

Is this a negative review of FDCServers?

For this particular VPS, yes.

The experience became bad enough that I no longer trusted the server with production traffic.

That is why I cancelled it.

But I do not think the evidence allows me to turn that into:

FDCServers is a bad hosting provider.

That is a much bigger claim.

My actual experience was mixed.

Things I appreciated:

• the VPS worked normally before these incidents

• support investigated the issue

• they attempted to solve it with a node migration

• they refunded me when I eventually left

Things I could not accept:

• severe intermittent storage stalls

• production requests timing out

• reads reaching 18.7 seconds

• flush latency reaching 53.6 seconds

• the issue returning on multiple days

• another severe stall after migration

So the strongest conclusion I am comfortable making is this:

One FDCServers VPS became unreliable enough for my production workload that I cancelled it.

That is what my evidence supports.

I have not retested their current infrastructure.

I cannot tell you how their VPS platform performs today.

Infrastructure changes constantly.

Hardware gets replaced.

Storage gets rebalanced.

Customers move.

Software gets updated.

Problems disappear.

Different problems appear.

A single past incident should not automatically become a permanent verdict about an entire provider.

What I actually proved

From inside the VPS, I had evidence for all of this:

read latency:       up to 18.7 s
flush latency:      up to 53.6 s
I/O PSI full:       almost 100%
CPU iowait:         almost 100%
I/O queue:          128+
Nginx:              blocked in filesystem reads
EXT4/jbd2:          blocked waiting for I/O
direct backend:     ~3 ms
HTTP through Nginx: timing out

I could demonstrate that:

1. the application backend could still respond quickly

2. filesystem-dependent processes were blocking

3. unrelated system components were being affected simultaneously

4. Linux showed extreme I/O pressure

5. the virtual block path showed severe queueing and latency

6. useful workload progress almost stopped

That was enough to locate the failure boundary.

It was enough to stop blaming the application.

And it was enough to make an operational decision.

What I could not prove

I could not prove:

• which physical disk handled the I/O

• whether any physical disk had failed

• whether the underlying problem was host contention

• whether distributed storage was involved

• whether storage networking was involved

• which exact provider-side component caused the stall

• whether every incident shared exactly the same host-side root cause

I think this distinction matters.

When writing an incident report, I try to separate three things:

what I observed

what I suspect

what I cannot know from my layer

Those are not the same thing.

The Linux debugging workflow I use now

This incident changed how I investigate slow or mysteriously unresponsive Linux servers.

I still begin with the basics:

top
free -h
df -h

But if something feels wrong, I quickly collect more useful evidence:

date -u
uptime

cat /proc/pressure/io
cat /proc/pressure/memory
cat /proc/pressure/cpu

vmstat 1 10
iostat -x 1 10

ps -eo state,pid,ppid,etime,wchan:50,comm,args \
  | awk 'NR==1 || $1 ~ /^D/'

ss -lntp

journalctl -k --since "30 min ago" --no-pager

Then I try to isolate the request path:

public request
      ↓
reverse proxy
      ↓
direct backend
      ↓
filesystem
      ↓
block device

Instead of asking:

Why is the server slow?

I now prefer:

At which layer does useful work stop completing?

That question gives you somewhere concrete to investigate.

Capture evidence before you reboot

There is one final habit I took away from this incident.

Sometimes rebooting is absolutely the correct production response.

Availability matters.

But rebooting also destroys evidence.

Before rebooting you might have:

D-state:     high
I/O PSI:     ~97%
iowait:      ~100%
queues:      large
requests:    failing

After reboot:

D-state:     0
latency:     milliseconds
requests:    healthy

Great.

The service is back.

But the state you needed to investigate has disappeared.

If the business impact allows you a few minutes, capture:

UTC timestamp
PSI
vmstat
iostat
D-state + wchan
kernel messages
socket queues
request timings

Then recover the server.

A few minutes of evidence can be worth hours of investigation later.

The server was running. The workload wasn't.

That is the sentence that best summarizes the incident.

The VPS remained online.

Nginx remained running.

The backend process remained alive.

And during the worst periods Linux still showed:

read latency:       18.7 seconds
flush latency:      53.6 seconds
I/O PSI full:       ~98%
CPU iowait:         ~100%
queue depth:        128+

The application itself could respond in about three milliseconds.

But the filesystem-backed request path was barely progressing.

Eventually the story became:

FDCServers investigated
        ↓
VPS migrated
        ↓
severe storage stall appeared again
        ↓
I cancelled the VPS
        ↓
FDCServers refunded me

I never learned which exact host-side component caused the failure.

And I do not need to pretend that I did.

The guest-side evidence was enough to identify where progress stopped.

Enough to stop blaming my application.

Enough to communicate the problem.

Enough to decide that production should run somewhere else.

That is what good observability is supposed to give you.

A VM can be online.

A process can be running.

A service can show active (running).

Ping can work.

And the machine can still be accomplishing almost nothing.

Linux will usually give you the clues.

The difficult part is catching them while the failure is still happening.

This is not a negative review of AVA Hosting.

It is simply a record of what happened to one VPS I used in production, what Linux reported inside the VM, and why those measurements made me cancel the server.

The VPS was small:

1 vCPU
2 GB RAM
25 GB NVMe
KVM

Nothing unusual.

Nginx was running. The backend was running. The machine had available RAM. The disk was not full.

But the server could not keep up with the workload.

So instead of guessing, I measured it.

The number that changed the diagnosis

I monitored the VPS for 60 seconds under its normal production traffic.

No CPU stress test.

No synthetic HTTP flood.

No disk benchmark.

The average CPU statistics were:

CPU user:     52.52%
CPU system:    8.07%
CPU softirq:   6.69%
CPU steal:    32.73%
CPU iowait:    0.00%
CPU idle:      0.00%

Maximum runnable queue: 11

The important line was:

CPU steal: 32.73%

At that point, this no longer looked like a simple case of “my application needs a bigger server.”

What CPU steal actually tells you

steal is a standard Linux CPU metric for virtualized environments.

Conceptually, it describes a situation like this:

the guest has runnable work
        ↓
the vCPU wants to execute it
        ↓
the vCPU is not actually running
        ↓
Linux accounts that time as steal

KVM explicitly supports steal-time accounting for virtual CPUs.

This distinction matters.

If my measurements had looked like this:

user/system: ~100%
steal:          0%
idle:           0%

I would have had a straightforward capacity problem.

The CPU would be available; my workload would simply be consuming all of it.

The solution could then be optimization or more vCPUs.

But that was not what I measured.

Roughly one third of the observed CPU time was being reported as steal.

It was sustained, not a random spike

A single ugly sample would not have convinced me.

Virtual machines have scheduler noise. Short spikes happen.

But the per-second measurements repeatedly looked like this:

%usr    %sys   %soft   %steal   %idle

53.54   10.10   5.05    31.31     0.00
56.44    6.93   5.94    30.69     0.00
51.52    7.07   6.06    35.35     0.00
55.45    6.93   5.94    31.68     0.00
51.49    8.91   5.94    33.66     0.00
52.53    7.07   7.07    33.33     0.00
52.48    6.93   7.92    32.67     0.00

The pattern was still there near the end of the observation:

50.98   10.78   6.86    31.37     0.00
48.48    8.08   9.09    34.34     0.00
55.00    5.00   7.00    33.00     0.00

So this was not one strange second.

For most of the observation, CPU steal stayed around 30% or higher while CPU idle remained at zero.

For intuition:

60 × 0.3273 ≈ 19.64 seconds

That does not mean the VPS froze for one continuous 19.64-second block.

It means the smaller intervals in which the vCPU was not scheduled accumulated to roughly that amount during one minute.

The application symptoms matched the CPU metrics

The backend was alive, but being alive is not the same as keeping up.

Before the observation, its listening queue showed:

Recv-Q: 168
Send-Q: 511

After 60 seconds:

Recv-Q: 166
Send-Q: 511

The exact difference between 168 and 166 is not interesting.

What matters is that the queue essentially did not drain.

The backend had an entire minute to catch up and remained behind.

There were also many CLOSE-WAIT connections. I would not use those alone as evidence of a hosting problem because application-side connection handling can produce them as well.

But taken together, the picture was difficult to ignore:

1 vCPU
32.73% CPU steal
0% CPU idle
~99% CPU pressure
runnable queue up to 11
backend queue not draining

The server was technically running.

It was not healthy.

I checked RAM and disk before blaming the host

High load by itself proves very little.

I wanted to make sure I was not blaming virtualization for a problem inside my own system.

Memory looked like this:

RAM total:      ~1.9 GiB
RAM available:  ~959 MiB

Swap total:     2 GiB
Swap used:      ~33 MiB

There was still substantial available memory.

The root filesystem was:

24 GB total
14 GB used
9.4 GB available
60% used

And the average CPU I/O wait was:

0.00%

So the obvious story was not:

RAM exhausted

or:

disk full

or:

CPU blocked waiting for storage

Meanwhile, Linux CPU Pressure Stall Information stayed close to 99%, and the one-vCPU VM had a load average around 3.5–4.

The machine had work waiting for CPU almost continuously.

Then I checked what AVA Hosting actually promises

This is where the case became more interesting.

AVA's current VPS page says:

“Guaranteed resources — no sharing”

It also says that every VPS receives guaranteed vCPU resources and that performance is never affected by other customers.

Its current Linux VPS page is even more specific. It says that a CPU-intensive workload from another tenant cannot introduce steal time into your instance and describes the vCPU allocation as dedicated.

AVA's unmanaged Linux VPS page currently says:

“CPU steal time is eliminated at the hypervisor level”

and describes CPU resources as a fixed allocation rather than a shared or burstable pool affected by noisy-neighbour contention.

Those claims are unusually specific.

This was not simply a VPS provider saying:

1 vCPU

without explaining how CPU resources are managed.

AVA was explicitly making claims about the exact phenomenon I was measuring.

And my VPS showed:

Average CPU steal: 32.73%

What I can prove — and what I cannot

This distinction matters.

I can say that:

• my AVA Hosting VPS had one vCPU;

• under normal production load it averaged 32.73% CPU steal;

• CPU idle averaged 0%;

• CPU pressure was close to 99%;

• the backend queue was not draining;

• RAM was available;

• the filesystem was only 60% used;

• average CPU I/O wait was 0%.

I cannot see AVA's physical host from inside the VM.

So I cannot prove:

• the number of neighbouring VMs;

• the host CPU utilization;

• AVA's actual CPU overcommit ratio;

• its CPU pinning configuration;

• which workload was consuming physical CPU;

• or whether the condition was intentional.

And I am not claiming that every VPS operated by AVA Hosting behaves this way.

I tested one VPS.

But I do think one conclusion is reasonable:

The behaviour I measured was very difficult to reconcile with AVA's specific claims about guaranteed CPU allocation and the absence of CPU steal from neighbouring workloads.

I do not need to make a stronger accusation than that.

One important caveat on AVA's own site

AVA's main VPS FAQ also says that VPS resources are not as strictly guaranteed as resources on a dedicated server.

That is a reasonable virtualization caveat.

A VPS is not bare metal, and 1 vCPU by itself does not mean that one entire physical CPU core belongs exclusively to one customer.

But that general caveat appears alongside much more specific statements elsewhere on AVA's site:

Guaranteed resources — no sharing

Your performance is never affected by other customers

Dedicated vCPU allocation

A co-hosted tenant cannot introduce steal time

CPU steal time is eliminated at the hypervisor level

That is why the 32.73% figure mattered to me.

Why I did not simply upgrade

The obvious fix for a saturated one-vCPU machine is to buy more CPU.

Normally, that makes sense.

But I wanted to distinguish two cases.

Case 1: I actually consumed the CPU I bought

CPU usage: high
steal:     ~0%
idle:      0%

Then the resource is being delivered and the workload needs more capacity.

Upgrade.

Case 2: a large percentage is steal

My server looked more like this:

CPU steal: 32.73%
CPU idle:   0.00%

Buying another vCPU might have improved total throughput.

But before paying for additional virtual CPUs, I wanted the first one to behave the way I expected from the product description.

So I did not upgrade.

I cancelled the VPS and requested a full refund.

AVA refunded me quickly

This part of the experience was good.

AVA Hosting returned the full amount.

They handled the refund quickly and did not spend days arguing with me about the CPU measurements.

I explained the problem, requested my money back, and they refunded it.

I appreciate that.

So my experience has two separate conclusions:

The VPS I received had a serious CPU-availability problem.

and:

AVA handled the refund properly.

Both can be true.

What I check on a new VPS now

This incident changed one part of my VPS deployment routine.

Specifications are still useful:

vCPU
RAM
disk
network
virtualization

But they are not measurements.

For a new Linux VPS, I now want to see what happens under the actual workload.

My first checks include:

mpstat 1 60
vmstat 1 60
cat /proc/pressure/cpu
ss -ltnp
free -h
df -h
iostat -xz 1 60

I care about the relationship between:

%steal
%idle
CPU pressure
run queue
application queues
iowait
available memory

because different combinations point to very different problems.

High CPU usage with almost zero steal?

My workload may simply need more CPU.

High iowait?

Look at storage.

Memory pressure and heavy swapping?

Look at RAM.

High steal, no idle CPU, continuous CPU pressure and an application queue that cannot drain?

Now I start looking beyond the application.

The actual lesson

The main lesson from this incident is not:

One vCPU is too small.

It is not:

Every AVA Hosting VPS has this problem.

And it is not:

A high steal value tells me exactly how the provider configured its hypervisor.

The lesson is much simpler.

A VPS specification tells me what the virtual machine has been provisioned to see.

Linux tells me what the workload is actually experiencing.

In my case, the important numbers were:

CPU steal:    32.73%
CPU idle:      0.00%
CPU pressure: ~99%

AVA Hosting currently advertises guaranteed CPU resources without sharing and explicitly says neighbouring workloads should not introduce CPU steal.

My VPS reported otherwise.

I cancelled it.

AVA returned all of my money quickly and without an argument.

And that was the end of that debugging day.

I am not writing this as a negative review of REGXA, and I am not trying to tell anyone whether they should or should not buy a VPS from them.

This is simply one of my days as a developer.

I moved a normal workload to a KVM VPS with:

2 vCPU
2 GB RAM
60 GB NVMe

Nginx was running. The backend was running. The server was reachable.

But everything behaved as if the machine were massively overloaded.

Requests accumulated. TLS operations became slow. Connections stayed open for far too long. Some requests eventually returned HTTP 504 after more than two minutes.

My first assumption was obvious:

something inside my VPS was consuming all the CPU.

Then I ran mpstat.

Average CPU steal:   92.58%

CPU 0 steal:         90.62%
CPU 1 steal:         94.57%

CPU user:             4.02%
CPU system:           1.47%
CPU iowait:           0.41%
CPU idle:             0.53%

That changed the investigation completely.

My applications were not consuming 90% of the processor.

They were barely getting processor time at all.

CPU usage and CPU steal are not the same thing

This distinction is what made the incident interesting.

If my application were actually saturating the CPUs, I would expect something like:

user:    80%
system:  10%
steal:    0%

That means the virtual machine is receiving CPU time and spending it on my code.

Instead, my VPS looked roughly like this:

user:     4%
system:   1%
steal:   93%

The Linux mpstat documentation defines %steal as time spent in involuntary wait by a virtual CPU while the hypervisor is servicing another virtual processor.

In practical terms, the guest has work ready to execute, but the hypervisor is not giving that vCPU execution time.

That is what I mean when I say CPU time was being taken away from the VPS.

I am not claiming that somebody at REGXA deliberately took two physical cores assigned exclusively to me and handed them to another customer. Guest metrics cannot prove that, and they cannot prove intent.

What they can show is much narrower:

my virtual CPUs wanted to execute work and repeatedly were not being scheduled.

Individual samples were repeatedly in the same range:

89.85% steal
92.16% steal
96.48% steal
94.87% steal
95.69% steal
95.95% steal
94.57% steal
93.75% steal

Linux Pressure Stall Information pointed in the same direction:

CPU PSI some avg10:   79.47
CPU PSI some avg60:   75.32
CPU PSI some avg300:  76.31

Load average:
5.85 / 5.75 / 5.73

Linux PSI measures periods in which tasks are stalled waiting on resources; for CPU pressure, the some metric represents time when at least some runnable work is stalled on CPU availability.

On a two-vCPU VM, these were not numbers I wanted to explain away as ordinary application load.

So I tested the obvious alternative explanation.

I removed all production traffic

Maybe my workload itself was somehow causing the problem.

Fair enough.

I moved the active workload to another server.

Production traffic stopped reaching the affected VPS.

The connection queues cleared.

Then I repeated the CPU measurement.

This time I got:

Average CPU steal:   94.17%

CPU 0 steal:         95.56%
CPU 1 steal:         92.83%

CPU user:             1.95%
CPU system:           0.59%
CPU iowait:           0.59%
CPU idle:             2.18%

The result became worse.

My applications were now barely doing anything:

user:    1.95%
system:  0.59%

But:

steal:  94.17%

Individual vmstat samples still showed:

91% steal
98% steal
98% steal
97% steal
97% steal
95% steal
96% steal
95% steal

At points, as many as 15 runnable processes were waiting for CPU.

For me, this was the strongest test of the entire incident.

I had removed the production workload.

The applications were consuming almost no processor time.

And the VM was still spending roughly 94% of measured CPU time in steal.

At that point, optimizing my backend was no longer a convincing answer.

Then I tested localhost

I wanted a second test that removed as much external infrastructure as possible.

So I tested HTTPS through:

127.0.0.1

That removes a lot of possible explanations:

• public DNS;

• my ISP;

• geographic distance;

• internet routing;

• transit providers;

• the network path between my computer and the data center.

The client and server are inside the same VM.

With production traffic still present, I ran ten localhost HTTPS requests.

Four out of ten failed during the TLS handshake.

The successful requests took:

29.30 s
22.77 s
12.25 s
11.87 s
11.12 s
 9.40 s

Some TLS handshakes alone took roughly nine seconds.

After I removed all production traffic, localhost improved, but it remained wildly inconsistent:

0.061 s
0.745 s
0.830 s
0.873 s
1.010 s
1.117 s
1.121 s
2.188 s
3.355 s

Think about that range.

The same local operation could take approximately:

61 ms

and then:

3.355 seconds

on another attempt.

There was no long-distance network between those two processes.

They were on the same machine.

The rest of the stack was showing the consequences

By this point, the application-level symptoms made more sense.

At one stage I saw approximately:

450 established connections
122 orphaned connections
110 FIN-WAIT-1
33 CLOSE-WAIT

The localhost backend listen queue reached roughly:

14–15

while the HTTPS queue reached roughly:

40

Nginx logged HTTP 504 responses with timings such as:

142.857 s
138.902 s
135.064 s
129.819 s
128.657 s

Other requests remained open for roughly 67–130 seconds.

I also saw:

secure-connect timeout: ~35 s
secure-connect timeout: ~41 s
database network timeouts
delayed TLS operations

Looking at those symptoms separately could have sent me in five different directions.

A 504?

Maybe Nginx timeouts.

Database errors?

Maybe MongoDB.

Slow TLS?

Maybe the network.

Growing connection queues?

Maybe backend concurrency.

But increasing an Nginx timeout does not create CPU time that the hypervisor is not scheduling.

It might simply turn a 60-second failure into a 120-second failure.

RAM and storage were not the explanation

I checked the obvious alternatives.

Memory:

Total RAM:       ~1.9 GiB
Available RAM:   ~1.0–1.1 GiB
Swap usage:      ~1–2 MiB

OOM events:      none
OOM killer:      none

Storage:

Filesystem:      ~58 GB
Used:            ~11 GB
Available:       ~44 GB
Usage:           ~20%

During the relevant CPU tests, I/O wait was generally below 1%.

So I did not have:

• an exhausted RAM situation;

• heavy swapping;

• an OOM problem;

• a full filesystem;

• enough I/O wait to explain the behaviour.

Meanwhile CPU steal remained above 90%.

A healthy KVM VPS looked completely different

I then ran the same kind of diagnostic on another KVM VPS.

That machine was actively serving normal traffic.

Its numbers:

Average CPU steal:   0.02%
CPU idle:           87.86%

CPU PSI avg10:       0.29
CPU PSI avg60:       0.63
CPU PSI avg300:      0.49

Load average:
0.47 / 0.33 / 0.14

Ten localhost HTTPS requests completed between roughly:

37 ms
and
69 ms

Later I measured another production VPS that really was using substantial CPU:

CPU user:     61.71%
CPU system:    5.08%
CPU idle:     24.09%
CPU steal:     0.13%

This became a useful mental model for me.

Busy because my code is running

user:   high
system: some
steal:  near zero

Busy because the VM is not getting scheduled

user:   low
system: low
steal:  extremely high

Both machines can feel slow to a user.

The cause is completely different.

Then REGXA confirmed resource contention

Until this point, all I had were measurements from inside the guest.

I could see what Linux was experiencing.

I could not see:

• the physical host;

• total host CPU utilization;

• scheduler weights;

• CPU quotas;

• neighbouring VMs;

• the provider's level of CPU overcommitment.

REGXA could.

Eventually, support told me that my VPS was running on:

shared CPU infrastructure

They explained that CPU resources were shared across multiple virtual machines and that performance could vary depending on load on the physical node.

More importantly, support explicitly attributed the elevated CPU steal I was observing to:

resource contention on the underlying infrastructure

They also said that the Frankfurt infrastructure was experiencing particularly high demand and that they could not change CPU quotas or scheduling policies or provide additional dedicated CPU resources for that shared VPS.

Their proposed technical solution was to move the VPS somewhere with lower utilization.

At that point, host-side contention was no longer merely my interpretation of Linux metrics.

It was also the explanation given by the provider.

The wording around CPU resources is what bothered me

There is one part of this experience I still find difficult to reconcile.

REGXA's current Frankfurt VPS page advertises:

Dedicated CPU Cores

for the infrastructure, while its System Policies describe fair CPU scheduling, CPU burst capability, throttling of excessive CPU usage, and dedicated CPU allocation for specific service tiers.

Support, meanwhile, described the VPS involved in my incident as running on shared CPU infrastructure whose performance depended on load on the physical node.

I am not making a legal claim about what the word “dedicated” must mean in every hosting product.

Nor do I know the exact physical-host configuration behind my instance.

But from a developer's point of view, these two facts are difficult to ignore:

CPU steal under workload:    92.58%
CPU steal with no traffic:   94.17%

And the provider itself later attributed those elevated values to resource contention.

That is enough for this case study.

Getting the refund became a second problem

Once the infrastructure problem was clear, I did not want to keep moving the VPS between locations and testing whether another node happened to be less busy.

I wanted to cancel it.

Initially, I was offered only a partial refund, and the proposed refund was to my REGXA account balance, not back to the card I had used to pay.

For me, provider credit is not equivalent to a refund if I am trying to stop using that provider.

So I continued the conversation.

I asked for:

100% of the payment

to be returned to:

the original payment method

I had to keep pushing for it.

Eventually, REGXA agreed and returned the full amount to the original payment method, describing the full refund as an exception.

I appreciate that they ultimately refunded everything.

But the fact that I had to keep pursuing the financial resolution after the infrastructure problem had already been acknowledged is part of my experience too.

The debugging mistake I almost made

The most valuable lesson here was not that one VPS performed badly.

Servers fail. Nodes become overloaded. Infrastructure has problems.

What matters to me as a developer is how easily I could have spent hours fixing the wrong layer.

I could have:

• increased Nginx timeouts;

• changed keepalive settings;

• lowered concurrency;

• added retries;

• investigated MongoDB;

• rewritten backend code;

• tuned connection pools.

Some of those changes might even have improved individual symptoms.

None of them would answer:

Why is the guest spending more than 90% of its CPU time waiting for the hypervisor?

That question lives below my application.

My VPS smoke test now includes %steal

When I provision a VPS, an HTTP 200 is no longer enough.

I want to see how the machine behaves while idle and under real load.

My basic checks now include:

mpstat -P ALL 1 20

vmstat 1 20

cat /proc/pressure/cpu

And for a web server, I want localhost latency too.

I look at:

%user
%system
%iowait
%idle
%steal
CPU PSI
run queue
localhost latency

Most importantly, when the numbers look suspicious:

I remove the workload and measure again.

In this incident, that one experiment produced the clearest result:

CPU user:     1.95%
CPU system:   0.59%
CPU steal:   94.17%

I am not claiming that every REGXA VPS behaves this way.

I tested one VPS and documented what happened to it.

But on that machine, the evidence was unusually clean: my applications were barely consuming CPU, Linux reported that the overwhelming majority of virtual CPU time was being lost to hypervisor scheduling, and REGXA later attributed that elevated steal to resource contention on its shared infrastructure.

That is why I am writing this.

Not as a rating.

Not as a recommendation.

Just as one of my days as a developer — and one Linux metric I will never ignore on a VPS again.

Browser
   ↓
Working server
   ↓
Nginx
   ↓
local proxy_cache
   ↓ cache MISS
Bunny Storage

The public server handled the application and HTTP delivery. Bunny Storage held the media. Nginx sat between them and cached remote objects on local SSD.

When a file was already cached, the system was fast.

Very fast.

The problem appeared when a file was not cached.

Old images could suddenly take much longer than popular ones. Some requests stalled before useful data started flowing. MP4 caching became increasingly complicated once byte ranges entered the picture. Nginx Slice introduced another class of failure involving inconsistent ETags.

Eventually I replaced Bunny Storage as the runtime origin with a separate private media server:

Browser
   ↓
Working server
   ↓
Nginx + local cache
   ↓ cache MISS
WireGuard
   ↓
Private media server
   ↓
Nginx
   ↓
SSD

The surprising part was not that a nearby private server could be fast.

The surprising part was how much better the entire system felt once a cache miss became cheap and predictable.

That changed how I think about caching.

I used to focus mainly on the hit ratio.

Now I care just as much about this question:

What happens when the object is not in cache?

This was Bunny Storage, not Bunny CDN

This distinction matters.

I was not benchmarking Bunny CDN.

Users were not fetching media from Bunny edge locations.

My Nginx server was fetching objects directly from Bunny Storage and then serving them itself.

The architecture was effectively:

User
   ↓
My server
   ↓
Bunny Storage

not:

User
   ↓
Bunny CDN
   ↓
Bunny Storage

Those are very different systems.

Bunny CDN is designed specifically for global content delivery, edge caching, and geographically distributed users. Bunny Storage is the storage layer.

I still consider Bunny a strong platform, especially when Storage is combined with its CDN.

This article is about a narrower case:

using Bunny Storage directly as the remote origin behind my own Nginx proxy cache.

That distinction is important because I am not comparing a private VPS with a full CDN.

I am comparing two origin architectures.

Why the original architecture made sense

The workload was mostly static media:

• AVIF images

• JPEG and PNG

• MP4

• WebM

• audio files

• many different object sizes

The files were effectively immutable after publishing.

I wanted the working server to control public delivery:

• URLs

• HTTP headers

• cache behavior

• 404 and 410 handling

• byte ranges

• video seeking

• fallback logic

• cache eviction

But I did not want all media permanently stored on that machine.

A reverse proxy cache seemed like a good compromise.

For a popular object:

request
↓
Nginx cache HIT
↓
local SSD
↓
response

Bunny Storage was completely removed from the request path after the first cache fill.

That worked exactly as intended.

The architecture looked good whenever I tested popular media.

That was also the trap.

A cache HIT can make a weak origin look excellent

Consider two files.

The first one is popular:

popular-image.avif
↓
requested frequently
↓
still in local cache
↓
HIT
↓
fast

The second one belongs to an old page:

old-image.avif
↓
not requested for a long time
↓
evicted from cache
↓
MISS
↓
remote origin required

If I test only the first object, I am testing my SSD.

I am not testing Bunny Storage.

The second request exposes the actual cold path.

This explained a strange production pattern I had noticed for a while:

Popular pages felt good.

Older pages sometimes felt dramatically worse.

A page could load most images immediately while one or two remained empty for a noticeable amount of time.

The error logs eventually explained why.

The request could stall before the image even started downloading

The important production errors were not primarily about reading a large file.

I repeatedly saw errors in the connection stage, including:

upstream timed out ... while SSL handshaking to upstream

That is a very specific failure.

The request flow looked more like this:

cache MISS
↓
resolve remote storage
↓
connect to upstream
↓
perform TLS handshake
↓
timeout

The browser was waiting for an image, but the server had not yet reached the useful object transfer stage.

This matched what I was seeing visually.

The problem was not always:

the image downloads slowly

Sometimes it was:

the image has not really started downloading yet

That distinction matters a lot for perceived performance.

Production logs showed that this was recurring

One diagnostic snapshot contained:

317

upstream timeout matches in the last:

5,000

lines of the media error log.

Another snapshot from the previous day contained:

578

matches.

These are log matches.

They are not unique users.

They are not unique files.

They are not a calculated failure percentage.

I do not want to turn them into a metric they do not represent.

What they do prove is much simpler:

The timeout pattern was recurring in production.

It was not one unlucky request observed during debugging.

Different storage addresses behaved very differently

During the investigation, the storage hostname resolved to addresses including:

109.61.89.53
109.61.89.54
109.61.89.55
109.61.89.57
79.127.226.193

Several measured paths were perfectly reasonable:

109.61.89.53     total ≈ 29.8 ms
109.61.89.54     total ≈ 28.5 ms
109.61.89.57     total ≈ 42.7 ms
79.127.226.193   total ≈ 43.6 ms

Then one measurement looked like this:

109.61.89.55

TCP    1.017782 s
TLS    1.048306 s
TOTAL  1.054474 s

This does not prove that Bunny Storage normally has one-second latency.

Clearly it did not.

Most measurements were tens of milliseconds.

What mattered was the variance.

The same storage hostname could lead to something like:

29 ms
30 ms
43 ms
1,054 ms

And the production logs showed worse cases that reached actual connection or TLS timeouts.

For a media-heavy page, that tail matters enormously.

Average latency can hide terrible page-level UX

Imagine a gallery loading 100 images.

Ninety-eight finish quickly.

Two do not.

The result might look like:

████████████
████████████
████████████

[ empty ]

████████████

[ still empty ]

████████████

The mean request time can still look acceptable.

The page does not feel acceptable.

This is why I stopped looking at media performance only through averages.

For this type of workload, I care more about:

MISS p95
MISS p99
upstream connect p95
upstream connect p99
slowest image on the page

A gallery is often judged by the slowest visible objects, not by the average object.

That is especially true when a user is scrolling quickly.

Why old media exposed the problem first

The local Nginx cache was large.

One July snapshot showed roughly:

cache size: ~36 GB
cache files: >355,000
configured cache limit: ~35 GB
root filesystem usage: ~93%

Nearby snapshots approached:

~400,000 cache entries

A finite cache cannot keep every object forever.

So the behavior naturally became:

popular media
↓
recently used
↓
likely cached
↓
fast

while old content behaved more like:

rare media
↓
not used recently
↓
more likely evicted
↓
cold MISS
↓
remote storage

The cache was hiding the origin problem on the most popular content.

Rare content revealed it.

That is why testing only the homepage or the most visited gallery would have given me a false sense of confidence.

Video exposed a second problem

Images revealed the latency problem.

Video revealed an architectural problem.

Browsers often do not download an MP4 from byte zero to the end.

They request ranges.

For example:

Range: bytes=0-1048575

A seek operation may later produce something like:

Range: bytes=50000000-51048575

A server can answer:

HTTP/1.1 206 Partial Content
Content-Range: bytes 50000000-51048575/500000000

This is normal HTTP byte-range behavior.

It is also essential for useful video seeking.

But it complicates caching.

The naive Range cache key creates fragmented objects

My early approach effectively included the requested browser Range in the cache identity:

proxy_set_header Range $http_range;
proxy_set_header If-Range $http_if_range;

proxy_cache_key "$scheme|$host|$request_uri|range=$http_range";

proxy_cache_valid 200 206 301 302 30d;

This protects against serving one cached partial response for a completely different range.

But browser ranges are arbitrary.

One client might request:

bytes=0-1048575

Another:

bytes=0-999999

Another:

bytes=50234881-51283456

So the cache can conceptually become:

video.mp4 + range A
video.mp4 + range B
video.mp4 + range C
video.mp4 + range D

instead of:

video.mp4

I did not measure exactly how much disk space this fragmentation consumed, so I am not going to invent a number.

But the architecture clearly allowed many cache entries for one physical video.

I wanted something more deterministic.

Nginx Slice looked like the correct answer

Nginx has a Slice module designed for caching large responses as fixed-size segments.

Instead of preserving arbitrary browser ranges:

0-734129
917283-1500000
2839123-3421091

I could normalize everything into predictable chunks:

0-1 MB
1-2 MB
2-3 MB
3-4 MB

The configuration looked roughly like this:

slice 1m;

proxy_set_header Range $slice_range;

proxy_cache_key "$scheme|$host|$uri|$slice_range";

proxy_cache_valid 200 206 30d;

This is much cleaner from a cache-reuse perspective.

If multiple users seek into the same region of a file, they can reuse the same normalized slices instead of generating arbitrary Range cache entries.

The design made sense.

Then production started producing:

etag mismatch in slice response while reading response header from upstream

That error changed the direction of the investigation.

What the ETag mismatch actually means

At first it is tempting to describe the pieces as having different "signatures."

That is too vague.

The relevant HTTP mechanism is the ETag.

An ETag is a validator associated with a representation of a resource.

Conceptually, if Nginx is assembling slices of one file, it expects something like:

slice 1
ETag: A

slice 2
ETag: A

slice 3
ETag: A

Now imagine this:

slice 1
ETag: A

slice 2
ETag: B

Nginx cannot safely assume those slices belong to the same representation.

If it ignored the mismatch, it could theoretically combine data from two different versions:

first part from version A
+
second part from version B

That could create corrupt output.

So rejecting inconsistent slices is the safe behavior.

The ETag problem happened in production

On July 30, repeated MP4 slice requests generated:

etag mismatch in slice response

while requests were reaching storage addresses including:

109.61.89.53
109.61.89.57
79.127.226.193
109.61.89.55

On July 31, the same class of error appeared again for another MP4.

The important conclusion is limited but solid:

Nginx received slice responses for the same MP4 whose validators were not consistent enough for safe slice assembly.

I did not log the literal ETag value returned by every individual subrequest at the time.

Because of that, I cannot honestly prove the exact reason for the difference.

Different upstream responses are a plausible factor.

They are not something I can claim as a fully proven internal Bunny failure mechanism.

The Nginx error itself is the confirmed fact.

Bunny Storage did support byte ranges

This is another distinction worth making.

I was receiving:

206 Partial Content

responses.

So the problem was not:

Bunny Storage cannot serve Range requests.

It could.

The narrower failure involved this combination:

Nginx Slice
+
multiple subrequests
+
ETag consistency requirements

At the same time, video requests could still encounter the independent origin issue:

upstream timed out while connecting to upstream

So video could suffer from two different classes of failure:

1. connection / TLS / long-tail origin latency
2. slice / validator consistency

That made the entire setup harder to reason about.

I eventually removed Slice from the MP4 path

I moved MP4 delivery back toward ordinary browser Range behavior.

Conceptually:

proxy_set_header Range $http_range;
proxy_set_header If-Range $http_if_range;

proxy_cache_key "$scheme|$host|$uri";

proxy_no_cache $http_range;

proxy_cache_valid 200 30d;

The exact production configuration evolved, but the principle was simple:

A partial response should not accidentally become the canonical complete cached object.

Removing remote slice assembly eliminated the ETag mismatch failure mode.

But it exposed the underlying problem again.

A cold range request still had to reach the remote storage origin.

I had improved the caching logic.

I had not simplified the origin.

At some point I realized I was recreating a delivery layer in front of storage

By then the configuration had accumulated a lot of moving parts:

proxy_cache
cache locking
background updates
stale content
custom cache keys
Range forwarding
206 handling
Nginx Slice
1 MB chunks
upstream keepalive
TLS session reuse
retries
connection timeouts

Every one of these features has a legitimate use.

But together they made me ask a more basic question:

What do I actually need from the origin?

For this workload, the answer was simple:

store immutable files
+
return bytes quickly

I did not need global edge distribution between two servers.

I did not need a complex public storage route.

I needed one working server to retrieve media from one origin server.

That led to a much simpler architecture.

The private origin architecture

The current model is:

Internet
   ↓
Working server
   ↓
Nginx
   │
   ├── HIT -> local SSD cache
   │
   └── MISS
          ↓
      WireGuard
          ↓
    Media origin
          ↓
        Nginx
          ↓
         SSD

The origin does almost nothing.

It stores static files and serves them.

The origin listener exists only on the private network.

The working server uses a fixed private address.

WireGuard provides the encrypted transport between the two systems.

Because the tunnel itself is encrypted, I can use ordinary HTTP inside it:

http://10.x.x.x:port

That removes another TLS handshake from the application-level origin path.

Static file origins are surprisingly cheap workloads

The media server does not render pages.

It does not run SSR.

It does not query a database.

It does not execute application business logic.

Its job is mostly:

open file
↓
send bytes

For a small dedicated origin, a machine in roughly this class can be enough for many workloads:

1 vCPU
1 GB RAM
hundreds of GB of SSD

The real constraints are usually more relevant than the amount of application compute:

• network capacity

• disk performance

• file descriptor limits

• concurrent connections

• filesystem space

• traffic shape

This is not a universal sizing recommendation.

A high-throughput video origin can obviously need much more.

But static media serving is fundamentally a much simpler workload than running an application stack.

Connection limits matter more than they appear

When I first moved traffic to the private origin, the server had to be configured like an actual media origin rather than a default low-traffic web server.

Important settings included:

worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
}

and a larger listener backlog:

listen 10.x.x.x:8080 backlog=8192;

At the kernel level:

net.core.somaxconn=8192
net.ipv4.tcp_max_syn_backlog=8192

The systemd file descriptor limit also needed to match:

[Service]
LimitNOFILE=65535

These settings are not magic numbers for every server.

The important point is consistency.

Raising worker_connections while leaving the real process nofile limit at 1024 does not produce the capacity you think it does.

The same applies to the socket backlog.

The application limit, process limit, and kernel limit need to agree.

After tuning, the origin became extremely boring

And boring is exactly what I wanted.

Local media-origin requests were roughly:

~0.5 to 0.9 ms

Small real-file requests from the working server through the private path were generally around:

CONNECT ≈ 9.5-12.5 ms
TTFB    ≈ 19-23 ms
TOTAL   ≈ 19-23 ms

I also checked sequential and parallel requests.

Results included:

10/10 successful
20/20 sequential successful
20/20 parallel successful

Connection pressure on the origin looked healthy:

SYN_RECV=0

and the server remained mostly idle while serving production traffic.

This does not mean a complete large media file downloads in 20 ms.

The measurement is about connection cost and time to first useful bytes.

A 500 MB video still needs enough throughput to move 500 MB.

Latency and bandwidth are different metrics.

The old 1-second result was not a benchmark, but it was still useful

The worst measured old path in that particular test was:

1.054474 seconds

The current small cold-origin path is around:

20 ms

That produces a ratio of roughly:

1054 / 20 ≈ 52

It would be wrong to write:

My VPS is 52 times faster than Bunny Storage.

That is not what the data proves.

The normal Bunny measurements in my tests were closer to:

29-44 ms

The 1.054-second request was an outlier.

The meaningful difference was the tail.

The old path could behave like:

30 ms
29 ms
43 ms
1,054 ms
timeout

The new path behaved much more like:

19 ms
20 ms
21 ms
20 ms
23 ms

What improved most was predictability.

For web performance, predictability can matter more than an already-good median.

The cache became smaller and the site still felt faster

This was one of the most counterintuitive observations.

The old cache had reached roughly:

35-36 GB

with hundreds of thousands of entries.

The newer cache started much smaller.

At one point it was around:

3.9 GB

with limits such as:

max_size = 25 GB
min_free = 8 GB

Yet old, rarely visited pages felt much faster.

Why?

Because the old architecture optimized this:

reduce the number of MISSes

The new architecture also optimized this:

reduce the cost of a MISS

That difference is fundamental.

A cache can make a slow origin appear excellent until an object is missing.

A fast origin makes cache misses less dangerous.

The new MP4 strategy is simpler too

With a stable private origin, one possible MP4 strategy is full-object cache filling.

Conceptually:

first request
↓
cache MISS
↓
fetch complete MP4 from private origin
↓
store one complete object

Then later browser Range requests can be served from the local cached file:

Browser
↓ Range request
Working Nginx
↓
local complete MP4
↓
206 Partial Content

That gives me:

one video
↓
one cache object

instead of a large collection of arbitrary remote Range fragments.

It also avoids remote slice assembly across multiple storage responses.

Full-object MP4 caching has a real trade-off

Suppose a video is:

500 MB

A user opens it, watches ten seconds, and leaves.

If the server fills the entire cache object, the origin may still transfer all:

500 MB

even though the user consumed only a fraction.

A pure Range strategy could transfer less.

So full-object caching is not automatically optimal.

It works well when:

• videos are not enormous

• users revisit the same files

• seeking is common

• local disk cache is cheap

• the private origin link has enough capacity

It may be a poor fit when:

• files are multi-gigabyte

• most videos are viewed once

• viewers abandon early

• origin bandwidth is expensive

For those workloads I would evaluate:

• fixed-size slicing with a stable origin

• HLS

• DASH

• a video CDN

• a dedicated video platform

The important part is that this is now a deliberate media-delivery decision.

It is no longer mixed with an unreliable cold-origin problem.

How I test media infrastructure now

One of the biggest mistakes I made early was testing the same object repeatedly.

For example:

curl https://example.com/image.avif
curl https://example.com/image.avif
curl https://example.com/image.avif

After the first request, I may simply be benchmarking:

local cache

That tells me very little about the origin.

A better test set includes:

different objects
cold objects
parallel requests
small files
larger files
Range requests
random video seeks

The distinction between one cold URL and fifty different cold URLs is enormous.

proxy_cache_lock can serialize multiple requests for the same cache key.

It does not magically combine fifty different cache misses into one origin request.

The Nginx metrics I care about now

If I were setting up logging from scratch, I would make sure the access log includes fields such as:

$upstream_addr
$upstream_connect_time
$upstream_header_time
$upstream_response_time
$upstream_cache_status
$request_time
$status

Then I would separate:

HIT
MISS
BYPASS
STALE
UPDATING

instead of calculating one global average.

For origin performance, I care about:

MISS p50
MISS p95
MISS p99

upstream_connect_time p50
upstream_connect_time p95
upstream_connect_time p99

A healthy cache can hide a terrible MISS distribution.

That is exactly what happened to me.

Video needs its own test plan

I would also never treat one successful MP4 request as proof that video delivery is healthy.

I would test:

Start of file

Range: bytes=0-1048575

Middle of file

Range: bytes=50000000-51048575

Near the end

A range close to the file boundary.

Random seeks

Several non-sequential ranges.

For each response I would inspect:

HTTP status
Content-Range
Content-Length
Accept-Ranges
ETag

And if I were using Nginx Slice, I would explicitly compare ETags between multiple slices.

That one test could have exposed my production MP4 issue much earlier.

Real production traffic showed that the cache was doing its job

After the migration, I monitored the working server and private origin at the same time.

In one 60-second production sample:

public TX to users:      ~96.84 Mbit/s
traffic from origin:      ~7.07 Mbit/s

There were no new critical Nginx errors during that window.

In another 30-second sample:

public TX:      61.36 Mbit/s
origin RX:       1.40 Mbit/s

The origin byte volume during that window was approximately:

2.28%

of public TX.

That is not a cache-hit ratio.

A byte ratio and a request hit ratio are different metrics.

Public traffic can also contain HTML, API responses, and other data.

But it demonstrates the behavior I wanted:

large public media output
↓
mostly served by local cache
↓
small stream of cold fills from origin

The private media server does not need to reproduce all public traffic.

It only needs to feed cache misses.

I do not think this proves that self-hosting is better

That would be the wrong lesson.

A private origin creates new responsibilities.

I now own:

• disk monitoring

• storage capacity planning

• backups

• restore procedures

• firewall configuration

• Nginx maintenance

• kernel tuning

• availability

• redundancy

If the private origin disappears, cached files may continue working for some time.

Cold objects will not.

So the origin is still a dependency.

If I need high availability, I need:

• replication

• another origin

• failover

• or a managed layer

A managed storage and CDN platform solves many of those operational problems for you.

That can be worth much more than the price of another VPS.

When Bunny would still be my choice

I would seriously consider Bunny again if I needed:

• global user distribution

• edge delivery

• managed geographic caching

• reduced infrastructure maintenance

• rapidly growing storage

• managed CDN behavior

And I would evaluate the platform in the architecture it is designed to provide:

Bunny Storage
+
Bunny CDN

rather than using my direct Storage-origin architecture as a proxy for Bunny CDN performance.

They are not the same product layer.

For my specific case, however, the traffic relationship was simple:

one working server
↓
one media origin

A private static server fits that topology extremely well.

The most important lesson: design the MISS path first

I spent a lot of time thinking about:

cache size
cache eviction
cache keys
Range caching
Nginx Slice
stale responses
keepalive
TLS reuse
timeouts
retries

Those things matter.

But the architectural change that improved the system most was much simpler:

I replaced the origin path.

That changed my caching philosophy.

I no longer start with:

How high can I make the cache-hit ratio?

I start with:

What happens on a cold MISS?

A healthy answer sounds like:

predictable connect time
predictable TTFB
correct Range support
consistent validators
understandable failure modes

Then the cache makes an already good path cheaper and faster.

That is a much safer architecture than depending on the cache to hide the path underneath it.

A cache should be an optimization, not life support

The old architecture often behaved like this:

HIT
↓
excellent

MISS
↓
unpredictable

The new architecture behaves much closer to this:

HIT
↓
excellent

MISS
↓
still fast

That difference changed the user experience more than increasing cache size ever did.

Old media now loads quickly even when it has not been requested recently.

Sometimes it is a HIT.

Sometimes it is a MISS.

From the browser, the difference is often difficult to notice.

That is what I wanted from the beginning.

The final origin is not sophisticated:

WireGuard
↓
Nginx
↓
SSD

No global edge network.

No complicated application runtime.

No special storage API in the request path.

Just a predictable place to get bytes when the cache does not have them.

After two months of tuning cache behavior, debugging TLS connection stalls, experimenting with MP4 byte ranges, and tracing etag mismatch in slice response, that boring cold path turned out to be the most valuable performance optimization of the entire system.

I actually had a reference problem, followed by an acceptance-logic problem.

The image pipeline was already doing something fairly sophisticated: instead of assigning one fixed AVIF quality to every image, it searched for the lowest encoder quality that still passed a perceptual-quality requirement.

My baseline rule was:

target = 60
single allowed worst sample = 58

For small collections, every sampled image had to reach the main target. For larger collections, I allowed one representative sample to fall to the lower floor while the rest still had to pass 60.

That policy was designed for normal source images.

Then I started receiving inputs with a different history:

higher-quality original: ~2 MB
          ↓
      lossy WebP: ~100 KB
          ↓
          AVIF

The original was no longer available.

Only the WebP remained.

At first, the question seemed simple:

If SSIMULACRA2 60 is acceptable for source → AVIF, why shouldn't 60 also be acceptable for WebP → AVIF?

The answer required me to stop thinking about SSIMULACRA2 as a generic "image quality score" and look more carefully at what the metric is actually measuring.

That eventually changed both the quality policy and the implementation.

SSIMULACRA2 measures a pair of images, not their history

A full-reference metric needs two images:

reference
candidate

SSIMULACRA2 compares those two images and estimates the perceptual distortion between them.

The published SSIMULACRA2 scale gives useful anchors:

50  ≈ medium / fair quality
70  ≈ high / good quality
80  ≈ very high quality
85  ≈ excellent quality
90  ≈ visually lossless
100 = mathematically lossless

The SSIMULACRA2 documentation describes these as approximate perceptual-quality reference points rather than universal codec thresholds.

That distinction matters.

60 is not an official "good quality" anchor.

It sits between the published 50 and 70 reference points.

For a web-oriented compression pipeline, that can be a perfectly reasonable trade-off. But it is still a relatively aggressive one.

Now consider what happens when the reference changes.

Direct encoding

If I still have the original:

original
   ↓
AVIF

then the metric compares:

original ↔ AVIF

The resulting score describes the perceptual change introduced by the AVIF conversion relative to the actual source.

That is exactly the question I want answered.

Encoding from a lossy intermediate

Now consider:

original
   ↓ lossy encode
WebP
   ↓ lossy encode
AVIF

If the original is gone, the comparison becomes:

WebP ↔ AVIF

The metric has no access to the image that existed before WebP.

That first generation of quality loss does not disappear from the pixels.

It disappears from the measurement history.

And that difference is fundamental.

Once a lossy intermediate becomes the reference, its defects become "correct"

Imagine the true original contains a very smooth gradient.

The first encoder introduces a small amount of banding:

smooth gradient
       ↓
minor banding

The WebP still looks acceptable.

Now that WebP becomes my reference.

When I create the AVIF, SSIMULACRA2 evaluates how faithfully the AVIF reproduces the WebP.

From the perspective of that comparison, the existing banding is no longer an error.

It is reference content.

The metric can detect additional damage:

WebP
 ↓
additional blur
ringing
edge changes
other distortion
 ↓
AVIF

But it cannot say:

This region was already degraded before the reference file existed.

That earlier state is simply unavailable.

This is one reason lossy transcoding deserves more care than direct encoding from the best available source.

A discussion in the libavif project makes the same broader point: when the source already contains compression artifacts, those artifacts can be carried into the resulting AVIF rather than somehow disappearing during conversion.

This does not mean:

WebP → AVIF is always bad

or:

AVIF automatically amplifies every WebP artifact

It means something narrower:

When the reference is already lossy, a high SSIMULACRA2 score means "close to this lossy reference," not "close to the original that existed before it."

That changed the way I defined the quality policy.

Source provenance became part of the quality policy

I stopped treating every input identically.

The policy I currently prefer is:

Canonical / high-quality source
target = 60
floor  = 58

Lossless WebP
target = 60
floor  = 58

Known already-lossy derivative
target = 65
floor  = 63

The important distinction is not:

JPEG vs WebP

It is:

best available source
vs
known lossy derivative

That matters because WebP itself does not imply loss.

WebP supports lossless encoding. The WebP lossless bitstream specification describes a format that reconstructs pixel values exactly.

So a true lossless WebP does not introduce an earlier lossy generation that needs special treatment.

Likewise, a JPEG might be the best available source, or it might be the fifth recompressed copy of another image.

The extension cannot tell me that.

The provenance can.

So this is too simplistic:

if (extension === ".webp") {
  target = 65;
}

A better model is conceptually:

if (sample.isKnownLossyDerivative) {
  target = 65;
} else {
  target = 60;
}

The exact mechanism used to determine provenance depends on the pipeline.

Sometimes it can be known from ingestion metadata.

Sometimes it is determined by where the source file came from.

Sometimes lossless versus lossy WebP can be detected independently.

What matters is that the quality policy describes the image's history, not just its filename.

Why I use 65 for the second lossy generation

This deserves a careful answer because it is easy to turn an engineering heuristic into fake science.

There is no official SSIMULACRA2 rule saying:

already lossy source
→ add exactly five points

There is no equation proving:

60 + generation loss = 65

SSIMULACRA2 scores should not be treated as a linear distortion budget that can simply be added across encoding generations.

So why 65?

Because I needed a production policy between two different goals.

My original 60 target is intentionally size-oriented:

50                  60                  70
|-------------------|-------------------|
fair           my baseline          high/good

The published scale places 70 around the high/good-quality region.

For a first encode from a good source, I am willing to operate below that point because file size is a major part of the objective.

For a second lossy generation, I want the encoder to introduce less additional distortion.

Moving from 60 to 65 does that without automatically pushing every image all the way to 70.

So I treat 65 as a guardrail:

60
more aggressive
smaller files
larger additional-loss budget

65
stricter second-generation guardrail

70
higher-quality region
potentially larger files

Could another workload reasonably choose 63?

Yes.

Could an archival pipeline choose 70, 80, or higher?

Absolutely.

The important statement is not:

65 is universally correct.

The statement I can defend is:

I do not want to give a known second-generation lossy encode the same distortion budget I give a direct encode from a good source.

For my size-versus-quality objective, 65 is where I currently draw that line.

Why the lower bound became 63

My original large-collection rule was:

target = 60
allowed single-sample floor = 58

That is a two-point exception:

60 - 58 = 2

If I raise the target to 65 while preserving the semantics of the original rule:

65 - 2 = 63

So:

target = 65
floor  = 63

Using 62 would quietly change the policy:

65 - 62 = 3

I would then be doing something strange:

1. becoming more conservative because the source is already lossy;

2. while simultaneously giving the worst sample a larger exception than before.

I could not justify that.

So I kept the same tolerance.

Again:

63 is not magic.

The important property is:

target - floor = 2

for both classes.

File size does not tell me how damaged the WebP is

A particularly tempting shortcut is to look at the compression ratio.

For example:

2 MB → 100 KB

That is roughly a 20× reduction.

It looks enormous.

It is natural to think:

That WebP must have been heavily damaged.

But that conclusion does not follow from file size alone.

Compression efficiency depends heavily on image characteristics:

• dimensions;

• noise;

• local detail;

• gradients;

• flat regions;

• line art;

• chroma content;

• entropy;

• source format;

• encoder settings.

A relatively simple illustration can sometimes compress extremely well while remaining visually convincing.

A noisy photograph may behave very differently.

This is why I do not implement logic such as:

if (compressionRatio >= 20) {
  target = 70;
} else if (compressionRatio >= 10) {
  target = 65;
} else {
  target = 60;
}

That mapping looks scientific because it contains numbers.

It isn't.

A compression ratio is not a perceptual metric.

The useful question is:

Has this image already passed through a lossy generation?

Not:

How dramatic does the byte reduction look?

Then I discovered the bigger problem: the new threshold was not actually enforced

At this point, I thought implementing the policy would be straightforward.

The code already had the concept of format-specific targets.

Simplified:

const PNG_TARGET = 60;
const JPEG_TARGET = 60;
const WEBP_TARGET = 65;

function targetForSample(sample) {
  if (sample.name.endsWith(".webp")) {
    return WEBP_TARGET;
  }

  if (sample.name.endsWith(".png")) {
    return PNG_TARGET;
  }

  return JPEG_TARGET;
}

That creates an attractive illusion.

The code "knows" WebP has a target of 65.

But the adaptive search that actually chose the AVIF quality still evaluated the candidate using the global threshold.

Conceptually:

passed =
  worstScore >= GLOBAL_FLOOR &&
  secondWorstScore >= GLOBAL_TARGET;

with:

GLOBAL_TARGET = 60;
GLOBAL_FLOOR = 58;

Meanwhile, the per-format target was used later for reporting individual samples.

That means one candidate could effectively produce:

WebP target = 65
score       = 61.2

log:
BELOW_TARGET

while the actual search predicate still concluded:

61.2 >= 60

PASS

This kind of bug is dangerous precisely because the system appears to know the correct requirement.

The logs may even display it.

Nothing crashes.

The build succeeds.

The encoder returns an output.

But the optimization process is solving a different problem from the one the logs describe.

The lesson is broader than image compression:

A policy that exists only in reporting is not an enforced policy.

The thresholds need to reach the actual acceptance predicate

The safer model is to make each sample carry the policy that applies to it.

For example:

function policyForSample(sample) {
  if (sample.isKnownLossyDerivative) {
    return {
      target: 65,
      floor: 63,
    };
  }

  return {
    target: 60,
    floor: 58,
  };
}

Then evaluate the actual measured result:

function assessSample(result) {
  const policy = policyForSample(result.sample);

  return {
    ...result,
    target: policy.target,
    floor: policy.floor,

    targetMargin: result.score - policy.target,
    floorMargin: result.score - policy.floor,

    meetsTarget: result.score >= policy.target,
    meetsFloor: result.score >= policy.floor,
  };
}

Now the important information travels with the measurement:

score
target
floor
margin

instead of having the score in one part of the system and the real requirement somewhere else.

The one-outlier rule becomes much clearer this way

Suppose I want this behavior.

Small collections

No exceptions:

every sampled image must satisfy its own target

Large collections

At most one representative sample may miss its normal target:

one sample:
floor <= score < target

all remaining samples:
score >= target

Then the implementation can reflect exactly that:

function evaluateCandidate(results, allowSingleOutlier) {
  const assessed = results.map(assessSample);

  const belowFloor = assessed.filter(
    (item) => !item.meetsFloor,
  );

  if (belowFloor.length > 0) {
    return {
      passed: false,
      reason: "sample-below-floor",
      assessed,
    };
  }

  const belowTarget = assessed.filter(
    (item) => !item.meetsTarget,
  );

  const allowedTargetMisses =
    allowSingleOutlier ? 1 : 0;

  return {
    passed:
      belowTarget.length <= allowedTargetMisses,

    reason:
      belowTarget.length <= allowedTargetMisses
        ? "passed"
        : "too-many-target-misses",

    assessed,
  };
}

This is much easier to reason about than:

sort all scores
take worst
take second-worst
compare both against global constants

Once thresholds vary between samples, that old shortcut becomes actively misleading.

Mixed thresholds break raw "worst score" ordering

Imagine two samples.

Sample A

score  = 63
target = 60

Margin:

+3

It passes comfortably.

Sample B

score  = 64
target = 65

Margin:

-1

It fails.

If I sort by raw score:

63
64

Sample A looks worse.

But relative to the actual policy:

Sample A = healthy
Sample B = failing

The meaningful quantity is no longer:

score

It is something closer to:

score - requiredTarget

or, even better, an explicit pass/fail assessment against each sample's own policy.

That is an important implementation detail.

Once thresholds become heterogeneous, algorithms built around:

const scores = results
  .map((r) => r.score)
  .sort((a, b) => a - b);

need to be reconsidered.

A normalized margin is more useful than the raw score

For heterogeneous policies, I find this representation easier:

targetMargin =
  score - target;

Examples:

score 64, target 60
margin +4

score 64, target 65
margin -1

Now zero has a consistent meaning:

margin >= 0
→ normal target passed

The same can be done for the floor:

floorMargin =
  score - floor;

For the allowed outlier:

targetMargin < 0
floorMargin >= 0

That describes the policy more precisely than reasoning about raw SSIMULACRA2 scores alone.

The adaptive search must use this predicate too

Fixing only the final validation step is not enough.

An adaptive encoder might test qualities such as:

q=35
q=43
q=39
q=41
q=40

trying to find:

the minimum encoder quality that passes.

The word passes is doing a lot of work here.

The search is only correct if its pass predicate is correct.

The architecture should be:

encode q=N
    ↓
measure SSIMULACRA2
    ↓
apply per-sample policy
    ↓
PASS / FAIL
    ↓
choose next q

Not:

encode q=N
    ↓
compare to global 60
    ↓
choose next q
    ↓
later discover WebP wanted 65

If the adaptive search uses the wrong predicate, everything downstream can be perfectly implemented and the chosen quality can still be wrong.

History-based quality prediction also needs policy awareness

Adaptive encoders often try to avoid starting every search from scratch.

Suppose previous results suggest:

similar images usually pass around q=38

The next search might start around 38.

That can save substantial encoding work.

But there is a hidden dependency:

"usually pass"

under which policy?

If the historical result was produced with:

target 60
floor 58

and the new source requires:

target 65
floor 63

the history is no longer describing exactly the same problem.

A correct final predicate protects correctness.

The search simply starts too low and climbs.

But efficiency suffers, and debugging becomes harder because old statistics are being mixed with a new acceptance regime.

I therefore prefer versioning the quality policy used by historical predictors.

Something like:

sim-v1-60-58

becoming:

sim-v2-source60-58-lossy65-63

That may look like housekeeping.

In production optimization code, it is part of reproducibility.

The best fix is still avoiding the second lossy generation

There is a simpler branch in this decision tree.

Suppose I still have:

original: 2 MB

and also:

WebP derivative: 100 KB

Then I do not want:

original
   ↓
WebP
   ↓
AVIF

I want:

original
   ↓
AVIF

That makes the original the reference again.

And now the normal:

60 / 58

policy applies.

A higher threshold on the second generation cannot recover information lost during the first.

This distinction is worth remembering:

WebP → AVIF @ 65

means:

Preserve the current WebP more faithfully.

It does not mean:

Recover detail that disappeared before the WebP was created.

If the information is no longer in the pixels, a perceptual metric cannot bring it back.

Quality should not be the only gate

Eventually I realized there was another question that had nothing to do with SSIMULACRA2.

Suppose:

source WebP = 100 KB

The AVIF that passes 65/63 is:

AVIF = 96 KB

The conversion passes the quality policy.

But what did I gain?

4 KB

For that I introduced:

• another encode;

• another lossy generation;

• additional CPU work;

• another transformation step;

• more pipeline complexity.

That may not be worth it.

Now consider:

WebP = 100 KB
AVIF = 65 KB

while still passing the same perceptual policy.

That is a much more interesting result.

So I now think of the conversion as having two gates.

Quality gate

Is the additional degradation acceptable?

Size gate

Is the size reduction meaningful enough
to justify another conversion?

Passing the first gate does not imply passing the second.

A pipeline could eventually encode that distinction explicitly:

const qualityPassed =
  evaluateCandidate(results, allowSingleOutlier)
    .passed;

const bytesSaved =
  sourceBytes - outputBytes;

const savingRatio =
  bytesSaved / sourceBytes;

const sizeWorthIt =
  savingRatio >= MIN_REQUIRED_SAVING;

const keepAvif =
  qualityPassed && sizeWorthIt;

I am not claiming one universal MIN_REQUIRED_SAVING.

That depends on the workload.

The useful architectural point is separating the decisions.

What would actually prove whether 65 is the right target?

A much stronger answer requires true originals.

I would build a representative test corpus and preserve the master for every image.

Then test:

A
original → AVIF
target 60

B
original → lossy WebP → AVIF
target 60

C
original → lossy WebP → AVIF
target 63

D
original → lossy WebP → AVIF
target 65

E
original → lossy WebP → AVIF
target 70

For every final AVIF I would record:

final byte size

selected encoder quality

SSIMULACRA2:
WebP → final AVIF

SSIMULACRA2:
true original → final AVIF

The second metric is the critical one.

It restores the information that is missing when the WebP intermediate becomes the only reference.

I would also inspect difficult cases manually:

• smooth gradients;

• fine line art;

• small text;

• highly textured images;

• dark gradients;

• saturated color transitions;

• already-visible WebP artifacts.

Objective metrics are extremely useful, but visual inspection is still valuable when deciding whether codec settings are appropriate.

I have not run this controlled experiment on a sufficiently representative corpus of preserved originals.

That means I cannot honestly say:

65 is scientifically optimal.

I can say:

65/63 is my current engineering policy
for known already-lossy derivatives.

Those are very different claims.

The second one is defensible.

Tests I would write before trusting the implementation

Once the quality policy became heterogeneous, I stopped thinking of it as a one-line constant change.

I would want explicit boundary tests.

Clean source at the exact target

target = 60
score  = 60

→ PASS

Clean source just below the target

target = 60
score  = 59.99

→ FAIL

unless it is the one permitted outlier in a collection where outliers are enabled.

Lossy derivative at the target

target = 65
score  = 65

→ PASS

Lossy derivative below target but above floor

target = 65
floor  = 63
score  = 64

Result:

allowed only if:
- single outlier is enabled
- no other sample misses its target

Exact lossy floor

score = 63

→ valid outlier boundary

Below the lossy floor

score = 62.99

→ FAIL

regardless of how the other samples perform.

Mixed-source collection

PNG
score  = 61
target = 60
→ PASS

lossy WebP
score  = 64
target = 65
→ TARGET MISS

The WebP must be treated as the failing sample even though its raw score is numerically higher than the PNG score.

That single test catches the flaw in raw-score sorting immediately.

The final mental model

I started with:

What SSIMULACRA2 number should I use for WebP?

That was the wrong abstraction.

The better questions are:

What is the best available reference?

Has this reference already gone
through lossy compression?

What additional distortion budget
am I willing to spend from here?

Does the actual search predicate
enforce that budget?

Is the resulting byte saving large
enough to justify another encode?

Those questions produce a much clearer decision tree.

My current rules

For this type of web-oriented compression workload, my current policy is:

BEST AVAILABLE / CANONICAL SOURCE

target = 60
floor  = 58

LOSSLESS WEBP

target = 60
floor  = 58

KNOWN ALREADY-LOSSY DERIVATIVE

target = 65
floor  = 63

The implementation rules matter just as much:

Do not classify solely by extension.

Use provenance when possible.

Apply per-sample thresholds
inside the actual PASS/FAIL predicate.

Do not compare mixed-source samples
against one global second-worst threshold.

Evaluate a score relative
to that sample's own target.

Make adaptive search use
the same predicate as final validation.

Version historical quality data
when the acceptance policy changes.

If the real original exists,
encode from it directly.

If AVIF barely reduces file size,
consider keeping the existing WebP.

And finally:

65/63 is an engineering policy,
not a universal SSIMULACRA2 law.

The bug was not in the metric

SSIMULACRA2 was doing exactly what it was supposed to do.

It compared the reference I gave it with the candidate I gave it.

The first mistake was expecting that score to somehow represent quality loss that occurred before the reference existed.

The second mistake was more practical: adding a stricter threshold without tracing whether that threshold actually reached the acceptance condition used by the adaptive search.

Both problems came from the same broader engineering mistake:

I was looking at the number without following the data and decision path around it.

A quality score has context.

A threshold has semantics.

And a configuration value only matters if the code that chooses the output actually obeys it.

That is the rule I trust more than any particular SSIMULACRA2 number.

What model should I switch to?

That was my first thought too.

On August 17, I checked my DeepSeek API usage and noticed that the balance was disappearing roughly five times faster than I was used to.

I knew a pricing update was coming, but I had mentally filed it away as a normal increase. Maybe 20%. Maybe 50%. Even twice the old price would have been annoying but manageable.

Once I looked at the actual numbers, I realized this was different.

Instead of immediately benchmarking Qwen, GLM, Kimi, MiniMax, Mistral and every other model I could find, I tried to separate two things that I had previously treated as one:

the model I wanted to use

and

the company selling me the compute to run it

That distinction ended up being much more useful than immediately changing models.

The pricing change that triggered the search

Before the change, DeepSeek V4 Flash was extremely inexpensive.

The old rates I had been using were:

Input, cache miss:  $0.14 / 1M tokens
Cached input:       $0.0028 / 1M tokens
Output:             $0.28 / 1M tokens

The current pricing for deepseek-v4-flash, corresponding to DeepSeek-V4-Flash-0731, is split between off-peak and peak periods.

Off-peak:

Input, cache miss:  $0.22 / 1M tokens
Cached input:       $0.007 / 1M tokens
Output:             $0.66 / 1M tokens

Peak:

Input, cache miss:  $0.44 / 1M tokens
Cached input:       $0.014 / 1M tokens
Output:             $1.32 / 1M tokens

DeepSeek lists the peak windows as 01:00 to 04:00 UTC and 06:00 to 10:00 UTC.

So when I say my spending appeared to increase by about five times, I am not claiming that every type of token received an identical 5x increase.

The exact change depends on the workload.

At peak pricing:

Cache-miss input
$0.14 -> $0.44
≈ 3.14x

Output
$0.28 -> $1.32
≈ 4.71x

Cached input
$0.0028 -> $0.014
= 5x

For an application that produces a significant amount of output, seeing the real bill move toward five times its previous level is not particularly surprising.

The question was what to do about it.

Replacing an LLM is not always the simplest engineering decision

It is easy to say:

DeepSeek became expensive. Just use another model.

Technically, I could.

There are plenty of alternatives now.

Qwen, GLM, Kimi, MiniMax, Mistral and others are all worth looking at.

But an LLM is not always interchangeable infrastructure.

Once you have a model working well inside an application, changing it can create a surprising amount of work.

Prompt behavior can change.

The same instruction may produce longer or shorter answers.

Tone can shift.

Sampling parameters may need adjustment.

Context handling can differ.

Edge cases you already solved can return in different forms.

You may save money on tokens and then spend engineering time trying to make the replacement behave like the model you already had.

I was not against changing models.

I simply did not want to start there.

Instead, I asked a narrower question:

Can I keep the exact same DeepSeek V4 Flash checkpoint and buy the inference somewhere else?

For an open-weight model, the answer can be yes.

The model provider and the inference provider do not have to be the same company

The checkpoint I was interested in is:

deepseek-ai/DeepSeek-V4-Flash-0731

Its weights are publicly available under the MIT License.

That means DeepSeek is not the only company technically capable of running the model.

This sounds obvious when written out, but I had not really been treating API purchasing this way.

My old mental model was basically:

My application
      |
      v
DeepSeek API
      |
      v
DeepSeek V4 Flash

But once the weights are available, the structure can look more like this:

                 DeepSeek V4 Flash
                       weights
                          |
          +---------------+---------------+
          |               |               |
          v               v               v
     DeepSeek API       Runware       Other providers
                                           |
                                           v
                                     Their own GPUs

And of course there is another option:

DeepSeek V4 Flash weights
          |
          +--> DeepSeek API
          |
          +--> Runware
          |
          +--> other inference platforms
          |
          +--> private infrastructure
          |
          +--> your own GPUs

The checkpoint can remain the same while the infrastructure underneath it changes completely.

That was the market I started researching.

I searched for the checkpoint, not for another chatbot

This changed how I approached the research.

I was not searching for:

cheapest AI API

or:

best DeepSeek alternative

Those searches mix together completely different models.

Instead, I wanted providers serving the exact checkpoint I was already using.

My research process was fairly repetitive.

First I asked an AI to help me create a detailed research prompt covering major inference providers, smaller GPU platforms, aggregators, regional providers and companies I had never heard of.

Then I ran the research repeatedly.

Usually I do something like 15 to 20 separate searches, because one search often surfaces the same popular companies again and again.

After that, I combined the results, compared the candidates and removed providers that did not actually expose the checkpoint I wanted.

Only then did I manually inspect the finalists.

That manual step matters.

I do not want an AI-generated ranking to make the final infrastructure decision for me.

For anything that will carry real traffic, I still want to check the provider myself:

Exact checkpoint?
Pricing?
API format?
Documentation?
Latency?
Errors?
Actual billing?

One of the providers that survived that process was Runware.

Runware had the exact checkpoint I wanted

The important part was not simply that Runware supported "DeepSeek."

It offered:

DeepSeek-V4-Flash-0731

That was exactly what I was looking for.

I wanted to remove as many variables as possible.

If I changed both the model and the provider at the same time, then any difference in output quality or behavior would be harder to attribute.

Keeping the same checkpoint makes the experiment much cleaner.

At the time I tested it, the Runware rates were:

Input:         $0.076 / 1M tokens
Cached input:  $0.014 / 1M tokens
Output:        $0.153 / 1M tokens

I checked those numbers more than once because the comparison looked strange.

I expected third-party inference to beat DeepSeek's new prices.

I did not expect it to beat DeepSeek's old prices too.

It was about 45% cheaper than the old DeepSeek API

Before the price increase, DeepSeek input cost:

$0.14 / 1M

Runware:

$0.076 / 1M

The difference is:

($0.14 - $0.076) / $0.14
≈ 45.7%

So Runware was roughly 45.7% cheaper for input than the old DeepSeek rate.

Output showed almost the same pattern.

Old DeepSeek:

$0.28 / 1M

Runware:

$0.153 / 1M

Difference:

($0.28 - $0.153) / $0.28
≈ 45.4%

Again, roughly 45% cheaper.

That was more interesting to me than the price increase itself.

DeepSeek's old V4 Flash API already felt cheap.

Yet another company was willing to serve the same public checkpoint for considerably less.

Against the new DeepSeek prices, the difference becomes much larger

Once the new DeepSeek rates are used as the baseline, the numbers become much more dramatic.

For cache-miss input during off-peak hours:

DeepSeek: $0.22
Runware:  $0.076

$0.22 / $0.076 ≈ 2.9x

For output:

DeepSeek: $0.66
Runware:  $0.153

$0.66 / $0.153 ≈ 4.3x

During peak hours:

Input:
$0.44 / $0.076 ≈ 5.8x

Output:
$1.32 / $0.153 ≈ 8.6x

So for those token categories, the gap can become very large.

There is one important exception.

DeepSeek's off-peak cached input currently costs:

$0.007 / 1M

Runware cached input costs:

$0.014 / 1M

In that specific case, DeepSeek is cheaper.

This is why I would not choose an inference provider based on one headline number.

Your actual token mix matters.

A workload dominated by cache hits can have very different economics from a conversational application producing lots of fresh output.

Price was only the first test

Finding a cheap endpoint is easy.

Finding a cheap endpoint I actually want to send traffic to is harder.

A provider can advertise extremely low token prices and still be useless if requests time out, capacity disappears or the API constantly returns errors.

So after finding Runware, I connected it and started sending real requests.

One of my early dashboard snapshots looked like this:

790 total requests
56 successful
734 errors

That obviously looks terrible.

Later I captured another state:

1,570 total requests
836 successful
734 errors

The difference between those two snapshots is what interested me:

+780 total requests
+780 successful requests
+0 additional errors

I want to be precise about what that means.

It does not prove perfect reliability.

It does not prove 99.999% uptime.

And those numbers alone do not explain why the original 734 requests failed.

I am deliberately not pretending they do.

The only conclusion I can draw from those two snapshots is that the next 780 requests increased the success counter by 780 while the error counter stayed exactly where it was.

I continued testing after that.

By the time I was writing this, I had already passed more than 1,000 successful AI requests through the Runware endpoint.

For a first day of testing, that was enough for me to continue.

Not enough to call the provider perfect.

Enough to keep using it.

Latency did not become the tradeoff I expected

Normally, when I see dramatically cheaper infrastructure, I expect to pay for it somewhere else.

Latency is an obvious place to look.

In my testing, that has not been an obvious problem so far.

The successful requests I inspected generally completed in fractions of a second to around 1.5 seconds.

Some short requests were billed at approximately:

$0.000005

to:

$0.000018

per request.

After hundreds of successful requests, the dashboard was still showing spending around one cent.

That was the point where the experiment became convincing enough for me personally.

I was no longer comparing two pricing pages.

I had real requests going through the endpoint.

I had responses returning.

I could inspect the latency.

And I could see the actual cost accumulating.

The interesting part is bigger than Runware

The most interesting thing here is not really one provider.

It is what open weights do to the inference market.

DeepSeek pays to develop a model.

Then the model weights become available under a permissive license.

At that point, other infrastructure companies can compete to run the same model.

They can compete on:

GPU cost
GPU utilization
batching
serving software
capacity
networking
geography
margins
latency
reliability
pricing

DeepSeek may have created the model, but another company may be able to operate the infrastructure more cheaply for a particular workload.

That creates an unusual competitive situation.

DeepSeek can effectively end up competing for inference revenue against companies running DeepSeek's own checkpoint.

When official V4 Flash inference cost $0.14 for input and $0.28 for output, that competition was easy to ignore.

The official API was already extremely cheap.

But when peak pricing becomes $0.44 for input and $1.32 for output while another provider offers the same checkpoint at $0.076 and $0.153, the infrastructure market becomes much more relevant.

This is also why I do not think of a provider like Runware as simply reselling DeepSeek API calls.

If a company is running the public weights on separate infrastructure, then the product being sold is the inference layer.

The model came from DeepSeek.

The compute came from somebody else.

Those can now be purchased independently.

Did DeepSeek make a mistake by releasing the weights?

I do not think the answer is obvious.

There is an easy argument to make:

DeepSeek created competitors for its own API business.

In a narrow sense, yes.

But that is only one side of it.

Making a model broadly deployable also makes it easier for the model to spread.

Developers integrate it.

Researchers use it.

Cloud platforms support it.

Inference companies optimize it.

Businesses can deploy it privately.

Tooling gets built around it.

The model becomes available in environments the original API provider might never serve directly.

That ecosystem has value too.

So I am not claiming that releasing V4 Flash was a bad business decision.

I think the more defensible conclusion is simpler:

Once model weights are publicly available under a permissive license, inference becomes a competitive infrastructure market.

The company that trained the model is no longer necessarily the cheapest company to run it.

When the official API is already cheap, developers may not care.

When the official API gets significantly more expensive, they start looking around.

I did.

Why I did not just self-host the checkpoint

Once you accept that the model and infrastructure can be separated, there is an obvious next question:

Why pay an inference provider at all?

Why not download the weights and serve the model yourself?

Because "the weights are available" and "inference is cheap to operate" are completely different statements.

DeepSeek's own deployment material includes an example using four GB300 GPUs on a single node.

And hardware is only the beginning.

A production deployment also has to deal with:

VRAM
model loading
inference engines
batching
KV cache
request concurrency
monitoring
networking
failover
capacity planning
power
cooling
maintenance
GPU utilization

The last point matters a lot.

Buying expensive hardware is one thing.

Keeping that hardware busy enough to justify its cost is another.

An inference provider can distribute those costs across many customers.

My own workload cannot do that nearly as efficiently.

If somebody is willing to serve the model to me for:

$0.076 / 1M input tokens

then owning the GPUs myself may be a much worse business decision, even though technically I could run the model.

At sufficiently large scale the calculation changes.

For me, it has not reached that point.

The API layer became interchangeable before the model did

This is the biggest change in how I think about LLM infrastructure after this experiment.

Previously I implicitly treated a model API as one product:

DeepSeek model + DeepSeek API = DeepSeek

Now I separate it into layers:

Model
  |
Checkpoint
  |
Inference engine
  |
GPU infrastructure
  |
API

If the model is closed, those layers may effectively be bundled together.

With an open-weight model, they do not have to be.

That means an application can sometimes change several infrastructure layers while leaving the actual model alone.

From an engineering perspective, that is useful.

Instead of doing this:

Model migration
        +
Prompt retesting
        +
Behavior changes
        +
New edge cases

you may be able to do this:

Same checkpoint
        +
Different inference provider

That is a much smaller migration surface.

This is not a sponsored article

I want to make this explicit because I have mentioned Runware throughout the article.

This is not an advertisement.

Runware did not contact me.

I did not contact Runware before testing the service.

Nobody asked me to write this article.

There is no affiliate agreement.

There is no referral payment.

There is no discount code.

There were no free credits in exchange for publishing this.

I found the provider while researching alternatives, created an account, added my own money and connected the API.

That is the entire relationship.

I have also only been testing it for one day.

A day is enough to form a first impression.

It is not enough to establish long-term reliability.

Something could break next week.

Pricing could change.

Performance could degrade.

Another provider could become cheaper.

If that happens, I have no problem switching again.

I am not trying to find an inference provider to be loyal to.

I am trying to buy reliable compute at a reasonable price.

For now, my experience is simply this:

More than 1,000 successful AI requests
Good latency in my testing
Very low token cost
No new errors during the measured 780-request window
One day of real usage

That is what I know.

Anything beyond that would be speculation.

The numbers that changed my approach

For me, these three sets of prices summarize the entire experiment.

Old DeepSeek V4 Flash:

Input:  $0.14 / 1M
Output: $0.28 / 1M

New DeepSeek V4 Flash peak pricing:

Input:  $0.44 / 1M
Output: $1.32 / 1M

Runware when I tested the same checkpoint:

Input:  $0.076 / 1M
Output: $0.153 / 1M

The exact winner will depend on caching, traffic patterns, latency requirements and how the providers change their pricing over time.

But I no longer think the right first question for an open-weight model is:

Who created this model?

The next question matters just as much:

Who can run this exact checkpoint well and cheaply?

From now on, that is how I plan to evaluate open-weight APIs.

First identify the model.

Then identify the exact checkpoint.

Then find the companies serving it.

Compare the token economics.

Send actual requests.

Measure errors and latency.

And only then decide where the inference should run.

In this case, that approach let me keep DeepSeek V4 Flash while paying substantially less for the compute behind it.

And I did not have to change the model at all.

The goal was not simply to publish articles in multiple languages. I wanted to see how a multilingual site behaves in the real world, how quickly search engines discover localized pages, how indexing scales when every article creates dozens of URLs, and where the practical limits start to appear.

It has now been one week since launch, and the first search data is starting to come in.

The numbers are still small, but they are already useful.

Why I chose Next.js

I built the project with Next.js because it gave me a solid foundation for internationalized routing, server-side rendering, static generation, reusable metadata, and a large number of localized pages.

For a multilingual site, the technical structure matters a lot.

Every language needs a stable URL. Pages need to be crawlable. Metadata has to be correct. Search engines need clear signals about which page belongs to which language.

Next.js made that part of the project much easier to manage.

That does not mean Next.js automatically improves rankings or makes Google index pages faster. It simply gives me the tools to build the structure properly.

Search engines still decide what to crawl, what to index, and what deserves to appear in search results.

The first week in Google

After the first week, Google Search Console showed:

• 439 impressions

• 5 clicks

Yandex also started discovering the site and generating some visibility.

Five clicks is obviously not a meaningful amount of traffic yet.

What interested me more was the number of impressions.

For a new personal project, seeing hundreds of search impressions after only a week confirmed that Google had already started discovering and testing pages from the site in search results.

The multilingual structure also means those impressions are not limited to a single language.

That is one of the main reasons I wanted to run this experiment in the first place.

21 languages create a lot of URLs

The interesting part of multilingual publishing is how quickly everything scales.

One article does not create one page.

On this site, one article can create 21 separate localized URLs.

That means:

• 5 articles can become 105 pages

• 10 articles can become 210 pages

• 20 articles can become 420 pages

The number of URLs grows much faster than the number of articles.

That gives search engines more possible entry points, but it also creates more work for crawling, indexing, internal linking, metadata management, and language targeting.

More pages do not automatically mean more traffic.

They simply create more opportunities to appear for different searches in different languages.

Indexing is slower than discovery

A few weeks into the project, the indexing numbers looked like this:

• 436 pages indexed

• 35 pages not indexed yet

So most of the pages in that snapshot had already entered Google's index, but some were still waiting.

This is where the difference between discovery and indexing becomes very noticeable.

My sitemap.xml is generated automatically and submitted through Google Search Console.

That helps Google discover the URLs, but submitting a sitemap does not mean every page will be indexed immediately.

Google can know that a URL exists without adding it to the index.

An indexed page can also sit there without receiving meaningful impressions or clicks.

These are separate stages, and I found it much more useful to track them separately instead of treating everything as one SEO metric.

Multilingual SEO also multiplies mistakes

There is another side to publishing in 21 languages.

If something is wrong, the problem can also scale very quickly.

A mistake in:

• canonical tags

• hreflang

• metadata

• internal links

• translations

• URL structure

can potentially affect many localized pages at once.

With a normal blog, one bad configuration might affect one article.

With 21 language versions, the same mistake can be repeated across 21 URLs.

That makes testing and validation much more important as the site grows.

What the first week actually tells me

I do not think 439 impressions and 5 clicks prove that multilingual SEO is already a success.

It is far too early for that.

What the numbers do show is that search engines are discovering the site, indexing a large percentage of its pages, and already showing some of them in search results.

For me, that is enough to make the experiment interesting.

The next question is much more important:

Which languages and articles will continue receiving impressions and eventually turn them into consistent organic traffic?

That is something only more data can answer.

What I learned so far

The biggest lesson is that multilingual publishing creates search opportunities very quickly, but visibility, indexing, rankings, and traffic are completely different things.

A page can be discovered without being indexed.

It can be indexed without ranking.

It can rank occasionally without receiving clicks.

And a language that generates impressions during the first week may not necessarily become one of the site's strongest sources of traffic later.

So instead of looking at one big number, I am now paying attention to each stage separately.

For a project that started mainly as a technical challenge, that has probably been the most useful part of the experiment so far.

You can check out the project here:

https://jsvar.com/en/blog