Skip to content
This page View Markdown Open in ChatGPT Open in Claude
WebSocket benchmark · Linode (Akamai Cloud) · Shared CPU · 14 October 2026

1 vCPU, 1 GB RAM : 26,484 WebSocket clients connected at once.

API Maker 4.0.0 on the smallest Linode server, a Nanode 1 GB (g6-nanode-1) : 1 shared vCPU and 1 GB of RAM at $5 a month, its MongoDB and Redis on the same machine. A second server opened WebSocket clients, each with both tokens and a subscription of its own, until the memory of the server was full, while autocannon made the calls which notify them : every run 60 seconds, the memory and CPU of every part sampled each second.

Clients connected 26,484 the memory was full : 149 MB left
Memory per client 8.6KB API Maker 3.3 · kernel 4.3 · Redis 1.0
New clients / second 2,602 both tokens and a subscription · 5,302 without one
Notification delivered in 2.51ms p50 at 1,000 a second · p99 3.82 ms

The server
Server Linode (Akamai Cloud), Nanode 1 GB (g6-nanode-1), shared cpu, $5 / month
CPU AMD EPYC 7642 48-Core Processor, 1 shared vCPU (KVM)
KVM guest, 1 thread per core · L1d 64 KiB (1 instance) · L2 512 KiB (1 instance) · L3 16 MiB (1 instance) · BogoMIPS 4600.00
Memory 1 GB (961 MB usable), swap 2.5 GB (496 MB of Linode + 2 GB made by the installer)
Disk, region 25 GB SSD, Mumbai, India (ap-west)
System Ubuntu 24.04.4 LTS, Linux 6.8.0-134, Docker 29.9.0
API Maker 4.0.0 : release build of api_maker_be_rust 423a34c with the new WebSocket core of 14 October 2026 (opt-level 3, fat LTO), installed by its installer (https://apimaker.dev/v4/install.sh), no reverse proxy
Settings the defaults of the installer : one worker thread per vCPU, up to 1,048,576 open files for the service (LimitNOFILE), and the sandboxes kept running for the whole run (removeSandboxInactiveSinceMinutes 1440)
Network limits as the installer sets them : 246,016 connections tracked by the firewall of Linux (net.netfilter.nf_conntrack_max), 16,384 waiting to be accepted (net.core.somaxconn). The memory of all its TCP connections together : raised by API Maker as the clients came, to 240 MB at the most (net.ipv4.tcp_mem, 25 % of the memory ; Linux gives 9 %)
Databases MongoDB 8.0.10 (the MongoDB of API Maker, a replica set of one member, in Docker) · Redis 7.0.5 (the Redis of API Maker, in Docker) : on the same server. No other database runs
Sandboxes TypeScript on Node.js 24.21.0 (an image of 404 MB built by API Maker), 2 sandboxes of the account (sandboxCountForAdmin 2, the default)
Load generator Dedicated 8 GB (g6-dedicated-4) : AMD EPYC 7713 64-Core Processor, 4 dedicated vCPUs, Ubuntu 24.04.4 LTS. The WebSocket clients : am_ws_load, a program in Rust made for these benchmarks, a socket and a task per client, from 11 addresses of the load generator. The HTTP calls : Node.js 24.21.0, autocannon 8.0.0
Network A VLAN of the same data centre, round trip 0.47 ms (ping)
Date 14 October 2026, 03:18 to 03:49 UTC

Only what the run needs was running : API Maker and its two sandboxes, its MongoDB and its Redis. API Maker was restarted and the swap emptied before the run. How every number was taken : How the WebSocket benchmarks are made. Every server side by side : All results. The APIs of this server : 1 vCPU · 1 GB, API benchmark.

The data

  • The clients connect to ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token> : the token of an API user, and the token of a person, a row of the customers table of the benchmark dataset (https://apimaker.dev/json/benchmark_dataset.json) made by a DB token generator. API Maker checks both.
  • A subscription is one REGISTER for the custom WebSocket event order-updates with a condition of its own : { conditionType: 'RESPONSE', criteria: { recipientId: '<the number of the client>' } }. It is the way one notification reaches one person.
  • A notification for one person is 141 bytes ; a row of the dataset told to the clients of a table is 321 bytes.
  • The tables are the ones of the API benchmarks, in MongoDB : ten rows of customers, ten columns each.

Results at a glance

New clients per second
Connect : the token of the API user5,905
Connect : both tokens5,302
Connect with both tokens and subscribe2,602
Subscribe to an event with a can user connect codethe code ran in the server2,097
Subscribe : the code reads the databasethe code ran in the sandbox959
Subscribe : the code reads the clockthe code ran in the sandbox1,960

100 clients at a time connect, are let in and leave, for 60 seconds.

Notifications per second
A notification for one person5,285 calls / second5,216
A broadcast73 calls / second72,245
Rows inserted in a table416 calls / second40,498

As many calls as the server takes : what the clients received each second.

How many clients fit

Clients connect 1,000 a second, level after level, each with both tokens and a subscription of its own ; at each level they wait 30 seconds while the server is sampled (the level the run stops at, 60). The run stopped adding clients at 26,484 : the memory of the server was full. 139 MB of its 961 MB were still counted as available, and Linux had moved 77 MB to the swap since the clients began to connect, to make that room : 62 MB were left (the run stops when what is available, less what Linux moved to the swap since the clients began to connect, is under 5 % of the memory of the server).

Clients connected and memory used during the run 00 MB6.6k240 MB13k481 MB20k721 MB26k961 MB 0 min5 min10 min15 min20 min25 min30 min clients connected memory used, of 961 MB
API MakerMongoDB of API MakerRedisSandboxesLinux : its sockets, Docker, the restAvailable
No client
at rest
5,000 clients
connected
7,500 clients
connected
10,000 clients
connected
15,000 clients
connected
20,000 clients
connected
25,000 clients
connected
26,484 clients
connected
Clients API Maker Redis Kernel MongoDB Sandboxes Server, used Available Swap CPU busy
None (at rest) 103 1 88 139 51 600 362 0 3.1 %
5,000 112 7 124 145 146 746 216 50 2.6 %
7,500 103 9 122 141 145 739 223 102 1.8 %
10,000 91 12 131 125 145 730 231 120 2.0 %
15,000 105 16 152 124 144 755 207 124 2.2 %
20,000 106 22 174 120 140 784 178 155 2.7 %
25,000 118 27 194 114 142 806 156 168 2.5 %
26,484 126 28 200 110 143 813 149 176 2.5 %

Memory in MB. API Maker is the resident memory of its process ; Redis is the memory of its data (used_memory) ; the kernel of Linux keeps every socket and every tracked connection in its slab ; a container is counted the way docker stats counts it. CPU busy is the whole server while the clients wait : the server pings every client once a minute, and each answers.

Memory when the server is full

With 26,484 clients connected and waiting, of the 961 MB of the server :

Part At rest, no client 26,484 clients More
API Maker (process) 103 MB 126 MB + 23 MB
Redis (container) 3 MB 30 MB + 27 MB
Linux kernel : sockets, tracked connections (slab) 88 MB 200 MB + 112 MB
MongoDB of API Maker (container) 139 MB 110 MB − 29 MB
Sandboxes (2 containers) 51 MB 143 MB + 93 MB
Whole server, used 600 MB 813 MB + 213 MB
Available 362 MB 149 MB
Swap used 0 MB 176 MB
Open files of API Maker 27 26,510
Keys of Redis · channels it listens on 13 · 5 52,984 · 26,489

MongoDB and the sandboxes grew during the runs before this one (the inserts, the code of the events), not with the clients.

What a client costs

Between 10,000 and 26,484 clients (5 levels), each client more cost the server 8.6 KB :

Where Per client What it is
API Maker 3.3 KB The client itself : its socket and the one task which reads it, its tokens and what they said, its subscription and the channel it is told on.
Linux kernel 4.3 KB The TCP socket, and the entry of the connection in the table of the firewall (Docker makes Linux track every connection).
Redis 1.0 KB The subscription : its key, its place in the list of its condition, and the channel API Maker listens on.
In all 8.6 KB 121,927 clients per GB of memory.

Measured over every level from the first after the second subscriptions were taken back : what Linux moved of API Maker to the swap is counted with it. The memory the whole server had available went down by 5.2 KB per client : MongoDB, the sandboxes and the cache of files give some back while the clients take theirs.

One more subscription, and a broadcast to everybody

With 7,500 clients connected, every client sent a second REGISTER, for an event without a condition (announcements), 1,000 of them a second : 7,500 were taken in 7.5 s. They waited 30 seconds with two subscriptions each, were told 3 broadcasts, one call of the emit event WS API each, and took the subscription back with UNREGISTER.

One subscription more 0.7KB per client : API Maker -0.0 KB · Redis 0.7 KB
A broadcast to 7.5k clients 0.16s until the last client read it
Half of them told after 92.1ms the first after 28.4 ms
Clients told / second 46,688 one call, every client
Broadcast Clients told The call answered after The first client Half of them 99 % of them The last one Clients told a second
1 7,500 of 7,500 0.18 s 49.6 ms 110 ms 178 ms 179 ms 41,822
2 7,500 of 7,500 0.17 s 26.3 ms 92.5 ms 163 ms 164 ms 45,670
3 7,500 of 7,500 0.16 s 28.4 ms 92.1 ms 159 ms 161 ms 46,688

One call of the emit event WS API tells every client : API Maker asks Redis for the clients of the event, publishes the notification for each of them, and the server which holds a client sends it. The times are counted from the moment the call is written to the moment a client reads its notification.

Memory, 7,500 clients One subscription each Two subscriptions each A subscription more, per client
API Maker 103 MB 95 MB -0.0 KB
Redis (its data) 9 MB 14 MB 0.7 KB
Keys of Redis · channels 15,016 · 7,505 22,517 · 15,005
Whole server, less available 0.7 KB

New clients per second

How many clients a second the server lets in : 100 clients at a time connect, are let in and leave, one after the other, for 60 seconds (after 5 seconds which are not counted). It is what a server faces when its clients all come back, after a deployment or a network cut. The times are counted from the start of the TCP connection.

Connect : the token of the API user

A client connects with x-am-authorization alone, is told CONNECTED, and leaves. API Maker checks the token of the API user and finds its groups : what every WebSocket client costs before it asks for anything.

Clients / second 5,905 354,665 clients in 60 s, 100 at a time
Let in after, p50 9.42ms connected 9.42 ms · TCP 0.36 ms
Let in after, p99 16.8ms max 32.4 ms
Refused or lost 0 not connected, not subscribed, no answer

Server during this run API Maker 0.9 vCPU · whole server busy 100 % · load generator busy 66 %

am_ws_load · Connect : the token of the API user · 60 s
$ am_ws_load churn -c 100 -d 60 "ws://<server>:38246/?x-am-authorization=<token>"
Running 60s test @ ws://<server>:38246
100 clients at a time : connect, CONNECTED, leave

┌─────────────┬─────────┬──────────┬──────────┬──────────┬─────────┬─────────┬──────────┐
│ Stat        │ 2.5%    │ 50%      │ 97.5%    │ 99%      │ Avg     │ Stdev   │ Max      │
├─────────────┼─────────┼──────────┼──────────┼──────────┼─────────┼─────────┼──────────┤
│ TCP connect │ 0.27 ms │ 0.36 ms  │ 0.91 ms  │ 1.21 ms  │ 0.43 ms │ 0.22 ms │ 8.5 ms   │
├─────────────┼─────────┼──────────┼──────────┼──────────┼─────────┼─────────┼──────────┤
│ CONNECTED   │ 4.94 ms │ 9.42 ms  │ 14.37 ms │ 16.78 ms │ 9.51 ms │ 2.38 ms │ 32.42 ms │
├─────────────┼─────────┼──────────┼──────────┼──────────┼─────────┼─────────┼──────────┤
│ Left        │ 10.9 ms │ 16.83 ms │ 23.87 ms │ 27.47 ms │ 16.9 ms │ 3.32 ms │ 48.51 ms │
└─────────────┴─────────┴──────────┴──────────┴──────────┴─────────┴─────────┴──────────┘
┌─────────────┬───────┬───────┬───────┬───────┬─────────┬────────┬───────┐
│ Stat        │ 1%    │ 2.5%  │ 50%   │ 97.5% │ Avg     │ Stdev  │ Min   │
├─────────────┼───────┼───────┼───────┼───────┼─────────┼────────┼───────┤
│ Clients/Sec │ 4,018 │ 4,931 │ 5,930 │ 6,672 │ 5,911.1 │ 408.05 │ 4,018 │
└─────────────┴───────┴───────┴───────┴───────┴─────────┴────────┴───────┘

Time from the start of the connection to each step. Clients counted once per second.
# of samples: 60

355k clients let in in 60.06s, 0 refused or lost

Connect : both tokens

A client connects with x-am-authorization and x-am-user-authorization, is told CONNECTED, and leaves. Both tokens are checked : the one of the API user, and the one of a person, made by the DB token generator of the account.

Clients / second 5,302 318,458 clients in 60 s, 100 at a time
Let in after, p50 10.5ms connected 10.5 ms · TCP 0.37 ms
Let in after, p99 17.7ms max 39.5 ms
Refused or lost 0 not connected, not subscribed, no answer

Server during this run API Maker 0.9 vCPU · whole server busy 100 % · load generator busy 56 %

am_ws_load · Connect : both tokens · 60 s
$ am_ws_load churn -c 100 -d 60 "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
Running 60s test @ ws://<server>:38246
100 clients at a time : connect, CONNECTED, leave

┌─────────────┬──────────┬──────────┬─────────┬──────────┬──────────┬─────────┬──────────┐
│ Stat        │ 2.5%     │ 50%      │ 97.5%   │ 99%      │ Avg      │ Stdev   │ Max      │
├─────────────┼──────────┼──────────┼─────────┼──────────┼──────────┼─────────┼──────────┤
│ TCP connect │ 0.27 ms  │ 0.37 ms  │ 0.97 ms │ 1.23 ms  │ 0.44 ms  │ 0.21 ms │ 5.34 ms  │
├─────────────┼──────────┼──────────┼─────────┼──────────┼──────────┼─────────┼──────────┤
│ CONNECTED   │ 5.22 ms  │ 10.49 ms │ 16 ms   │ 17.71 ms │ 10.56 ms │ 2.6 ms  │ 39.55 ms │
├─────────────┼──────────┼──────────┼─────────┼──────────┼──────────┼─────────┼──────────┤
│ Left        │ 12.06 ms │ 18.77 ms │ 26.1 ms │ 29.02 ms │ 18.82 ms │ 3.43 ms │ 50.37 ms │
└─────────────┴──────────┴──────────┴─────────┴──────────┴──────────┴─────────┴──────────┘
┌─────────────┬───────┬───────┬───────┬───────┬─────────┬────────┬───────┐
│ Stat        │ 1%    │ 2.5%  │ 50%   │ 97.5% │ Avg     │ Stdev  │ Min   │
├─────────────┼───────┼───────┼───────┼───────┼─────────┼────────┼───────┤
│ Clients/Sec │ 4,092 │ 4,682 │ 5,315 │ 5,751 │ 5,307.6 │ 292.19 │ 4,092 │
└─────────────┴───────┴───────┴───────┴───────┴─────────┴────────┴───────┘

Time from the start of the connection to each step. Clients counted once per second.
# of samples: 60

318k clients let in in 60.06s, 0 refused or lost

Connect with both tokens and subscribe

The whole of what an application does : connect with both tokens, CONNECTED, one REGISTER for a custom WebSocket event with a condition of its own (recipientId : its user id), the answer, then leave. API Maker checks the groups and the auth provider of the event, keeps the subscription in Redis and listens on its channel ; when the client leaves, all of it is taken back.

Clients / second 2,602 156,295 clients in 60 s, 100 at a time
Let in after, p50 33.2ms connected 8.00 ms · TCP 0.34 ms
Let in after, p99 47.5ms max 65.7 ms
Refused or lost 0 not connected, not subscribed, no answer

Server during this run API Maker 0.7 vCPU · Redis 0.1 vCPU · whole server busy 100 % · load generator busy 33 %

am_ws_load · Connect with both tokens and subscribe · 60 s
$ am_ws_load churn -c 100 -d 60 "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>" --register 'order-updates'
Running 60s test @ ws://<server>:38246
100 clients at a time : connect, CONNECTED, REGISTER, leave

┌─────────────┬──────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐
│ Stat        │ 2.5%     │ 50%      │ 97.5%    │ 99%      │ Avg      │ Stdev   │ Max      │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ TCP connect │ 0.27 ms  │ 0.34 ms  │ 0.66 ms  │ 0.77 ms  │ 0.37 ms  │ 0.12 ms │ 4.08 ms  │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ CONNECTED   │ 3.86 ms  │ 8 ms     │ 14.14 ms │ 15.9 ms  │ 8.26 ms  │ 2.58 ms │ 27.74 ms │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ REGISTER    │ 23.02 ms │ 33.22 ms │ 44.22 ms │ 47.49 ms │ 33.27 ms │ 5.18 ms │ 65.66 ms │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ Left        │ 27.95 ms │ 38.27 ms │ 49.89 ms │ 53.54 ms │ 38.39 ms │ 5.29 ms │ 71.55 ms │
└─────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘
┌─────────────┬───────┬───────┬───────┬───────┬─────────┬────────┬───────┐
│ Stat        │ 1%    │ 2.5%  │ 50%   │ 97.5% │ Avg     │ Stdev  │ Min   │
├─────────────┼───────┼───────┼───────┼───────┼─────────┼────────┼───────┤
│ Clients/Sec │ 2,086 │ 2,400 │ 2,625 │ 2,763 │ 2,604.9 │ 110.13 │ 2,086 │
└─────────────┴───────┴───────┴───────┴───────┴─────────┴────────┴───────┘

Time from the start of the connection to each step. Clients counted once per second.
# of samples: 60

156k clients let in in 60.06s, 0 refused or lost

Subscribe to an event with a can user connect code

The same, on an event whose can user connect code decides who may subscribe : it reads the person of x-am-user-authorization from g.req.auth. The code ran in the server itself (lean code, QuickJS).

Clients / second 2,097 125,939 clients in 60 s, 100 at a time
Let in after, p50 41.7ms connected 9.86 ms · TCP 0.35 ms
Let in after, p99 57.8ms max 78.8 ms
Refused or lost 0 not connected, not subscribed, no answer

Server during this run API Maker 0.7 vCPU · Redis 0.1 vCPU · whole server busy 100 % · load generator busy 27 %

am_ws_load · Subscribe to an event with a can user connect code · 60 s
$ am_ws_load churn -c 100 -d 60 "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>" --register 'checked-token'
Running 60s test @ ws://<server>:38246
100 clients at a time : connect, CONNECTED, REGISTER, leave

┌─────────────┬──────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐
│ Stat        │ 2.5%     │ 50%      │ 97.5%    │ 99%      │ Avg      │ Stdev   │ Max      │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ TCP connect │ 0.27 ms  │ 0.35 ms  │ 0.7 ms   │ 0.84 ms  │ 0.39 ms  │ 0.16 ms │ 8.18 ms  │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ CONNECTED   │ 4.36 ms  │ 9.86 ms  │ 17.63 ms │ 20.43 ms │ 10.1 ms  │ 3.35 ms │ 42.11 ms │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ REGISTER    │ 29.01 ms │ 41.73 ms │ 54.37 ms │ 57.76 ms │ 41.72 ms │ 6.21 ms │ 78.78 ms │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ Left        │ 35.01 ms │ 47.49 ms │ 61.06 ms │ 64.96 ms │ 47.65 ms │ 6.42 ms │ 88.58 ms │
└─────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘
┌─────────────┬───────┬───────┬───────┬───────┬─────────┬───────┬───────┐
│ Stat        │ 1%    │ 2.5%  │ 50%   │ 97.5% │ Avg     │ Stdev │ Min   │
├─────────────┼───────┼───────┼───────┼───────┼─────────┼───────┼───────┤
│ Clients/Sec │ 1,794 │ 1,850 │ 2,101 │ 2,220 │ 2,099.0 │ 86.01 │ 1,794 │
└─────────────┴───────┴───────┴───────┴───────┴─────────┴───────┴───────┘

Time from the start of the connection to each step. Clients counted once per second.
# of samples: 60

126k clients let in in 60.06s, 0 refused or lost

Subscribe : the code reads the database

The same, with a can user connect code which also reads the person from the database (g.sys.db.getAll with a find) before it answers : one read of MongoDB for every subscription. The code ran in a sandbox of the account.

Clients / second 959 57,569 clients in 60 s, 100 at a time
Let in after, p50 93.8ms connected 14.0 ms · TCP 0.35 ms
Let in after, p99 144ms max 194 ms
Refused or lost 0 not connected, not subscribed, no answer

Server during this run API Maker 0.3 vCPU · Redis 0.1 vCPU · MongoDB 0.2 vCPU · sandboxes 0.3 vCPU · whole server busy 100 % · load generator busy 13 %

am_ws_load · Subscribe : the code reads the database · 60 s
$ am_ws_load churn -c 100 -d 60 "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>" --register 'checked-database'
Running 60s test @ ws://<server>:38246
100 clients at a time : connect, CONNECTED, REGISTER, leave

┌─────────────┬──────────┬───────────┬───────────┬───────────┬──────────┬──────────┬───────────┐
│ Stat        │ 2.5%     │ 50%       │ 97.5%     │ 99%       │ Avg      │ Stdev    │ Max       │
├─────────────┼──────────┼───────────┼───────────┼───────────┼──────────┼──────────┼───────────┤
│ TCP connect │ 0.27 ms  │ 0.35 ms   │ 0.62 ms   │ 0.7 ms    │ 0.38 ms  │ 0.1 ms   │ 4.28 ms   │
├─────────────┼──────────┼───────────┼───────────┼───────────┼──────────┼──────────┼───────────┤
│ CONNECTED   │ 6.48 ms  │ 14.03 ms  │ 24.3 ms   │ 27.14 ms  │ 14.41 ms │ 4.67 ms  │ 83.14 ms  │
├─────────────┼──────────┼───────────┼───────────┼───────────┼──────────┼──────────┼───────────┤
│ REGISTER    │ 65.5 ms  │ 93.82 ms  │ 128.06 ms │ 143.87 ms │ 94.41 ms │ 15.51 ms │ 194.43 ms │
├─────────────┼──────────┼───────────┼───────────┼───────────┼──────────┼──────────┼───────────┤
│ Left        │ 74.62 ms │ 103.74 ms │ 139.39 ms │ 157.18 ms │ 104.3 ms │ 16.16 ms │ 224.38 ms │
└─────────────┴──────────┴───────────┴───────────┴───────────┴──────────┴──────────┴───────────┘
┌─────────────┬─────┬──────┬─────┬───────┬───────┬───────┬─────┐
│ Stat        │ 1%  │ 2.5% │ 50% │ 97.5% │ Avg   │ Stdev │ Min │
├─────────────┼─────┼──────┼─────┼───────┼───────┼───────┼─────┤
│ Clients/Sec │ 632 │ 727  │ 970 │ 1,026 │ 959.5 │ 65.68 │ 632 │
└─────────────┴─────┴──────┴─────┴───────┴───────┴───────┴─────┘

Time from the start of the connection to each step. Clients counted once per second.
# of samples: 60

58k clients let in in 60.06s, 0 refused or lost

Subscribe : the code reads the clock

The same, with a can user connect code which reads the time of the machine (Date.now()) : code which is not lean, so API Maker runs it in a sandbox of the account. The code ran in a sandbox of the account.

Clients / second 1,960 117,720 clients in 60 s, 100 at a time
Let in after, p50 45.5ms connected 8.11 ms · TCP 0.35 ms
Let in after, p99 65.1ms max 88.3 ms
Refused or lost 0 not connected, not subscribed, no answer

Server during this run API Maker 0.6 vCPU · Redis 0.1 vCPU · sandboxes 0.1 vCPU · whole server busy 100 % · load generator busy 32 %

am_ws_load · Subscribe : the code reads the clock · 60 s
$ am_ws_load churn -c 100 -d 60 "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>" --register 'checked-sandbox'
Running 60s test @ ws://<server>:38246
100 clients at a time : connect, CONNECTED, REGISTER, leave

┌─────────────┬──────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐
│ Stat        │ 2.5%     │ 50%      │ 97.5%    │ 99%      │ Avg      │ Stdev   │ Max      │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ TCP connect │ 0.27 ms  │ 0.35 ms  │ 0.66 ms  │ 0.76 ms  │ 0.38 ms  │ 0.12 ms │ 3.97 ms  │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ CONNECTED   │ 4.31 ms  │ 8.11 ms  │ 12.94 ms │ 14.61 ms │ 8.3 ms   │ 2.21 ms │ 32.58 ms │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ REGISTER    │ 32.35 ms │ 45.47 ms │ 59.81 ms │ 65.12 ms │ 45.66 ms │ 6.83 ms │ 88.32 ms │
├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ Left        │ 37.5 ms  │ 50.78 ms │ 65.66 ms │ 71.68 ms │ 51 ms    │ 7 ms    │ 96.83 ms │
└─────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘
┌─────────────┬───────┬───────┬───────┬───────┬─────────┬───────┬───────┐
│ Stat        │ 1%    │ 2.5%  │ 50%   │ 97.5% │ Avg     │ Stdev │ Min   │
├─────────────┼───────┼───────┼───────┼───────┼─────────┼───────┼───────┤
│ Clients/Sec │ 1,665 │ 1,673 │ 1,957 │ 2,132 │ 1,962.0 │ 86.63 │ 1,665 │
└─────────────┴───────┴───────┴───────┴───────┴─────────┴───────┴───────┘

Time from the start of the connection to each step. Clients counted once per second.
# of samples: 60

118k clients let in in 60.06s, 0 refused or lost

Can user connect code

An event of the WebSocket Events page can carry a can user connect code : API Maker runs it for every subscription to the event, with the tokens of the client in g.req.auth and the subscription in g.req.eventData, and lets the client subscribe when it returns { canConnect: true }. Three codes were measured : one which reads the person of the token, one which also reads the database, one which reads the clock.

Can user connect code of the event checked-token : ran in the server
import * as T from 'types';

async function main(g: T.IAMGlobal) {
    // The person of x-am-user-authorization : only somebody who signed in subscribes, from a country we serve.
    const person = g.req.auth.authAMDB;
    if (!person) return { canConnect: false, errorText: 'Sign in first.' };
    return { canConnect: person.country === 'India', errorText: 'Not available in your country.' };
};
module.exports = main;
Can user connect code of the event checked-database : ran in the sandbox
import * as T from 'types';

async function main(g: T.IAMGlobal) {
    const person = g.req.auth.authAMDB;
    if (!person) return { canConnect: false, errorText: 'Sign in first.' };
    // Still a customer ? One read of the database for every subscription.
    const rows = await g.sys.db.getAll({ instance: 'mongodb', database: 'am_bench', collection: 'customers', queryParams: { find: { customer_id: person.customer_id } } });
    return { canConnect: rows.length === 1 && rows[0].country === 'India', errorText: 'Not a customer any more.' };
};
module.exports = main;
Can user connect code of the event checked-sandbox : ran in the sandbox
import * as T from 'types';

async function main(g: T.IAMGlobal) {
    const person = g.req.auth.authAMDB;
    if (!person) return { canConnect: false, errorText: 'Sign in first.' };
    // The time of the machine : this code runs in the sandbox of the account.
    const hour = new Date(Date.now()).getUTCHours();
    return { canConnect: person.country === 'India' && hour >= 0, errorText: 'Not available in your country.' };
};
module.exports = main;
Can user connect code Runs in Clients / second Subscribed after, p50 p99 Against no code
No code – 2,602 33.2 ms 47.5 ms –
Reads the person of the token the server (QuickJS) 2,097 41.7 ms 57.8 ms 19 % fewer
Reads the person from the database a sandbox of the account 959 93.8 ms 144 ms 63 % fewer
Reads the clock a sandbox of the account 1,960 45.5 ms 65.1 ms 25 % fewer

Lean code (it only reads what it is handed) runs in the server itself, with QuickJS, on the thread which serves the client. Code which needs more runs in a sandbox of the account : every subscription is then a round trip to the sandbox.

Notifications

With 5,000 clients connected, autocannon makes the HTTP calls which notify them, for 60 seconds, and the clients count what they receive and how late. Each kind of notification is measured twice : at a pace far under what the server takes, which shows how long a notification takes, and with no pace, which shows how many a second the server tells. Delivery is the time from the moment autocannon writes the request to the moment the client reads the notification : the HTTP call, API Maker, Redis, and the WebSocket.

A notification for one person

autocannon calls the emit event WS API with both tokens ; every call names another one of the 5,000 connected clients (recipientId), which is the only one told. The clients measure how long each notification took from the moment its request was written.

Calls / second (paced) 1,000 60,000 calls in 60 s, 10 connections
Notifications / second 983 60,009 received by 5,000 clients
Delivery p50 2.51ms average 2.50 ms
Delivery p99 3.82ms max 8.06 ms
Not delivered 0 0 calls failed

Server during this run API Maker 0.2 vCPU · Redis 0.0 vCPU · whole server busy 29 % · load generator busy 15 %

autocannon · A notification for one person, 1,000 a second

$ autocannon -c 10 -d 60 -R 1000 -l -m POST -b '{"name":"order-updates","eventData":{"recipientId":"<id of a client>","t":<now>,"type":"order-updated","order_no":1001,"status":"SHIPPED"}}' -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" -H "content-type: application/json" "http://<server>:38246/api/system-api/bench/emit-event-ws"
Running 60s test @ http://<server>:38246/api/system-api/bench/emit-event-ws
10 connections

┌─────────┬──────┬──────┬───────┬──────┬─────────┬─────────┬───────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%  │ Avg     │ Stdev   │ Max   │
├─────────┼──────┼──────┼───────┼──────┼─────────┼─────────┼───────┤
│ Latency │ 1 ms │ 2 ms │ 3 ms  │ 3 ms │ 1.69 ms │ 0.73 ms │ 10 ms │
└─────────┴──────┴──────┴───────┴──────┴─────────┴─────────┴───────┘
┌───────────┬────────┬────────┬────────┬────────┬────────┬───────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%    │ 97.5%  │ Avg    │ Stdev │ Min    │
├───────────┼────────┼────────┼────────┼────────┼────────┼───────┼────────┤
│ Req/Sec   │ 1,000  │ 1,000  │ 1,000  │ 1,000  │ 1,000  │ 0     │ 1,000  │
├───────────┼────────┼────────┼────────┼────────┼────────┼───────┼────────┤
│ Bytes/Sec │ 357 kB │ 357 kB │ 357 kB │ 357 kB │ 357 kB │ 0 B   │ 357 kB │
└───────────┴────────┴────────┴────────┴────────┴────────┴───────┴────────┘

Req/Bytes counts sampled once per second.
# of samples: 60

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 0            │
├────────────┼──────────────┤
│ 0.01       │ 0            │
├────────────┼──────────────┤
│ 0.1        │ 0            │
├────────────┼──────────────┤
│ 1          │ 1            │
├────────────┼──────────────┤
│ 2.5        │ 1            │
├────────────┼──────────────┤
│ 10         │ 1            │
├────────────┼──────────────┤
│ 25         │ 1            │
├────────────┼──────────────┤
│ 50         │ 2            │
├────────────┼──────────────┤
│ 75         │ 2            │
├────────────┼──────────────┤
│ 90         │ 3            │
├────────────┼──────────────┤
│ 97.5       │ 3            │
├────────────┼──────────────┤
│ 99         │ 3            │
├────────────┼──────────────┤
│ 99.9       │ 4            │
├────────────┼──────────────┤
│ 99.99      │ 7            │
├────────────┼──────────────┤
│ 99.999     │ 9            │
└────────────┴──────────────┘

61k requests in 60.05s, 21.4 MB read

am_ws_load · what the clients received

$ am_ws_load listen "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
5,000 clients subscribed, 64.07s

┌──────────┬─────────┬─────────┬─────────┬─────────┬────────┬─────────┬─────────┐
│ Stat     │ 2.5%    │ 50%     │ 97.5%   │ 99%     │ Avg    │ Stdev   │ Max     │
├──────────┼─────────┼─────────┼─────────┼─────────┼────────┼─────────┼─────────┤
│ Delivery │ 1.48 ms │ 2.51 ms │ 3.54 ms │ 3.82 ms │ 2.5 ms │ 0.54 ms │ 8.06 ms │
└──────────┴─────────┴─────────┴─────────┴─────────┴────────┴─────────┴─────────┘
┌───────────┬────┬──────┬───────┬───────┬───────┬────────┬─────┐
│ Stat      │ 1% │ 2.5% │ 50%   │ 97.5% │ Avg   │ Stdev  │ Min │
├───────────┼────┼──────┼───────┼───────┼───────┼────────┼─────┤
│ Notif/Sec │ 0  │ 0    │ 1,000 │ 1,000 │ 951.8 │ 213.94 │ 0   │
└───────────┴────┴──────┴───────┴───────┴───────┴────────┴─────┘

Delivery : from the moment the request is written to the moment the client reads the notification.
# of samples: 64

60k notifications received in 64.07s (60,009 of 60,000 expected), 10.7 MB read

Calls / second 5,285 317,042 calls in 60 s, 100 connections
Notifications / second 5,216 317,140 received by 5,000 clients
Delivery p50 18.6ms average 18.8 ms
Delivery p99 30.0ms max 65.6 ms
Not delivered 0 0 calls failed

Server during this run API Maker 0.8 vCPU · Redis 0.1 vCPU · whole server busy 100 % · load generator busy 42 %

autocannon · A notification for one person, as many as the server takes

$ autocannon -c 100 -d 60 -w 4 -l -m POST -b '{"name":"order-updates","eventData":{"recipientId":"<id of a client>","t":<now>,"type":"order-updated","order_no":1001,"status":"SHIPPED"}}' -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" -H "content-type: application/json" "http://<server>:38246/api/system-api/bench/emit-event-ws"
Running 60s test @ http://<server>:38246/api/system-api/bench/emit-event-ws
100 connections
4 workers

┌─────────┬───────┬───────┬───────┬───────┬──────────┬─────────┬───────┐
│ Stat    │ 2.5%  │ 50%   │ 97.5% │ 99%   │ Avg      │ Stdev   │ Max   │
├─────────┼───────┼───────┼───────┼───────┼──────────┼─────────┼───────┤
│ Latency │ 12 ms │ 18 ms │ 26 ms │ 30 ms │ 18.38 ms │ 3.72 ms │ 65 ms │
└─────────┴───────┴───────┴───────┴───────┴──────────┴─────────┴───────┘
┌───────────┬─────────┬─────────┬────────┬─────────┬──────────┬────────┬─────────┐
│ Stat      │ 1%      │ 2.5%    │ 50%    │ 97.5%   │ Avg      │ Stdev  │ Min     │
├───────────┼─────────┼─────────┼────────┼─────────┼──────────┼────────┼─────────┤
│ Req/Sec   │ 3,957   │ 4,571   │ 5,315  │ 5,679   │ 5,284.52 │ 290.39 │ 3,956   │
├───────────┼─────────┼─────────┼────────┼─────────┼──────────┼────────┼─────────┤
│ Bytes/Sec │ 1.41 MB │ 1.63 MB │ 1.9 MB │ 2.03 MB │ 1.89 MB  │ 104 kB │ 1.41 MB │
└───────────┴─────────┴─────────┴────────┴─────────┴──────────┴────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 240

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 4            │
├────────────┼──────────────┤
│ 0.01       │ 5            │
├────────────┼──────────────┤
│ 0.1        │ 7            │
├────────────┼──────────────┤
│ 1          │ 9            │
├────────────┼──────────────┤
│ 2.5        │ 12           │
├────────────┼──────────────┤
│ 10         │ 14           │
├────────────┼──────────────┤
│ 25         │ 16           │
├────────────┼──────────────┤
│ 50         │ 18           │
├────────────┼──────────────┤
│ 75         │ 20           │
├────────────┼──────────────┤
│ 90         │ 22           │
├────────────┼──────────────┤
│ 97.5       │ 26           │
├────────────┼──────────────┤
│ 99         │ 30           │
├────────────┼──────────────┤
│ 99.9       │ 50           │
├────────────┼──────────────┤
│ 99.99      │ 60           │
├────────────┼──────────────┤
│ 99.999     │ 62           │
└────────────┴──────────────┘

317k requests in 60.04s, 113 MB read

am_ws_load · what the clients received

$ am_ws_load listen "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
5,000 clients subscribed, 63.81s

┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬─────────┬─────────┐
│ Stat     │ 2.5%     │ 50%      │ 97.5%    │ 99%      │ Avg      │ Stdev   │ Max     │
├──────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼─────────┤
│ Delivery │ 11.98 ms │ 18.64 ms │ 26.26 ms │ 30.03 ms │ 18.77 ms │ 3.68 ms │ 65.6 ms │
└──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴─────────┴─────────┘
┌───────────┬────┬──────┬───────┬───────┬─────────┬────────┬─────┐
│ Stat      │ 1% │ 2.5% │ 50%   │ 97.5% │ Avg     │ Stdev  │ Min │
├───────────┼────┼──────┼───────┼───────┼─────────┼────────┼─────┤
│ Notif/Sec │ 0  │ 0    │ 5,336 │ 5,634 │ 5,080.5 │ 998.31 │ 0   │
└───────────┴────┴──────┴───────┴───────┴─────────┴────────┴─────┘

Delivery : from the moment the request is written to the moment the client reads the notification.
# of samples: 64

317k notifications received in 63.81s (317,140 of 317,042 expected), 56.8 MB read

A broadcast

1,000 more clients subscribe to an event without a condition : every call of the emit event WS API reaches all 1,000 of them.

Calls / second (paced) 5 300 calls in 60 s, 1 connection
Notifications / second 4,931 301,000 received by 1,000 clients
Delivery p50 10.3ms average 11.1 ms
Delivery p99 21.6ms max 29.4 ms
Not delivered 0 0 calls failed

Server during this run API Maker 0.1 vCPU · whole server busy 10 % · load generator busy 15 %

autocannon · A broadcast to 1,000 clients, 5 a second

$ autocannon -c 1 -d 60 -R 5 -l -m POST -b '{"name":"announcements","eventData":{"t":<now>,"type":"announcement","text":"Hello from API Maker"}}' -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" -H "content-type: application/json" "http://<server>:38246/api/system-api/bench/emit-event-ws"
Running 60s test @ http://<server>:38246/api/system-api/bench/emit-event-ws
1 connections

┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬───────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%   │ Avg     │ Stdev   │ Max   │
├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼───────┤
│ Latency │ 1 ms │ 9 ms │ 20 ms │ 22 ms │ 9.24 ms │ 5.49 ms │ 29 ms │
└─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴───────┘
┌───────────┬────────┬────────┬────────┬────────┬────────┬───────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%    │ 97.5%  │ Avg    │ Stdev │ Min    │
├───────────┼────────┼────────┼────────┼────────┼────────┼───────┼────────┤
│ Req/Sec   │ 5      │ 5      │ 5      │ 5      │ 5      │ 0     │ 5      │
├───────────┼────────┼────────┼────────┼────────┼────────┼───────┼────────┤
│ Bytes/Sec │ 1.8 kB │ 1.8 kB │ 1.8 kB │ 1.8 kB │ 1.8 kB │ 0 B   │ 1.8 kB │
└───────────┴────────┴────────┴────────┴────────┴────────┴───────┴────────┘

Req/Bytes counts sampled once per second.
# of samples: 60

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 1            │
├────────────┼──────────────┤
│ 0.01       │ 1            │
├────────────┼──────────────┤
│ 0.1        │ 1            │
├────────────┼──────────────┤
│ 1          │ 1            │
├────────────┼──────────────┤
│ 2.5        │ 1            │
├────────────┼──────────────┤
│ 10         │ 2            │
├────────────┼──────────────┤
│ 25         │ 5            │
├────────────┼──────────────┤
│ 50         │ 9            │
├────────────┼──────────────┤
│ 75         │ 13           │
├────────────┼──────────────┤
│ 90         │ 17           │
├────────────┼──────────────┤
│ 97.5       │ 20           │
├────────────┼──────────────┤
│ 99         │ 22           │
├────────────┼──────────────┤
│ 99.9       │ 26           │
├────────────┼──────────────┤
│ 99.99      │ 29           │
├────────────┼──────────────┤
│ 99.999     │ 29           │
└────────────┴──────────────┘

305 requests in 60.03s, 108 kB read

am_ws_load · what the clients received

$ am_ws_load listen "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
1,000 clients subscribed, 64.05s

┌──────────┬─────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐
│ Stat     │ 2.5%    │ 50%      │ 97.5%    │ 99%      │ Avg      │ Stdev   │ Max      │
├──────────┼─────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ Delivery │ 5.28 ms │ 10.33 ms │ 20.14 ms │ 21.57 ms │ 11.05 ms │ 4.08 ms │ 29.41 ms │
└──────────┴─────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘
┌───────────┬────┬──────┬───────┬───────┬─────────┬─────────┬─────┐
│ Stat      │ 1% │ 2.5% │ 50%   │ 97.5% │ Avg     │ Stdev   │ Min │
├───────────┼────┼──────┼───────┼───────┼─────────┼─────────┼─────┤
│ Notif/Sec │ 0  │ 0    │ 5,000 │ 5,000 │ 4,774.2 │ 1006.74 │ 0   │
└───────────┴────┴──────┴───────┴───────┴─────────┴─────────┴─────┘

Delivery : from the moment the request is written to the moment the client reads the notification.
# of samples: 64

301k notifications received in 64.05s (301,000 of 300,000 expected), 46.2 MB read

Calls / second 73 4,376 calls in 60 s, 10 connections
Notifications / second 72,245 4,385,000 received by 1,000 clients
Delivery p50 122ms average 124 ms
Delivery p99 204ms max 333 ms
Not delivered 0 0 calls failed

Server during this run API Maker 0.8 vCPU · Redis 0.1 vCPU · whole server busy 100 % · load generator busy 63 %

autocannon · A broadcast to 1,000 clients, as many as the server takes

$ autocannon -c 10 -d 60 -w 2 -l -m POST -b '{"name":"announcements","eventData":{"t":<now>,"type":"announcement","text":"Hello from API Maker"}}' -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" -H "content-type: application/json" "http://<server>:38246/api/system-api/bench/emit-event-ws"
Running 60s test @ http://<server>:38246/api/system-api/bench/emit-event-ws
10 connections
2 workers

┌─────────┬───────┬────────┬────────┬────────┬───────────┬──────────┬────────┐
│ Stat    │ 2.5%  │ 50%    │ 97.5%  │ 99%    │ Avg       │ Stdev    │ Max    │
├─────────┼───────┼────────┼────────┼────────┼───────────┼──────────┼────────┤
│ Latency │ 79 ms │ 135 ms │ 199 ms │ 215 ms │ 136.48 ms │ 29.92 ms │ 332 ms │
└─────────┴───────┴────────┴────────┴────────┴───────────┴──────────┴────────┘
┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ Stat      │ 1%      │ 2.5%    │ 50%     │ 97.5%   │ Avg     │ Stdev   │ Min     │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Req/Sec   │ 56      │ 59      │ 73      │ 87      │ 72.94   │ 6.89    │ 56      │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Bytes/Sec │ 20.2 kB │ 21.2 kB │ 26.3 kB │ 31.3 kB │ 26.3 kB │ 2.48 kB │ 20.2 kB │
└───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 120

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 34           │
├────────────┼──────────────┤
│ 0.01       │ 34           │
├────────────┼──────────────┤
│ 0.1        │ 47           │
├────────────┼──────────────┤
│ 1          │ 69           │
├────────────┼──────────────┤
│ 2.5        │ 79           │
├────────────┼──────────────┤
│ 10         │ 101          │
├────────────┼──────────────┤
│ 25         │ 117          │
├────────────┼──────────────┤
│ 50         │ 135          │
├────────────┼──────────────┤
│ 75         │ 155          │
├────────────┼──────────────┤
│ 90         │ 174          │
├────────────┼──────────────┤
│ 97.5       │ 199          │
├────────────┼──────────────┤
│ 99         │ 215          │
├────────────┼──────────────┤
│ 99.9       │ 266          │
├────────────┼──────────────┤
│ 99.99      │ 332          │
├────────────┼──────────────┤
│ 99.999     │ 332          │
└────────────┴──────────────┘

4k requests in 60.03s, 1.58 MB read

am_ws_load · what the clients received

$ am_ws_load listen "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
1,000 clients subscribed, 63.70s

┌──────────┬──────────┬───────────┬───────────┬──────────┬───────────┬─────────┬───────────┐
│ Stat     │ 2.5%     │ 50%       │ 97.5%     │ 99%      │ Avg       │ Stdev   │ Max       │
├──────────┼──────────┼───────────┼───────────┼──────────┼───────────┼─────────┼───────────┤
│ Delivery │ 70.78 ms │ 122.37 ms │ 186.37 ms │ 203.9 ms │ 124.18 ms │ 29.3 ms │ 332.54 ms │
└──────────┴──────────┴───────────┴───────────┴──────────┴───────────┴─────────┴───────────┘
┌───────────┬────┬──────┬────────┬────────┬──────────┬──────────┬─────┐
│ Stat      │ 1% │ 2.5% │ 50%    │ 97.5%  │ Avg      │ Stdev    │ Min │
├───────────┼────┼──────┼────────┼────────┼──────────┼──────────┼─────┤
│ Notif/Sec │ 0  │ 0    │ 73,403 │ 84,274 │ 70,307.9 │ 14890.79 │ 0   │
└───────────┴────┴──────┴────────┴────────┴──────────┴──────────┴─────┘

Delivery : from the moment the request is written to the moment the client reads the notification.
# of samples: 64

4.38M notifications received in 63.70s (4,385,000 of 4,376,000 expected), 672.8 MB read

Rows inserted in a table

100 clients subscribe to the insert API of a table (SCHEMA_POST_BULK_INSERT on inserts_public, getEventData: true) ; autocannon inserts rows of the dataset through the insert API, and every row saved is told to each of the 100 clients, with its columns.

Calls / second (paced) 50 3,000 calls in 60 s, 10 connections
Notifications / second 4,930 301,000 received by 100 clients
Delivery p50 51.3ms average 53.4 ms
Delivery p99 103ms max 127 ms
Not delivered 0 0 calls failed

Server during this run API Maker 0.1 vCPU · Redis 0.0 vCPU · MongoDB 0.1 vCPU · whole server busy 21 % · load generator busy 13 %

autocannon · Inserts told to 100 clients, 50 inserts a second

$ autocannon -c 10 -d 60 -R 50 -l -m POST -b '{"customer_id":1,"first_name":"BENCH","last_name":"INSERT","phone":<now>,"last_update":"2006-02-15T04:34:00.330Z","pincode":111111,"shipping_id":2,"dob":"2022-10-30T03:25:03.000Z","country":"India"}' -H "content-type: application/json" "http://<server>:38246/api/schema/bench/mongodb/am_bench/inserts_public/save-single-or-multiple"
Running 60s test @ http://<server>:38246/api/schema/bench/mongodb/am_bench/inserts_public/save-single-or-multiple
10 connections

┌─────────┬──────┬───────┬───────┬───────┬──────────┬──────────┬───────┐
│ Stat    │ 2.5% │ 50%   │ 97.5% │ 99%   │ Avg      │ Stdev    │ Max   │
├─────────┼──────┼───────┼───────┼───────┼──────────┼──────────┼───────┤
│ Latency │ 1 ms │ 17 ms │ 51 ms │ 58 ms │ 19.31 ms │ 13.54 ms │ 92 ms │
└─────────┴──────┴───────┴───────┴───────┴──────────┴──────────┴───────┘
┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬────────┬─────────┐
│ Stat      │ 1%      │ 2.5%    │ 50%     │ 97.5%   │ Avg     │ Stdev  │ Min     │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┼─────────┤
│ Req/Sec   │ 50      │ 50      │ 50      │ 50      │ 50      │ 0      │ 50      │
├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┼─────────┤
│ Bytes/Sec │ 30.2 kB │ 30.2 kB │ 30.3 kB │ 30.3 kB │ 30.2 kB │ 3.49 B │ 30.2 kB │
└───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 60

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 1            │
├────────────┼──────────────┤
│ 0.01       │ 1            │
├────────────┼──────────────┤
│ 0.1        │ 1            │
├────────────┼──────────────┤
│ 1          │ 1            │
├────────────┼──────────────┤
│ 2.5        │ 1            │
├────────────┼──────────────┤
│ 10         │ 4            │
├────────────┼──────────────┤
│ 25         │ 8            │
├────────────┼──────────────┤
│ 50         │ 17           │
├────────────┼──────────────┤
│ 75         │ 28           │
├────────────┼──────────────┤
│ 90         │ 38           │
├────────────┼──────────────┤
│ 97.5       │ 51           │
├────────────┼──────────────┤
│ 99         │ 58           │
├────────────┼──────────────┤
│ 99.9       │ 71           │
├────────────┼──────────────┤
│ 99.99      │ 89           │
├────────────┼──────────────┤
│ 99.999     │ 92           │
└────────────┴──────────────┘

3k requests in 60.04s, 1.81 MB read

am_ws_load · what the clients received

$ am_ws_load listen "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
100 clients subscribed, 64.06s

┌──────────┬──────────┬──────────┬──────────┬───────────┬──────────┬──────────┬──────────┐
│ Stat     │ 2.5%     │ 50%      │ 97.5%    │ 99%       │ Avg      │ Stdev    │ Max      │
├──────────┼──────────┼──────────┼──────────┼───────────┼──────────┼──────────┼──────────┤
│ Delivery │ 19.22 ms │ 51.33 ms │ 94.59 ms │ 103.36 ms │ 53.35 ms │ 18.86 ms │ 127.3 ms │
└──────────┴──────────┴──────────┴──────────┴───────────┴──────────┴──────────┴──────────┘
┌───────────┬────┬──────┬───────┬───────┬─────────┬─────────┬─────┐
│ Stat      │ 1% │ 2.5% │ 50%   │ 97.5% │ Avg     │ Stdev   │ Min │
├───────────┼────┼──────┼───────┼───────┼─────────┼─────────┼─────┤
│ Notif/Sec │ 0  │ 0    │ 5,000 │ 5,000 │ 4,774.2 │ 1006.74 │ 0   │
└───────────┴────┴──────┴───────┴───────┴─────────┴─────────┴─────┘

Delivery : from the moment the request is written to the moment the client reads the notification.
# of samples: 64

301k notifications received in 64.06s (301,000 of 300,000 expected), 94.4 MB read

Calls / second 416 24,934 calls in 60 s, 100 connections
Notifications / second 40,498 2,503,300 received by 100 clients
Delivery p50 574ms average 571 ms
Delivery p99 785ms max 993 ms
Not delivered 0 0 calls failed

Server during this run API Maker 0.6 vCPU · Redis 0.1 vCPU · MongoDB 0.2 vCPU · whole server busy 99 % · load generator busy 36 %

autocannon · Inserts told to 100 clients, as many as the server takes

$ autocannon -c 100 -d 60 -w 4 -l -m POST -b '{"customer_id":1,"first_name":"BENCH","last_name":"INSERT","phone":<now>,"last_update":"2006-02-15T04:34:00.330Z","pincode":111111,"shipping_id":2,"dob":"2022-10-30T03:25:03.000Z","country":"India"}' -H "content-type: application/json" "http://<server>:38246/api/schema/bench/mongodb/am_bench/inserts_public/save-single-or-multiple"
Running 60s test @ http://<server>:38246/api/schema/bench/mongodb/am_bench/inserts_public/save-single-or-multiple
100 connections
4 workers

┌─────────┬────────┬────────┬────────┬────────┬──────────┬──────────┬────────┐
│ Stat    │ 2.5%   │ 50%    │ 97.5%  │ 99%    │ Avg      │ Stdev    │ Max    │
├─────────┼────────┼────────┼────────┼────────┼──────────┼──────────┼────────┤
│ Latency │ 155 ms │ 239 ms │ 332 ms │ 363 ms │ 239.7 ms │ 43.51 ms │ 481 ms │
└─────────┴────────┴────────┴────────┴────────┴──────────┴──────────┴────────┘
┌───────────┬────────┬────────┬────────┬────────┬────────┬─────────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%    │ 97.5%  │ Avg    │ Stdev   │ Min    │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Req/Sec   │ 302    │ 339    │ 417    │ 461    │ 415.57 │ 31.84   │ 302    │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Bytes/Sec │ 183 kB │ 205 kB │ 252 kB │ 279 kB │ 251 kB │ 19.3 kB │ 183 kB │
└───────────┴────────┴────────┴────────┴────────┴────────┴─────────┴────────┘

Req/Bytes counts sampled once per second.
# of samples: 240

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 53           │
├────────────┼──────────────┤
│ 0.01       │ 65           │
├────────────┼──────────────┤
│ 0.1        │ 86           │
├────────────┼──────────────┤
│ 1          │ 136          │
├────────────┼──────────────┤
│ 2.5        │ 155          │
├────────────┼──────────────┤
│ 10         │ 189          │
├────────────┼──────────────┤
│ 25         │ 213          │
├────────────┼──────────────┤
│ 50         │ 239          │
├────────────┼──────────────┤
│ 75         │ 265          │
├────────────┼──────────────┤
│ 90         │ 292          │
├────────────┼──────────────┤
│ 97.5       │ 332          │
├────────────┼──────────────┤
│ 99         │ 363          │
├────────────┼──────────────┤
│ 99.9       │ 436          │
├────────────┼──────────────┤
│ 99.99      │ 478          │
├────────────┼──────────────┤
│ 99.999     │ 481          │
└────────────┴──────────────┘

25k requests in 60.05s, 15.1 MB read

am_ws_load · what the clients received

$ am_ws_load listen "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
100 clients subscribed, 64.82s

┌──────────┬───────────┬───────────┬──────────┬───────────┬───────────┬──────────┬───────────┐
│ Stat     │ 2.5%      │ 50%       │ 97.5%    │ 99%       │ Avg       │ Stdev    │ Max       │
├──────────┼───────────┼───────────┼──────────┼───────────┼───────────┼──────────┼───────────┤
│ Delivery │ 410.62 ms │ 574.46 ms │ 720.9 ms │ 785.41 ms │ 571.38 ms │ 78.34 ms │ 992.77 ms │
└──────────┴───────────┴───────────┴──────────┴───────────┴───────────┴──────────┴───────────┘
┌───────────┬────┬──────┬────────┬────────┬──────────┬─────────┬─────┐
│ Stat      │ 1% │ 2.5% │ 50%    │ 97.5%  │ Avg      │ Stdev   │ Min │
├───────────┼────┼──────┼────────┼────────┼──────────┼─────────┼─────┤
│ Notif/Sec │ 0  │ 0    │ 41,651 │ 46,300 │ 39,596.8 │ 9525.96 │ 0   │
└───────────┴────┴──────┴────────┴────────┴──────────┴─────────┴─────┘

Delivery : from the moment the request is written to the moment the client reads the notification.
# of samples: 65

2.50M notifications received in 64.82s (2,503,300 of 2,493,400 expected), 785.2 MB read

With every client connected

The same runs once more, with 26,484 clients connected and waiting : what a server which is full of WebSocket clients still does.

No client, or 5,000 26,484 clients connected
A notification for one person, 1,000 a second : delivery p50 2.51 ms 2.49 ms
… p99 3.82 ms 7.48 ms
A notification for one person, as many as the server takes 5,216 / s 5,226 / s
Get all with both tokens, 100 connections (autocannon) 2,095 req/s 1,983 req/s
… latency p50 / p99 48 ms / 68 ms 50 ms / 92 ms

Calls / second (paced) 1,000 60,000 calls in 60 s, 10 connections
Notifications / second 983 60,009 received by 26,484 clients
Delivery p50 2.49ms average 2.60 ms
Delivery p99 7.48ms max 25.4 ms
Not delivered 0 0 calls failed

Server during this run API Maker 0.2 vCPU · Redis 0.0 vCPU · whole server busy 29 % · load generator busy 15 %

autocannon · A notification for one person, 1,000 a second, every client connected

$ autocannon -c 10 -d 60 -R 1000 -l -m POST -b '{"name":"order-updates","eventData":{"recipientId":"<id of a client>","t":<now>,"type":"order-updated","order_no":1001,"status":"SHIPPED"}}' -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" -H "content-type: application/json" "http://<server>:38246/api/system-api/bench/emit-event-ws"
Running 60s test @ http://<server>:38246/api/system-api/bench/emit-event-ws
10 connections

┌─────────┬──────┬──────┬───────┬──────┬────────┬─────────┬───────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%  │ Avg    │ Stdev   │ Max   │
├─────────┼──────┼──────┼───────┼──────┼────────┼─────────┼───────┤
│ Latency │ 1 ms │ 2 ms │ 5 ms  │ 8 ms │ 1.9 ms │ 1.37 ms │ 25 ms │
└─────────┴──────┴──────┴───────┴──────┴────────┴─────────┴───────┘
┌───────────┬────────┬────────┬────────┬────────┬────────┬───────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%    │ 97.5%  │ Avg    │ Stdev │ Min    │
├───────────┼────────┼────────┼────────┼────────┼────────┼───────┼────────┤
│ Req/Sec   │ 993    │ 994    │ 1,000  │ 1,006  │ 1,000  │ 2.26  │ 993    │
├───────────┼────────┼────────┼────────┼────────┼────────┼───────┼────────┤
│ Bytes/Sec │ 355 kB │ 355 kB │ 357 kB │ 359 kB │ 357 kB │ 805 B │ 355 kB │
└───────────┴────────┴────────┴────────┴────────┴────────┴───────┴────────┘

Req/Bytes counts sampled once per second.
# of samples: 60

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 0            │
├────────────┼──────────────┤
│ 0.01       │ 0            │
├────────────┼──────────────┤
│ 0.1        │ 0            │
├────────────┼──────────────┤
│ 1          │ 1            │
├────────────┼──────────────┤
│ 2.5        │ 1            │
├────────────┼──────────────┤
│ 10         │ 1            │
├────────────┼──────────────┤
│ 25         │ 1            │
├────────────┼──────────────┤
│ 50         │ 2            │
├────────────┼──────────────┤
│ 75         │ 2            │
├────────────┼──────────────┤
│ 90         │ 3            │
├────────────┼──────────────┤
│ 97.5       │ 5            │
├────────────┼──────────────┤
│ 99         │ 8            │
├────────────┼──────────────┤
│ 99.9       │ 14           │
├────────────┼──────────────┤
│ 99.99      │ 23           │
├────────────┼──────────────┤
│ 99.999     │ 25           │
└────────────┴──────────────┘

61k requests in 60.03s, 21.4 MB read

am_ws_load · what the clients received

$ am_ws_load listen "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
26,484 clients subscribed, 64.04s

┌──────────┬─────────┬─────────┬─────────┬─────────┬────────┬─────────┬──────────┐
│ Stat     │ 2.5%    │ 50%     │ 97.5%   │ 99%     │ Avg    │ Stdev   │ Max      │
├──────────┼─────────┼─────────┼─────────┼─────────┼────────┼─────────┼──────────┤
│ Delivery │ 1.47 ms │ 2.49 ms │ 4.32 ms │ 7.48 ms │ 2.6 ms │ 1.03 ms │ 25.44 ms │
└──────────┴─────────┴─────────┴─────────┴─────────┴────────┴─────────┴──────────┘
┌───────────┬────┬──────┬───────┬───────┬───────┬────────┬─────┐
│ Stat      │ 1% │ 2.5% │ 50%   │ 97.5% │ Avg   │ Stdev  │ Min │
├───────────┼────┼──────┼───────┼───────┼───────┼────────┼─────┤
│ Notif/Sec │ 0  │ 0    │ 1,000 │ 1,000 │ 951.8 │ 213.94 │ 0   │
└───────────┴────┴──────┴───────┴───────┴───────┴────────┴─────┘

Delivery : from the moment the request is written to the moment the client reads the notification.
# of samples: 64

60k notifications received in 64.04s (60,009 of 60,000 expected), 10.8 MB read

Calls / second 5,297 317,793 calls in 60 s, 100 connections
Notifications / second 5,226 317,890 received by 26,484 clients
Delivery p50 18.5ms average 18.7 ms
Delivery p99 30.2ms max 91.6 ms
Not delivered 0 0 calls failed

Server during this run API Maker 0.8 vCPU · Redis 0.1 vCPU · whole server busy 100 % · load generator busy 37 %

autocannon · A notification for one person, as many as the server takes, every client connected

$ autocannon -c 100 -d 60 -w 4 -l -m POST -b '{"name":"order-updates","eventData":{"recipientId":"<id of a client>","t":<now>,"type":"order-updated","order_no":1001,"status":"SHIPPED"}}' -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" -H "content-type: application/json" "http://<server>:38246/api/system-api/bench/emit-event-ws"
Running 60s test @ http://<server>:38246/api/system-api/bench/emit-event-ws
100 connections
4 workers

┌─────────┬───────┬───────┬───────┬───────┬──────────┬─────────┬───────┐
│ Stat    │ 2.5%  │ 50%   │ 97.5% │ 99%   │ Avg      │ Stdev   │ Max   │
├─────────┼───────┼───────┼───────┼───────┼──────────┼─────────┼───────┤
│ Latency │ 11 ms │ 18 ms │ 26 ms │ 30 ms │ 18.34 ms │ 4.01 ms │ 91 ms │
└─────────┴───────┴───────┴───────┴───────┴──────────┴─────────┴───────┘
┌───────────┬─────────┬─────────┬─────────┬─────────┬──────────┬────────┬─────────┐
│ Stat      │ 1%      │ 2.5%    │ 50%     │ 97.5%   │ Avg      │ Stdev  │ Min     │
├───────────┼─────────┼─────────┼─────────┼─────────┼──────────┼────────┼─────────┤
│ Req/Sec   │ 4,103   │ 4,743   │ 5,279   │ 5,723   │ 5,297.27 │ 336.78 │ 4,101   │
├───────────┼─────────┼─────────┼─────────┼─────────┼──────────┼────────┼─────────┤
│ Bytes/Sec │ 1.46 MB │ 1.69 MB │ 1.89 MB │ 2.04 MB │ 1.89 MB  │ 120 kB │ 1.46 MB │
└───────────┴─────────┴─────────┴─────────┴─────────┴──────────┴────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 240

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 4            │
├────────────┼──────────────┤
│ 0.01       │ 5            │
├────────────┼──────────────┤
│ 0.1        │ 6            │
├────────────┼──────────────┤
│ 1          │ 9            │
├────────────┼──────────────┤
│ 2.5        │ 11           │
├────────────┼──────────────┤
│ 10         │ 14           │
├────────────┼──────────────┤
│ 25         │ 16           │
├────────────┼──────────────┤
│ 50         │ 18           │
├────────────┼──────────────┤
│ 75         │ 20           │
├────────────┼──────────────┤
│ 90         │ 23           │
├────────────┼──────────────┤
│ 97.5       │ 26           │
├────────────┼──────────────┤
│ 99         │ 30           │
├────────────┼──────────────┤
│ 99.9       │ 50           │
├────────────┼──────────────┤
│ 99.99      │ 78           │
├────────────┼──────────────┤
│ 99.999     │ 89           │
└────────────┴──────────────┘

318k requests in 60.04s, 113 MB read

am_ws_load · what the clients received

$ am_ws_load listen "ws://<server>:38246/?x-am-authorization=<token>&x-am-user-authorization=<token>"
26,484 clients subscribed, 63.84s

┌──────────┬──────────┬─────────┬──────────┬──────────┬──────────┬─────────┬──────────┐
│ Stat     │ 2.5%     │ 50%     │ 97.5%    │ 99%      │ Avg      │ Stdev   │ Max      │
├──────────┼──────────┼─────────┼──────────┼──────────┼──────────┼─────────┼──────────┤
│ Delivery │ 11.82 ms │ 18.5 ms │ 26.66 ms │ 30.16 ms │ 18.72 ms │ 3.97 ms │ 91.58 ms │
└──────────┴──────────┴─────────┴──────────┴──────────┴──────────┴─────────┴──────────┘
┌───────────┬────┬──────┬───────┬───────┬─────────┬────────┬─────┐
│ Stat      │ 1% │ 2.5% │ 50%   │ 97.5% │ Avg     │ Stdev  │ Min │
├───────────┼────┼──────┼───────┼───────┼─────────┼────────┼─────┤
│ Notif/Sec │ 0  │ 0    │ 5,256 │ 5,725 │ 5,111.6 │ 996.02 │ 0   │
└───────────┴────┴──────┴───────┴───────┴─────────┴────────┴─────┘

Delivery : from the moment the request is written to the moment the client reads the notification.
# of samples: 64

318k notifications received in 63.84s (317,890 of 317,793 expected), 57.1 MB read

Requests / second 1,983 118,976 requests in 60 s
Latency p50 50ms average 49.92 ms
Latency p99 92ms max 161 ms
Errors 0 non-2xx, errors, timeouts

Server during this run API Maker 0.5 vCPU · Redis 0.0 vCPU · MongoDB 0.3 vCPU · whole server busy 100 % · load generator busy 24 %

autocannon · get all with both tokens · 26,484 WebSocket clients connected

$ autocannon -c 100 -d 60 -w 4 -l -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" "http://<server>:38246/api/schema/bench/mongodb/am_bench/customers"
Running 60s test @ http://<server>:38246/api/schema/bench/mongodb/am_bench/customers
100 connections
4 workers

┌─────────┬───────┬───────┬───────┬───────┬──────────┬──────────┬────────┐
│ Stat    │ 2.5%  │ 50%   │ 97.5% │ 99%   │ Avg      │ Stdev    │ Max    │
├─────────┼───────┼───────┼───────┼───────┼──────────┼──────────┼────────┤
│ Latency │ 26 ms │ 50 ms │ 74 ms │ 92 ms │ 49.92 ms │ 12.53 ms │ 161 ms │
└─────────┴───────┴───────┴───────┴───────┴──────────┴──────────┴────────┘
┌───────────┬─────────┬────────┬─────────┬─────────┬──────────┬────────┬─────────┐
│ Stat      │ 1%      │ 2.5%   │ 50%     │ 97.5%   │ Avg      │ Stdev  │ Min     │
├───────────┼─────────┼────────┼─────────┼─────────┼──────────┼────────┼─────────┤
│ Req/Sec   │ 1,648   │ 1,756  │ 1,985   │ 2,087   │ 1,983.04 │ 70.75  │ 1,648   │
├───────────┼─────────┼────────┼─────────┼─────────┼──────────┼────────┼─────────┤
│ Bytes/Sec │ 4.51 MB │ 4.8 MB │ 5.43 MB │ 5.71 MB │ 5.42 MB  │ 193 kB │ 4.51 MB │
└───────────┴─────────┴────────┴─────────┴─────────┴──────────┴────────┴─────────┘

Req/Bytes counts sampled once per second.
# of samples: 240

┌────────────┬──────────────┐
│ Percentile │ Latency (ms) │
├────────────┼──────────────┤
│ 0.001      │ 1            │
├────────────┼──────────────┤
│ 0.01       │ 2            │
├────────────┼──────────────┤
│ 0.1        │ 8            │
├────────────┼──────────────┤
│ 1          │ 18           │
├────────────┼──────────────┤
│ 2.5        │ 26           │
├────────────┼──────────────┤
│ 10         │ 36           │
├────────────┼──────────────┤
│ 25         │ 45           │
├────────────┼──────────────┤
│ 50         │ 50           │
├────────────┼──────────────┤
│ 75         │ 55           │
├────────────┼──────────────┤
│ 90         │ 60           │
├────────────┼──────────────┤
│ 97.5       │ 74           │
├────────────┼──────────────┤
│ 99         │ 92           │
├────────────┼──────────────┤
│ 99.9       │ 145          │
├────────────┼──────────────┤
│ 99.99      │ 154          │
├────────────┼──────────────┤
│ 99.999     │ 157          │
└────────────┴──────────────┘

119k requests in 60.03s, 325 MB read

A restart

API Maker was restarted (systemctl restart api_maker_be) with 19,863 clients connected : the newest left first, down to 75 % of the 26,484 it held. Every client of a restarted server comes back within the same seconds and waits for its turn with its request in the kernel, about 4 KB each : a server needs that room. It takes what Redis keeps of them out first, then closes every client with the code 1001 (Server is going away) ; each client waits one to five seconds, as an application does, connects again with the same tokens and sends its REGISTER again. The server which comes back lets 256 clients in at a time and handles the messages of 256 at a time, the others wait for their turn : its memory comes back to what it was, not higher.

Clients dropped 19,863 of 19,863 connected
All back after 10.0s the restart itself : 2.5 s
A client was away, p50 7.4s its own wait of 1 to 5 s counted
A client was away, p99 9.1s the longest : 9.2 s

After the restart, with the 19,863 clients back : API Maker 121 MB (126 MB before it), 314 MB of memory available. They were then told again, each for itself, 1,000 notifications a second for 60 seconds : all 60,000 arrived, in 2.52 ms (p50) and 4.18 ms (p99). A client whose subscription had not come back with it would have been told nothing.

Every client leaves at once

At the end of the run, the 19,863 clients closed their connection in the same second, as when a network or a load balancer in front of the server goes. The server had closed every one of their sockets after 1.5 s, and Redis held nothing of them any more after 2.6 s. A socket is closed at once, and Redis is told behind it.

Questions

How many WebSocket connections can API Maker hold on 1 shared vCPU and 1 GB ?

26,484 clients were connected at the same time on this server, each with both tokens and a subscription of its own, with MongoDB, Redis and two sandboxes on the same machine. Memory is what ends first : a client costs about 8.6 KB of the server in all.

How much memory does a WebSocket client cost ?

About 8.6 KB : 3.3 KB in the API Maker process, 4.3 KB in the Linux kernel (its socket) and 1.0 KB in Redis (its subscription). Measured between 10,000 and 26,484 clients which wait, each connected with both tokens and one subscription.

How many clients can connect each second ?

2,602 clients a second connected with both tokens, subscribed and left, 100 at a time ; 5,302 a second without a subscription. With a can user connect code on the event : 2,097 a second.

How fast does a notification reach a client ?

At 1,000 notifications a second, each for one person among 5,000 : 2.51 ms from the HTTP call to the client (p50), 3.82 ms for 99 % of them. With as many calls as the server takes : 5,216 notifications a second.

Do connected WebSocket clients slow the APIs down ?

Get all with both tokens answered 2,095 requests a second with no WebSocket client, and 1,983 with 26,484 clients connected and waiting. A client which waits costs memory ; its CPU is a ping a minute.