# API Maker WebSocket benchmark : 2 vCPU, 4 GB RAM (Linode dedicated CPU) > API Maker WebSockets on 2 dedicated vCPUs, 4 GB (Linode) : 338,006 clients connected at once, 8 KB each, 3,464 new clients a second, with memory and outputs. Source: https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html WebSocket benchmark · Linode (Akamai Cloud) · Dedicated CPU · 14 October 2026 **2 vCPU, 4 GB RAM : 338,006 WebSocket clients connected at once.** API Maker 4.0.0 on a Linode Dedicated 4 GB (g6-dedicated-2) : 2 dedicated vCPUs and 4 GB of RAM at $36 a month, its MongoDB and Redis on the same machine. A second server opened WebSocket clients, each with both tokens and a subscription of its own, until the memory of the server was full, while autocannon made the calls which notify them : every run 60 seconds, the memory and CPU of every part sampled each second. Clients connected 338,006 the memory was full : 451 MB left Memory per client 8.5KB API Maker 3.1 · kernel 4.3 · Redis 1.0 New clients / second 3,464 both tokens and a subscription · 7,077 without one Notification delivered in 1.74ms p50 at 1,000 a second · p99 3.70 ms ## The server | | | |---|---| | Server | Linode (Akamai Cloud), Dedicated 4 GB (g6-dedicated-2), dedicated cpu, $36 / month | | CPU | AMD EPYC 7642 48-Core Processor, 2 dedicated vCPUs (KVM)
KVM guest, 1 thread per core · L1d 128 KiB (2 instances) · L2 1 MiB (2 instances) · L3 32 MiB (2 instances) · BogoMIPS 4600.00 | | Memory | 4 GB (3,915 MB usable), swap 4.3 GB (496 MB of Linode + 3.8 GB made by the installer) | | Disk, region | 80 GB SSD, Mumbai, India (ap-west) | | System | Ubuntu 24.04.4 LTS, Linux 6.8.0-134, Docker 29.9.0 | | API Maker | 4.0.0 : release build of api_maker_be_rust 423a34c with the new WebSocket core of 14 October 2026 (opt-level 3, fat LTO), installed by its installer (https://apimaker.dev/v4/install.sh), no reverse proxy | | Settings | the defaults of the installer : one worker thread per vCPU, up to 1,048,576 open files for the service (`LimitNOFILE`), and the sandboxes kept running for the whole run (removeSandboxInactiveSinceMinutes 1440) | | Network limits | as the installer sets them : 1,002,240 connections tracked by the firewall of Linux (`net.netfilter.nf_conntrack_max`), 16,384 waiting to be accepted (`net.core.somaxconn`). The memory of all its TCP connections together : raised by API Maker as the clients came, to 979 MB at the most (`net.ipv4.tcp_mem`, 25 % of the memory ; Linux gives 9 %) | | Databases | MongoDB 8.0.10 (the MongoDB of API Maker, a replica set of one member, in Docker) · Redis 7.0.5 (the Redis of API Maker, in Docker) : on the same server. No other database runs | | Sandboxes | TypeScript on Node.js 24.21.0 (an image of 404 MB built by API Maker), 2 sandboxes of the account (sandboxCountForAdmin 2, the default) | | Load generator | Dedicated 8 GB (g6-dedicated-4) : AMD EPYC 7642 48-Core Processor, 4 dedicated vCPUs, Ubuntu 24.04.4 LTS. The WebSocket clients : am_ws_load, a program in Rust made for these benchmarks, a socket and a task per client, from 21 addresses of the load generator. The HTTP calls : Node.js 24.21.0, autocannon 8.0.0 | | Network | A VLAN of the same data centre, round trip 0.42 ms (ping) | | Date | 14 October 2026, 03:18 to 03:57 UTC | Only what the run needs was running : API Maker and its two sandboxes, its MongoDB and its Redis. API Maker was restarted and the swap emptied before the run. How every number was taken : [How the WebSocket benchmarks are made](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/methodology.html). Every server side by side : [All results](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/summary.html). The APIs of this server : [2 vCPU · 4 GB, API benchmark](https://docs.apimaker.dev/v4/docs/benchmarks/linode-dedicated-2-vcpu-4gb.html). ## The data - **The clients** connect to `ws://:38246/?x-am-authorization=&x-am-user-authorization=` : the token of an API user, and the token of a person, a row of the `customers` table of the [benchmark dataset](https://apimaker.dev/json/benchmark_dataset.json) (`https://apimaker.dev/json/benchmark_dataset.json`) made by a DB token generator. API Maker checks both. - **A subscription** is one `REGISTER` for the custom WebSocket event `order-updates` with a condition of its own : `{ conditionType: 'RESPONSE', criteria: { recipientId: '' } }`. It is the way [one notification reaches one person](https://docs.apimaker.dev/v4/docs/pages/web-socket-event-page.html). - **A notification** for one person is 141 bytes ; a row of the dataset told to the clients of a table is 321 bytes. - **The tables** are the ones of the [API benchmarks](https://docs.apimaker.dev/v4/docs/benchmarks/methodology.html#the-data), in MongoDB : ten rows of customers, ten columns each. ## Results at a glance **New clients per second** [Connect : the token of the API user](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#connect-1-token) · 7,417 [Connect : both tokens](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#connect-2-tokens) · 7,077 [Connect with both tokens and subscribe](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#subscribe) · 3,464 [Subscribe to an event with a can user connect code](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#can-user-connect) · the code ran in the server · 2,891 [Subscribe : the code reads the database](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#can-user-connect-database) · the code ran in the sandbox · 1,368 [Subscribe : the code reads the clock](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#can-user-connect-sandbox) · the code ran in the sandbox · 2,550 100 clients at a time connect, are let in and leave, for 60 seconds. **Notifications per second** [A notification for one person](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#one-person) · 6,757 calls / second · 6,618 [A broadcast](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#broadcast) · 71 calls / second · 69,874 [Rows inserted in a table](https://docs.apimaker.dev/v4/docs/benchmarks/websockets/linode-dedicated-2-vcpu-4gb.html#insert) · 506 calls / second · 49,312 As many calls as the server takes : what the clients received each second. ## How many clients fit Clients connect 2,000 a second, level after level, each with both tokens and a subscription of its own ; at each level they wait 30 seconds while the server is sampled (the level the run stops at, 60). The run stopped adding clients at **338,006** : the memory of the server was full. 430 MB of its 3,916 MB were still counted as available, and Linux had moved 240 MB to the swap since the clients began to connect, to make that room : 190 MB were left (the run stops when what is available, less what Linux moved to the swap since the clients began to connect, is under 5 % of the memory of the server). > Diagram : Clients connected and memory used during the run > > The run of this server, second after second : the WebSocket clients connected, up to 338,006, and the memory the server used, up to 3,496 MB of 3,915 MB. API Maker · MongoDB of API Maker · Redis · Sandboxes · Linux : its sockets, Docker, the rest · Available No client at rest 10,000 clients connected 25,000 clients connected 50,000 clients connected 100,000 clients connected 150,000 clients connected 200,000 clients connected 250,000 clients connected 300,000 clients connected 338,006 clients connected | Clients | API Maker | Redis | Kernel | MongoDB | Sandboxes | Server, used | Available | Swap | CPU busy | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| | None (at rest) | 109 | 1 | 202 | 123 | 51 | **767** | 3,149 | 0 | 1.7 % | | **10,000** | 148 | 12 | 271 | 155 | 220 | **1,100** | 2,816 | 0 | 1.7 % | | **25,000** | 200 | 27 | 311 | 190 | 220 | **1,239** | 2,677 | 0 | 1.5 % | | **50,000** | 278 | 52 | 414 | 186 | 220 | **1,429** | 2,488 | 0 | 2.9 % | | **100,000** | 438 | 102 | 627 | 186 | 219 | **1,852** | 2,064 | 0 | 3.1 % | | **150,000** | 603 | 157 | 838 | 187 | 219 | **2,238** | 1,678 | 0 | 5.1 % | | **200,000** | 753 | 204 | 1,050 | 187 | 219 | **2,600** | 1,316 | 0 | 8.3 % | | **250,000** | 892 | 250 | 1,264 | 186 | 219 | **3,010** | 906 | 0 | 10.9 % | | **300,000** | 1,008 | 313 | 1,469 | 157 | 159 | **3,331** | 585 | 136 | 13.6 % | | **338,006** | 1,108 | 347 | 1,627 | 125 | 99 | **3,465** | 451 | 270 | 13.8 % | Memory in MB. API Maker is the resident memory of its process ; Redis is the memory of its data (`used_memory`) ; the kernel of Linux keeps every socket and every tracked connection in its slab ; a container is counted the way `docker stats` counts it. CPU busy is the whole server while the clients wait : the server pings every client once a minute, and each answers. ### Memory when the server is full With **338,006 clients** connected and waiting, of the 3,916 MB of the server : | Part | At rest, no client | 338,006 clients | More | |---|--:|--:|--:| | **API Maker** (process) | 109 MB | **1,108 MB** | + 1,000 MB | | Redis (container) | 3 MB | **356 MB** | + 353 MB | | Linux kernel : sockets, tracked connections (slab) | 202 MB | **1,627 MB** | + 1,424 MB | | MongoDB of API Maker (container) | 123 MB | **125 MB** | + 1 MB | | Sandboxes (2 containers) | 51 MB | **99 MB** | + 48 MB | | **Whole server**, used | 767 MB | **3,465 MB** | + 2,699 MB | | Available | 3,149 MB | **451 MB** | | | Swap used | 0 MB | **270 MB** | | | Open files of API Maker | 27 | **338,034** | | | Keys of Redis · channels it listens on | 13 · 5 | **676,028 · 338,011** | | MongoDB and the sandboxes grew during the runs before this one (the inserts, the code of the events), not with the clients. ### What a client costs Between 100,000 and 250,000 clients (4 levels), each client more cost the server **8.5 KB** : | Where | Per client | What it is | |---|--:|---| | **API Maker** | **3.1 KB** | The client itself : its socket and the one task which reads it, its tokens and what they said, its subscription and the channel it is told on. | | Linux kernel | 4.3 KB | The TCP socket, and the entry of the connection in the table of the firewall (Docker makes Linux track every connection). | | Redis | 1.0 KB | The subscription : its key, its place in the list of its condition, and the channel API Maker listens on. | | **In all** | **8.5 KB** | 123,507 clients per GB of memory. | Measured over the levels at which nothing was in the swap. The memory the whole server had available went down by 7.9 KB per client : MongoDB, the sandboxes and the cache of files give some back while the clients take theirs. ## One more subscription, and a broadcast to everybody With 50,000 clients connected, every client sent a second `REGISTER`, for an event without a condition (`announcements`), 2,000 of them a second : 50,000 were taken in 25.0 s. They waited 30 seconds with two subscriptions each, were told 3 broadcasts, one call of the emit event WS API each, and took the subscription back with `UNREGISTER`. One subscription more 1.2KB per client : API Maker 0.5 KB · Redis 0.8 KB A broadcast to 50k clients 0.91s until the last client read it Half of them told after 493ms the first after 172 ms Clients told / second 54,802 one call, every client | Broadcast | Clients told | The call answered after | The first client | Half of them | 99 % of them | The last one | Clients told a second | |---|--:|--:|--:|--:|--:|--:|--:| | 1 | 50,000 of 50,000 | 0.93 s | 179 ms | 534 ms | 904 ms | **916 ms** | 54,587 | | 2 | 50,000 of 50,000 | 0.92 s | 172 ms | 493 ms | 898 ms | **912 ms** | 54,802 | | 3 | 50,000 of 50,000 | 1.02 s | 168 ms | 579 ms | 998 ms | **1,013 ms** | 49,346 | One call of the emit event WS API tells every client : API Maker asks Redis for the clients of the event, publishes the notification for each of them, and the server which holds a client sends it. The times are counted from the moment the call is written to the moment a client reads its notification. | Memory, 50,000 clients | One subscription each | Two subscriptions each | A subscription more, per client | |---|--:|--:|--:| | API Maker | 278 MB | 300 MB | 0.5 KB | | Redis (its data) | 52 MB | 88 MB | 0.8 KB | | Keys of Redis · channels | 100,016 · 50,005 | 150,017 · 100,005 | | | **Whole server**, less available | | | **1.2 KB** | ## New clients per second How many clients a second the server lets in : 100 clients at a time connect, are let in and leave, one after the other, for 60 seconds (after 5 seconds which are not counted). It is what a server faces when its clients all come back, after a deployment or a network cut. The times are counted from the start of the TCP connection. ### Connect : the token of the API user A client connects with `x-am-authorization` alone, is told `CONNECTED`, and leaves. API Maker checks the token of the API user and finds its groups : what every WebSocket client costs before it asks for anything. Clients / second 7,417 445,467 clients in 60 s, 100 at a time Let in after, p50 7.34ms connected 7.34 ms · TCP 0.25 ms Let in after, p99 13.6ms max 26.4 ms Refused or lost 0 not connected, not subscribed, no answer (Server during this run) API Maker **1.9 vCPU** · MongoDB **0.0 vCPU** · whole server busy **99 %** · load generator busy **60 %** am_ws_load · Connect : the token of the API user · 60 s ```text $ am_ws_load churn -c 100 -d 60 "ws://:38246/?x-am-authorization=" Running 60s test @ ws://:38246 100 clients at a time : connect, CONNECTED, leave ┌─────────────┬─────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────────┼─────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ TCP connect │ 0.18 ms │ 0.25 ms │ 0.67 ms │ 1.33 ms │ 0.3 ms │ 0.32 ms │ 11.53 ms │ ├─────────────┼─────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ CONNECTED │ 2.26 ms │ 7.34 ms │ 12.43 ms │ 13.56 ms │ 7.25 ms │ 2.76 ms │ 26.37 ms │ ├─────────────┼─────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ Left │ 7.39 ms │ 13.44 ms │ 19.87 ms │ 21.15 ms │ 13.45 ms │ 3.3 ms │ 34.5 ms │ └─────────────┴─────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘ ┌─────────────┬───────┬───────┬───────┬───────┬─────────┬────────┬───────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├─────────────┼───────┼───────┼───────┼───────┼─────────┼────────┼───────┤ │ Clients/Sec │ 7,056 │ 7,181 │ 7,336 │ 7,836 │ 7,424.4 │ 215.76 │ 7,056 │ └─────────────┴───────┴───────┴───────┴───────┴─────────┴────────┴───────┘ Time from the start of the connection to each step. Clients counted once per second. # of samples: 60 445k clients let in in 60.06s, 0 refused or lost ``` ### Connect : both tokens A client connects with `x-am-authorization` and `x-am-user-authorization`, is told `CONNECTED`, and leaves. Both tokens are checked : the one of the API user, and the one of a person, made by the DB token generator of the account. Clients / second 7,077 425,048 clients in 60 s, 100 at a time Let in after, p50 7.76ms connected 7.76 ms · TCP 0.25 ms Let in after, p99 14.5ms max 31.6 ms Refused or lost 0 not connected, not subscribed, no answer (Server during this run) API Maker **1.9 vCPU** · MongoDB **0.0 vCPU** · whole server busy **100 %** · load generator busy **58 %** am_ws_load · Connect : both tokens · 60 s ```text $ am_ws_load churn -c 100 -d 60 "ws://:38246/?x-am-authorization=&x-am-user-authorization=" Running 60s test @ ws://:38246 100 clients at a time : connect, CONNECTED, leave ┌─────────────┬─────────┬──────────┬──────────┬──────────┬─────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────────┼─────────┼──────────┼──────────┼──────────┼─────────┼─────────┼──────────┤ │ TCP connect │ 0.18 ms │ 0.25 ms │ 1.02 ms │ 1.49 ms │ 0.32 ms │ 0.31 ms │ 13.77 ms │ ├─────────────┼─────────┼──────────┼──────────┼──────────┼─────────┼─────────┼──────────┤ │ CONNECTED │ 2.41 ms │ 7.76 ms │ 13.36 ms │ 14.49 ms │ 7.71 ms │ 2.93 ms │ 31.63 ms │ ├─────────────┼─────────┼──────────┼──────────┼──────────┼─────────┼─────────┼──────────┤ │ Left │ 7.62 ms │ 14.02 ms │ 21.2 ms │ 22.64 ms │ 14.1 ms │ 3.54 ms │ 37.25 ms │ └─────────────┴─────────┴──────────┴──────────┴──────────┴─────────┴─────────┴──────────┘ ┌─────────────┬───────┬───────┬───────┬───────┬─────────┬────────┬───────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├─────────────┼───────┼───────┼───────┼───────┼─────────┼────────┼───────┤ │ Clients/Sec │ 6,437 │ 6,464 │ 7,111 │ 7,603 │ 7,084.1 │ 330.46 │ 6,437 │ └─────────────┴───────┴───────┴───────┴───────┴─────────┴────────┴───────┘ Time from the start of the connection to each step. Clients counted once per second. # of samples: 60 425k clients let in in 60.06s, 0 refused or lost ``` ### Connect with both tokens and subscribe The whole of what an application does : connect with both tokens, `CONNECTED`, one `REGISTER` for a custom WebSocket event with a condition of its own (`recipientId` : its user id), the answer, then leave. API Maker checks the groups and the auth provider of the event, keeps the subscription in Redis and listens on its channel ; when the client leaves, all of it is taken back. Clients / second 3,464 208,028 clients in 60 s, 100 at a time Let in after, p50 25.1ms connected 5.65 ms · TCP 0.23 ms Let in after, p99 35.0ms max 46.3 ms Refused or lost 0 not connected, not subscribed, no answer (Server during this run) API Maker **1.5 vCPU** · Redis **0.2 vCPU** · whole server busy **100 %** · load generator busy **36 %** am_ws_load · Connect with both tokens and subscribe · 60 s ```text $ am_ws_load churn -c 100 -d 60 "ws://:38246/?x-am-authorization=&x-am-user-authorization=" --register 'order-updates' Running 60s test @ ws://:38246 100 clients at a time : connect, CONNECTED, REGISTER, leave ┌─────────────┬──────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ TCP connect │ 0.18 ms │ 0.23 ms │ 0.41 ms │ 0.62 ms │ 0.26 ms │ 0.17 ms │ 11.2 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ CONNECTED │ 1.78 ms │ 5.65 ms │ 10.66 ms │ 11.66 ms │ 5.79 ms │ 2.32 ms │ 22.85 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ REGISTER │ 16.31 ms │ 25.15 ms │ 33.15 ms │ 35.04 ms │ 25.02 ms │ 4.22 ms │ 46.3 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ Left │ 20.05 ms │ 28.83 ms │ 37.7 ms │ 39.55 ms │ 28.85 ms │ 4.41 ms │ 52.58 ms │ └─────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘ ┌─────────────┬───────┬───────┬───────┬───────┬─────────┬───────┬───────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├─────────────┼───────┼───────┼───────┼───────┼─────────┼───────┼───────┤ │ Clients/Sec │ 3,378 │ 3,383 │ 3,463 │ 3,576 │ 3,467.1 │ 50.75 │ 3,378 │ └─────────────┴───────┴───────┴───────┴───────┴─────────┴───────┴───────┘ Time from the start of the connection to each step. Clients counted once per second. # of samples: 60 208k clients let in in 60.06s, 0 refused or lost ``` ### Subscribe to an event with a can user connect code The same, on an event whose *can user connect code* decides who may subscribe : it reads the person of `x-am-user-authorization` from `g.req.auth`. The code ran in the server itself (lean code, QuickJS). Clients / second 2,891 173,650 clients in 60 s, 100 at a time Let in after, p50 30.1ms connected 6.26 ms · TCP 0.24 ms Let in after, p99 43.1ms max 90.8 ms Refused or lost 0 not connected, not subscribed, no answer (Server during this run) API Maker **1.5 vCPU** · Redis **0.2 vCPU** · whole server busy **100 %** · load generator busy **32 %** am_ws_load · Subscribe to an event with a can user connect code · 60 s ```text $ am_ws_load churn -c 100 -d 60 "ws://:38246/?x-am-authorization=&x-am-user-authorization=" --register 'checked-token' Running 60s test @ ws://:38246 100 clients at a time : connect, CONNECTED, REGISTER, leave ┌─────────────┬──────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ TCP connect │ 0.19 ms │ 0.24 ms │ 0.49 ms │ 0.73 ms │ 0.27 ms │ 0.19 ms │ 11.05 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ CONNECTED │ 2 ms │ 6.26 ms │ 11.59 ms │ 12.79 ms │ 6.4 ms │ 2.52 ms │ 28.5 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ REGISTER │ 19.94 ms │ 30.14 ms │ 40.38 ms │ 43.1 ms │ 30.17 ms │ 5.36 ms │ 90.82 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ Left │ 24.14 ms │ 34.4 ms │ 45.47 ms │ 48.48 ms │ 34.56 ms │ 5.62 ms │ 97.28 ms │ └─────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘ ┌─────────────┬───────┬───────┬───────┬───────┬─────────┬───────┬───────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├─────────────┼───────┼───────┼───────┼───────┼─────────┼───────┼───────┤ │ Clients/Sec │ 2,280 │ 2,790 │ 2,895 │ 3,022 │ 2,894.2 │ 96.02 │ 2,280 │ └─────────────┴───────┴───────┴───────┴───────┴─────────┴───────┴───────┘ Time from the start of the connection to each step. Clients counted once per second. # of samples: 60 174k clients let in in 60.06s, 0 refused or lost ``` ### Subscribe : the code reads the database The same, with a can user connect code which also reads the person from the database (`g.sys.db.getAll` with a `find`) before it answers : one read of MongoDB for every subscription. The code ran in a sandbox of the account. Clients / second 1,368 82,156 clients in 60 s, 100 at a time Let in after, p50 66.6ms connected 9.68 ms · TCP 0.25 ms Let in after, p99 93.1ms max 114 ms Refused or lost 0 not connected, not subscribed, no answer (Server during this run) API Maker **0.7 vCPU** · Redis **0.1 vCPU** · MongoDB **0.3 vCPU** · sandboxes **0.7 vCPU** · whole server busy **100 %** · load generator busy **16 %** am_ws_load · Subscribe : the code reads the database · 60 s ```text $ am_ws_load churn -c 100 -d 60 "ws://:38246/?x-am-authorization=&x-am-user-authorization=" --register 'checked-database' Running 60s test @ ws://:38246 100 clients at a time : connect, CONNECTED, REGISTER, leave ┌─────────────┬──────────┬──────────┬──────────┬───────────┬──────────┬──────────┬───────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────────┼──────────┼──────────┼──────────┼───────────┼──────────┼──────────┼───────────┤ │ TCP connect │ 0.19 ms │ 0.25 ms │ 0.41 ms │ 0.5 ms │ 0.26 ms │ 0.1 ms │ 5.93 ms │ ├─────────────┼──────────┼──────────┼──────────┼───────────┼──────────┼──────────┼───────────┤ │ CONNECTED │ 2.96 ms │ 9.68 ms │ 18.5 ms │ 20.14 ms │ 9.94 ms │ 4.02 ms │ 29.7 ms │ ├─────────────┼──────────┼──────────┼──────────┼───────────┼──────────┼──────────┼───────────┤ │ REGISTER │ 45.86 ms │ 66.56 ms │ 88.58 ms │ 93.12 ms │ 66.62 ms │ 10.72 ms │ 114.11 ms │ ├─────────────┼──────────┼──────────┼──────────┼───────────┼──────────┼──────────┼───────────┤ │ Left │ 52.03 ms │ 73.02 ms │ 95.62 ms │ 100.48 ms │ 73.1 ms │ 10.97 ms │ 125.82 ms │ └─────────────┴──────────┴──────────┴──────────┴───────────┴──────────┴──────────┴───────────┘ ┌─────────────┬───────┬───────┬───────┬───────┬─────────┬───────┬───────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├─────────────┼───────┼───────┼───────┼───────┼─────────┼───────┼───────┤ │ Clients/Sec │ 1,314 │ 1,314 │ 1,375 │ 1,405 │ 1,369.3 │ 26.47 │ 1,314 │ └─────────────┴───────┴───────┴───────┴───────┴─────────┴───────┴───────┘ Time from the start of the connection to each step. Clients counted once per second. # of samples: 60 82k clients let in in 60.06s, 0 refused or lost ``` ### Subscribe : the code reads the clock The same, with a can user connect code which reads the time of the machine (`Date.now()`) : code which is not lean, so API Maker runs it in a sandbox of the account. The code ran in a sandbox of the account. Clients / second 2,550 153,134 clients in 60 s, 100 at a time Let in after, p50 35.3ms connected 6.06 ms · TCP 0.24 ms Let in after, p99 48.8ms max 76.8 ms Refused or lost 0 not connected, not subscribed, no answer (Server during this run) API Maker **1.3 vCPU** · Redis **0.2 vCPU** · sandboxes **0.3 vCPU** · whole server busy **100 %** · load generator busy **30 %** am_ws_load · Subscribe : the code reads the clock · 60 s ```text $ am_ws_load churn -c 100 -d 60 "ws://:38246/?x-am-authorization=&x-am-user-authorization=" --register 'checked-sandbox' Running 60s test @ ws://:38246 100 clients at a time : connect, CONNECTED, REGISTER, leave ┌─────────────┬──────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ TCP connect │ 0.18 ms │ 0.24 ms │ 0.42 ms │ 0.62 ms │ 0.26 ms │ 0.17 ms │ 8.04 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ CONNECTED │ 2.04 ms │ 6.06 ms │ 10.97 ms │ 12.02 ms │ 6.17 ms │ 2.29 ms │ 22.59 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ REGISTER │ 23.97 ms │ 35.33 ms │ 46.34 ms │ 48.77 ms │ 35.24 ms │ 5.63 ms │ 76.8 ms │ ├─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ Left │ 27.81 ms │ 39.17 ms │ 50.82 ms │ 53.28 ms │ 39.2 ms │ 5.79 ms │ 85.06 ms │ └─────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘ ┌─────────────┬───────┬───────┬───────┬───────┬─────────┬───────┬───────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├─────────────┼───────┼───────┼───────┼───────┼─────────┼───────┼───────┤ │ Clients/Sec │ 2,468 │ 2,469 │ 2,551 │ 2,632 │ 2,552.2 │ 38.28 │ 2,468 │ └─────────────┴───────┴───────┴───────┴───────┴─────────┴───────┴───────┘ Time from the start of the connection to each step. Clients counted once per second. # of samples: 60 153k clients let in in 60.06s, 0 refused or lost ``` ## Can user connect code An event of the WebSocket Events page can carry a *can user connect code* : API Maker runs it for every subscription to the event, with the tokens of the client in `g.req.auth` and the subscription in `g.req.eventData`, and lets the client subscribe when it returns `{ canConnect: true }`. Three codes were measured : one which reads the person of the token, one which also reads the database, one which reads the clock. **Reads the person of the token** **Can user connect code of the event checked-token : ran in the server** ```typescript import * as T from 'types'; async function main(g: T.IAMGlobal) { // The person of x-am-user-authorization : only somebody who signed in subscribes, from a country we serve. const person = g.req.auth.authAMDB; if (!person) return { canConnect: false, errorText: 'Sign in first.' }; return { canConnect: person.country === 'India', errorText: 'Not available in your country.' }; }; module.exports = main; ``` **Reads the database** **Can user connect code of the event checked-database : ran in the sandbox** ```typescript import * as T from 'types'; async function main(g: T.IAMGlobal) { const person = g.req.auth.authAMDB; if (!person) return { canConnect: false, errorText: 'Sign in first.' }; // Still a customer ? One read of the database for every subscription. const rows = await g.sys.db.getAll({ instance: 'mongodb', database: 'am_bench', collection: 'customers', queryParams: { find: { customer_id: person.customer_id } } }); return { canConnect: rows.length === 1 && rows[0].country === 'India', errorText: 'Not a customer any more.' }; }; module.exports = main; ``` **Reads the clock** **Can user connect code of the event checked-sandbox : ran in the sandbox** ```typescript import * as T from 'types'; async function main(g: T.IAMGlobal) { const person = g.req.auth.authAMDB; if (!person) return { canConnect: false, errorText: 'Sign in first.' }; // The time of the machine : this code runs in the sandbox of the account. const hour = new Date(Date.now()).getUTCHours(); return { canConnect: person.country === 'India' && hour >= 0, errorText: 'Not available in your country.' }; }; module.exports = main; ``` | Can user connect code | Runs in | Clients / second | Subscribed after, p50 | p99 | Against no code | |---|---|--:|--:|--:|--:| | No code | – | **3,464** | 25.1 ms | 35.0 ms | – | | Reads the person of the token | the server (QuickJS) | **2,891** | 30.1 ms | 43.1 ms | 17 % fewer | | Reads the person from the database | a sandbox of the account | **1,368** | 66.6 ms | 93.1 ms | 61 % fewer | | Reads the clock | a sandbox of the account | **2,550** | 35.3 ms | 48.8 ms | 26 % fewer | Lean code (it only reads what it is handed) runs in the server itself, with QuickJS, on the thread which serves the client. Code which needs more runs in a sandbox of the account : every subscription is then a round trip to the sandbox. ## Notifications With 10,000 clients connected, autocannon makes the HTTP calls which notify them, for 60 seconds, and the clients count what they receive and how late. Each kind of notification is measured twice : at a pace far under what the server takes, which shows how long a notification takes, and with no pace, which shows how many a second the server tells. *Delivery* is the time from the moment autocannon writes the request to the moment the client reads the notification : the HTTP call, API Maker, Redis, and the WebSocket. ### A notification for one person autocannon calls the [emit event WS API](https://docs.apimaker.dev/v4/docs/apis-all/system-apis/system-generated-emit-event-ws-api.html) with both tokens ; every call names another one of the 10,000 connected clients (`recipientId`), which is the only one told. The clients measure how long each notification took from the moment its request was written. **1,000 calls a second** Calls / second (paced) 1,000 60,006 calls in 60 s, 10 connections Notifications / second 983 60,016 received by 10,000 clients Delivery p50 1.74ms average 1.79 ms Delivery p99 3.70ms max 7.91 ms Not delivered 0 0 calls failed (Server during this run) API Maker **0.3 vCPU** · Redis **0.0 vCPU** · whole server busy **19 %** · load generator busy **11 %** autocannon · A notification for one person, 1,000 a second ```text $ autocannon -c 10 -d 60 -R 1000 -l -m POST -b '{"name":"order-updates","eventData":{"recipientId":"","t":,"type":"order-updated","order_no":1001,"status":"SHIPPED"}}' -H "x-am-authorization: " -H "x-am-user-authorization: " -H "content-type: application/json" "http://:38246/api/system-api/bench/emit-event-ws" Running 60s test @ http://:38246/api/system-api/bench/emit-event-ws 10 connections ┌─────────┬──────┬──────┬───────┬──────┬─────────┬────────┬──────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼──────┼──────┼───────┼──────┼─────────┼────────┼──────┤ │ Latency │ 1 ms │ 1 ms │ 3 ms │ 3 ms │ 1.35 ms │ 0.6 ms │ 8 ms │ └─────────┴──────┴──────┴───────┴──────┴─────────┴────────┴──────┘ ┌───────────┬────────┬────────┬────────┬────────┬─────────┬───────┬────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────────┼────────┼────────┼────────┼─────────┼───────┼────────┤ │ Req/Sec │ 997 │ 998 │ 1,000 │ 1,006 │ 1,000.1 │ 1.52 │ 997 │ ├───────────┼────────┼────────┼────────┼────────┼─────────┼───────┼────────┤ │ Bytes/Sec │ 356 kB │ 356 kB │ 357 kB │ 359 kB │ 357 kB │ 526 B │ 356 kB │ └───────────┴────────┴────────┴────────┴────────┴─────────┴───────┴────────┘ Req/Bytes counts sampled once per second. # of samples: 60 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 0 │ ├────────────┼──────────────┤ │ 0.01 │ 0 │ ├────────────┼──────────────┤ │ 0.1 │ 0 │ ├────────────┼──────────────┤ │ 1 │ 1 │ ├────────────┼──────────────┤ │ 2.5 │ 1 │ ├────────────┼──────────────┤ │ 10 │ 1 │ ├────────────┼──────────────┤ │ 25 │ 1 │ ├────────────┼──────────────┤ │ 50 │ 1 │ ├────────────┼──────────────┤ │ 75 │ 2 │ ├────────────┼──────────────┤ │ 90 │ 2 │ ├────────────┼──────────────┤ │ 97.5 │ 3 │ ├────────────┼──────────────┤ │ 99 │ 3 │ ├────────────┼──────────────┤ │ 99.9 │ 5 │ ├────────────┼──────────────┤ │ 99.99 │ 7 │ ├────────────┼──────────────┤ │ 99.999 │ 8 │ └────────────┴──────────────┘ 61k requests in 60.04s, 21.4 MB read ``` am_ws_load · what the clients received ```text $ am_ws_load listen "ws://:38246/?x-am-authorization=&x-am-user-authorization=" 10,000 clients subscribed, 64.06s ┌──────────┬──────┬─────────┬─────────┬────────┬─────────┬─────────┬─────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├──────────┼──────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┤ │ Delivery │ 1 ms │ 1.74 ms │ 2.85 ms │ 3.7 ms │ 1.79 ms │ 0.51 ms │ 7.91 ms │ └──────────┴──────┴─────────┴─────────┴────────┴─────────┴─────────┴─────────┘ ┌───────────┬────┬──────┬───────┬───────┬───────┬────────┬─────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────┼──────┼───────┼───────┼───────┼────────┼─────┤ │ Notif/Sec │ 0 │ 0 │ 1,000 │ 1,000 │ 951.9 │ 213.44 │ 0 │ └───────────┴────┴──────┴───────┴───────┴───────┴────────┴─────┘ Delivery : from the moment the request is written to the moment the client reads the notification. # of samples: 64 60k notifications received in 64.06s (60,016 of 60,006 expected), 10.7 MB read ``` **As many as the server takes** Calls / second 6,757 405,367 calls in 60 s, 100 connections Notifications / second 6,618 405,464 received by 10,000 clients Delivery p50 14.7ms average 14.6 ms Delivery p99 21.5ms max 39.4 ms Not delivered 0 0 calls failed (Server during this run) API Maker **1.6 vCPU** · Redis **0.1 vCPU** · whole server busy **99 %** · load generator busy **39 %** autocannon · A notification for one person, as many as the server takes ```text $ autocannon -c 100 -d 60 -w 4 -l -m POST -b '{"name":"order-updates","eventData":{"recipientId":"","t":,"type":"order-updated","order_no":1001,"status":"SHIPPED"}}' -H "x-am-authorization: " -H "x-am-user-authorization: " -H "content-type: application/json" "http://:38246/api/system-api/bench/emit-event-ws" Running 60s test @ http://:38246/api/system-api/bench/emit-event-ws 100 connections 4 workers ┌─────────┬──────┬───────┬───────┬───────┬──────────┬─────────┬───────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼──────┼───────┼───────┼───────┼──────────┼─────────┼───────┤ │ Latency │ 9 ms │ 14 ms │ 20 ms │ 21 ms │ 14.26 ms │ 2.76 ms │ 52 ms │ └─────────┴──────┴───────┴───────┴───────┴──────────┴─────────┴───────┘ ┌───────────┬─────────┬─────────┬─────────┬────────┬─────────┬─────────┬─────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┤ │ Req/Sec │ 6,527 │ 6,555 │ 6,755 │ 6,991 │ 6,756.6 │ 96.88 │ 6,527 │ ├───────────┼─────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┤ │ Bytes/Sec │ 2.33 MB │ 2.34 MB │ 2.41 MB │ 2.5 MB │ 2.41 MB │ 34.4 kB │ 2.33 MB │ └───────────┴─────────┴─────────┴─────────┴────────┴─────────┴─────────┴─────────┘ Req/Bytes counts sampled once per second. # of samples: 240 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 3 │ ├────────────┼──────────────┤ │ 0.01 │ 4 │ ├────────────┼──────────────┤ │ 0.1 │ 5 │ ├────────────┼──────────────┤ │ 1 │ 7 │ ├────────────┼──────────────┤ │ 2.5 │ 9 │ ├────────────┼──────────────┤ │ 10 │ 11 │ ├────────────┼──────────────┤ │ 25 │ 13 │ ├────────────┼──────────────┤ │ 50 │ 14 │ ├────────────┼──────────────┤ │ 75 │ 16 │ ├────────────┼──────────────┤ │ 90 │ 18 │ ├────────────┼──────────────┤ │ 97.5 │ 20 │ ├────────────┼──────────────┤ │ 99 │ 21 │ ├────────────┼──────────────┤ │ 99.9 │ 25 │ ├────────────┼──────────────┤ │ 99.99 │ 40 │ ├────────────┼──────────────┤ │ 99.999 │ 48 │ └────────────┴──────────────┘ 405k requests in 60.03s, 145 MB read ``` am_ws_load · what the clients received ```text $ am_ws_load listen "ws://:38246/?x-am-authorization=&x-am-user-authorization=" 10,000 clients subscribed, 64.27s ┌──────────┬─────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├──────────┼─────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ Delivery │ 9.09 ms │ 14.67 ms │ 20.14 ms │ 21.52 ms │ 14.63 ms │ 2.72 ms │ 39.39 ms │ └──────────┴─────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘ ┌───────────┬────┬──────┬───────┬───────┬─────────┬─────────┬─────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────┼──────┼───────┼───────┼─────────┼─────────┼─────┤ │ Notif/Sec │ 0 │ 0 │ 6,742 │ 6,908 │ 6,461.3 │ 1336.22 │ 0 │ └───────────┴────┴──────┴───────┴───────┴─────────┴─────────┴─────┘ Delivery : from the moment the request is written to the moment the client reads the notification. # of samples: 64 405k notifications received in 64.27s (405,464 of 405,367 expected), 72.6 MB read ``` ### A broadcast 1,000 more clients subscribe to an event without a condition : every call of the emit event WS API reaches all 1,000 of them. **5 calls a second** Calls / second (paced) 5 300 calls in 60 s, 1 connection Notifications / second 4,930 301,000 received by 1,000 clients Delivery p50 10.3ms average 10.5 ms Delivery p99 18.3ms max 23.3 ms Not delivered 0 0 calls failed (Server during this run) API Maker **0.2 vCPU** · whole server busy **9 %** · load generator busy **8 %** autocannon · A broadcast to 1,000 clients, 5 a second ```text $ autocannon -c 1 -d 60 -R 5 -l -m POST -b '{"name":"announcements","eventData":{"t":,"type":"announcement","text":"Hello from API Maker"}}' -H "x-am-authorization: " -H "x-am-user-authorization: " -H "content-type: application/json" "http://:38246/api/system-api/bench/emit-event-ws" Running 60s test @ http://:38246/api/system-api/bench/emit-event-ws 1 connections ┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬───────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼───────┤ │ Latency │ 1 ms │ 9 ms │ 18 ms │ 19 ms │ 9.21 ms │ 5.09 ms │ 23 ms │ └─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴───────┘ ┌───────────┬────────┬────────┬────────┬────────┬────────┬───────┬────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────────┼────────┼────────┼────────┼────────┼───────┼────────┤ │ Req/Sec │ 5 │ 5 │ 5 │ 5 │ 5 │ 0 │ 5 │ ├───────────┼────────┼────────┼────────┼────────┼────────┼───────┼────────┤ │ Bytes/Sec │ 1.8 kB │ 1.8 kB │ 1.8 kB │ 1.8 kB │ 1.8 kB │ 0 B │ 1.8 kB │ └───────────┴────────┴────────┴────────┴────────┴────────┴───────┴────────┘ Req/Bytes counts sampled once per second. # of samples: 60 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 1 │ ├────────────┼──────────────┤ │ 0.01 │ 1 │ ├────────────┼──────────────┤ │ 0.1 │ 1 │ ├────────────┼──────────────┤ │ 1 │ 1 │ ├────────────┼──────────────┤ │ 2.5 │ 1 │ ├────────────┼──────────────┤ │ 10 │ 2 │ ├────────────┼──────────────┤ │ 25 │ 5 │ ├────────────┼──────────────┤ │ 50 │ 9 │ ├────────────┼──────────────┤ │ 75 │ 13 │ ├────────────┼──────────────┤ │ 90 │ 16 │ ├────────────┼──────────────┤ │ 97.5 │ 18 │ ├────────────┼──────────────┤ │ 99 │ 19 │ ├────────────┼──────────────┤ │ 99.9 │ 21 │ ├────────────┼──────────────┤ │ 99.99 │ 23 │ ├────────────┼──────────────┤ │ 99.999 │ 23 │ └────────────┴──────────────┘ 305 requests in 60.05s, 108 kB read ``` am_ws_load · what the clients received ```text $ am_ws_load listen "ws://:38246/?x-am-authorization=&x-am-user-authorization=" 1,000 clients subscribed, 64.06s ┌──────────┬─────────┬──────────┬──────────┬──────────┬──────────┬─────────┬─────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├──────────┼─────────┼──────────┼──────────┼──────────┼──────────┼─────────┼─────────┤ │ Delivery │ 3.91 ms │ 10.35 ms │ 17.71 ms │ 18.32 ms │ 10.46 ms │ 4.18 ms │ 23.3 ms │ └──────────┴─────────┴──────────┴──────────┴──────────┴──────────┴─────────┴─────────┘ ┌───────────┬────┬──────┬───────┬───────┬─────────┬─────────┬─────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────┼──────┼───────┼───────┼─────────┼─────────┼─────┤ │ Notif/Sec │ 0 │ 0 │ 5,000 │ 5,000 │ 4,774.2 │ 1006.74 │ 0 │ └───────────┴────┴──────┴───────┴───────┴─────────┴─────────┴─────┘ Delivery : from the moment the request is written to the moment the client reads the notification. # of samples: 64 301k notifications received in 64.06s (301,000 of 300,000 expected), 46.2 MB read ``` **As many as the server takes** Calls / second 71 4,266 calls in 60 s, 10 connections Notifications / second 69,874 4,275,000 received by 1,000 clients Delivery p50 126ms average 124 ms Delivery p99 207ms max 270 ms Not delivered 0 0 calls failed (Server during this run) API Maker **1.8 vCPU** · Redis **0.1 vCPU** · whole server busy **98 %** · load generator busy **54 %** autocannon · A broadcast to 1,000 clients, as many as the server takes ```text $ autocannon -c 10 -d 60 -w 2 -l -m POST -b '{"name":"announcements","eventData":{"t":,"type":"announcement","text":"Hello from API Maker"}}' -H "x-am-authorization: " -H "x-am-user-authorization: " -H "content-type: application/json" "http://:38246/api/system-api/bench/emit-event-ws" Running 60s test @ http://:38246/api/system-api/bench/emit-event-ws 10 connections 2 workers ┌─────────┬───────┬────────┬────────┬────────┬───────────┬──────────┬────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼───────┼────────┼────────┼────────┼───────────┼──────────┼────────┤ │ Latency │ 71 ms │ 141 ms │ 209 ms │ 223 ms │ 139.88 ms │ 33.12 ms │ 343 ms │ └─────────┴───────┴────────┴────────┴────────┴───────────┴──────────┴────────┘ ┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬───────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────┤ │ Req/Sec │ 64 │ 64 │ 70 │ 88 │ 71.1 │ 5.11 │ 64 │ ├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────┤ │ Bytes/Sec │ 23.1 kB │ 23.1 kB │ 25.2 kB │ 31.7 kB │ 25.6 kB │ 1.84 kB │ 23 kB │ └───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴───────┘ Req/Bytes counts sampled once per second. # of samples: 120 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 27 │ ├────────────┼──────────────┤ │ 0.01 │ 27 │ ├────────────┼──────────────┤ │ 0.1 │ 40 │ ├────────────┼──────────────┤ │ 1 │ 58 │ ├────────────┼──────────────┤ │ 2.5 │ 71 │ ├────────────┼──────────────┤ │ 10 │ 98 │ ├────────────┼──────────────┤ │ 25 │ 121 │ ├────────────┼──────────────┤ │ 50 │ 141 │ ├────────────┼──────────────┤ │ 75 │ 158 │ ├────────────┼──────────────┤ │ 90 │ 180 │ ├────────────┼──────────────┤ │ 97.5 │ 209 │ ├────────────┼──────────────┤ │ 99 │ 223 │ ├────────────┼──────────────┤ │ 99.9 │ 266 │ ├────────────┼──────────────┤ │ 99.99 │ 343 │ ├────────────┼──────────────┤ │ 99.999 │ 343 │ └────────────┴──────────────┘ 4k requests in 60.02s, 1.54 MB read ``` am_ws_load · what the clients received ```text $ am_ws_load listen "ws://:38246/?x-am-authorization=&x-am-user-authorization=" 1,000 clients subscribed, 64.19s ┌──────────┬─────────┬──────────┬───────────┬───────────┬───────────┬──────────┬───────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├──────────┼─────────┼──────────┼───────────┼───────────┼───────────┼──────────┼───────────┤ │ Delivery │ 56.9 ms │ 125.5 ms │ 190.72 ms │ 206.59 ms │ 124.16 ms │ 32.88 ms │ 270.34 ms │ └──────────┴─────────┴──────────┴───────────┴───────────┴───────────┴──────────┴───────────┘ ┌───────────┬────┬──────┬────────┬────────┬──────────┬──────────┬─────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────┼──────┼────────┼────────┼──────────┼──────────┼─────┤ │ Notif/Sec │ 0 │ 0 │ 69,793 │ 89,366 │ 68,154.5 │ 14561.23 │ 0 │ └───────────┴────┴──────┴────────┴────────┴──────────┴──────────┴─────┘ Delivery : from the moment the request is written to the moment the client reads the notification. # of samples: 64 4.28M notifications received in 64.19s (4,275,000 of 4,266,000 expected), 655.9 MB read ``` ### Rows inserted in a table 100 clients subscribe to the insert API of a table (`SCHEMA_POST_BULK_INSERT` on `inserts_public`, `getEventData: true`) ; autocannon inserts rows of the [dataset](https://apimaker.dev/json/benchmark_dataset.json) through the [insert API](https://docs.apimaker.dev/v4/docs/apis-all/schema-apis/auto-generated-schema-based-save-single-or-multiple-api.html), and every row saved is told to each of the 100 clients, with its columns. **50 calls a second** Calls / second (paced) 50 3,000 calls in 60 s, 10 connections Notifications / second 4,970 301,000 received by 100 clients Delivery p50 35.1ms average 34.3 ms Delivery p99 54.8ms max 63.4 ms Not delivered 0 0 calls failed (Server during this run) API Maker **0.2 vCPU** · Redis **0.0 vCPU** · MongoDB **0.0 vCPU** · whole server busy **13 %** · load generator busy **11 %** autocannon · Inserts told to 100 clients, 50 inserts a second ```text $ autocannon -c 10 -d 60 -R 50 -l -m POST -b '{"customer_id":1,"first_name":"BENCH","last_name":"INSERT","phone":,"last_update":"2006-02-15T04:34:00.330Z","pincode":111111,"shipping_id":2,"dob":"2022-10-30T03:25:03.000Z","country":"India"}' -H "content-type: application/json" "http://:38246/api/schema/bench/mongodb/am_bench/inserts_public/save-single-or-multiple" Running 60s test @ http://:38246/api/schema/bench/mongodb/am_bench/inserts_public/save-single-or-multiple 10 connections ┌─────────┬──────┬───────┬───────┬───────┬─────────┬────────┬───────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼──────┼───────┼───────┼───────┼─────────┼────────┼───────┤ │ Latency │ 1 ms │ 10 ms │ 27 ms │ 30 ms │ 11.2 ms │ 7.3 ms │ 41 ms │ └─────────┴──────┴───────┴───────┴───────┴─────────┴────────┴───────┘ ┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬───────┬─────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────┼─────────┤ │ Req/Sec │ 50 │ 50 │ 50 │ 50 │ 50 │ 0 │ 50 │ ├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼───────┼─────────┤ │ Bytes/Sec │ 30.2 kB │ 30.2 kB │ 30.3 kB │ 30.3 kB │ 30.2 kB │ 4 B │ 30.2 kB │ └───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴───────┴─────────┘ Req/Bytes counts sampled once per second. # of samples: 60 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 1 │ ├────────────┼──────────────┤ │ 0.01 │ 1 │ ├────────────┼──────────────┤ │ 0.1 │ 1 │ ├────────────┼──────────────┤ │ 1 │ 1 │ ├────────────┼──────────────┤ │ 2.5 │ 1 │ ├────────────┼──────────────┤ │ 10 │ 2 │ ├────────────┼──────────────┤ │ 25 │ 5 │ ├────────────┼──────────────┤ │ 50 │ 10 │ ├────────────┼──────────────┤ │ 75 │ 16 │ ├────────────┼──────────────┤ │ 90 │ 21 │ ├────────────┼──────────────┤ │ 97.5 │ 27 │ ├────────────┼──────────────┤ │ 99 │ 30 │ ├────────────┼──────────────┤ │ 99.9 │ 35 │ ├────────────┼──────────────┤ │ 99.99 │ 39 │ ├────────────┼──────────────┤ │ 99.999 │ 41 │ └────────────┴──────────────┘ 3k requests in 60.06s, 1.81 MB read ``` am_ws_load · what the clients received ```text $ am_ws_load listen "ws://:38246/?x-am-authorization=&x-am-user-authorization=" 100 clients subscribed, 63.57s ┌──────────┬─────────┬──────────┬──────────┬──────────┬──────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├──────────┼─────────┼──────────┼──────────┼──────────┼──────────┼─────────┼──────────┤ │ Delivery │ 12.6 ms │ 35.07 ms │ 51.74 ms │ 54.75 ms │ 34.27 ms │ 9.91 ms │ 63.42 ms │ └──────────┴─────────┴──────────┴──────────┴──────────┴──────────┴─────────┴──────────┘ ┌───────────┬────┬──────┬───────┬───────┬─────────┬─────────┬─────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────┼──────┼───────┼───────┼─────────┼─────────┼─────┤ │ Notif/Sec │ 0 │ 0 │ 5,000 │ 5,000 │ 4,774.2 │ 1006.74 │ 0 │ └───────────┴────┴──────┴───────┴───────┴─────────┴─────────┴─────┘ Delivery : from the moment the request is written to the moment the client reads the notification. # of samples: 64 301k notifications received in 63.57s (301,000 of 300,000 expected), 94.4 MB read ``` **As many as the server takes** Calls / second 506 30,386 calls in 60 s, 100 connections Notifications / second 49,312 3,046,100 received by 100 clients Delivery p50 693ms average 674 ms Delivery p99 845ms max 999 ms Not delivered 0 0 calls failed (Server during this run) API Maker **1.4 vCPU** · Redis **0.1 vCPU** · MongoDB **0.3 vCPU** · whole server busy **98 %** · load generator busy **39 %** autocannon · Inserts told to 100 clients, as many as the server takes ```text $ autocannon -c 100 -d 60 -w 4 -l -m POST -b '{"customer_id":1,"first_name":"BENCH","last_name":"INSERT","phone":,"last_update":"2006-02-15T04:34:00.330Z","pincode":111111,"shipping_id":2,"dob":"2022-10-30T03:25:03.000Z","country":"India"}' -H "content-type: application/json" "http://:38246/api/schema/bench/mongodb/am_bench/inserts_public/save-single-or-multiple" Running 60s test @ http://:38246/api/schema/bench/mongodb/am_bench/inserts_public/save-single-or-multiple 100 connections 4 workers ┌─────────┬────────┬────────┬────────┬────────┬──────────┬──────────┬────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼────────┼────────┼────────┼────────┼──────────┼──────────┼────────┤ │ Latency │ 105 ms │ 196 ms │ 294 ms │ 322 ms │ 196.7 ms │ 47.66 ms │ 462 ms │ └─────────┴────────┴────────┴────────┴────────┴──────────┴──────────┴────────┘ ┌───────────┬────────┬────────┬────────┬────────┬────────┬─────────┬────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤ │ Req/Sec │ 445 │ 446 │ 483 │ 641 │ 506.44 │ 56.91 │ 445 │ ├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤ │ Bytes/Sec │ 269 kB │ 270 kB │ 292 kB │ 388 kB │ 306 kB │ 34.4 kB │ 269 kB │ └───────────┴────────┴────────┴────────┴────────┴────────┴─────────┴────────┘ Req/Bytes counts sampled once per second. # of samples: 240 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 23 │ ├────────────┼──────────────┤ │ 0.01 │ 32 │ ├────────────┼──────────────┤ │ 0.1 │ 50 │ ├────────────┼──────────────┤ │ 1 │ 87 │ ├────────────┼──────────────┤ │ 2.5 │ 105 │ ├────────────┼──────────────┤ │ 10 │ 138 │ ├────────────┼──────────────┤ │ 25 │ 165 │ ├────────────┼──────────────┤ │ 50 │ 196 │ ├────────────┼──────────────┤ │ 75 │ 227 │ ├────────────┼──────────────┤ │ 90 │ 255 │ ├────────────┼──────────────┤ │ 97.5 │ 294 │ ├────────────┼──────────────┤ │ 99 │ 322 │ ├────────────┼──────────────┤ │ 99.9 │ 397 │ ├────────────┼──────────────┤ │ 99.99 │ 430 │ ├────────────┼──────────────┤ │ 99.999 │ 462 │ └────────────┴──────────────┘ 30k requests in 60.03s, 18.4 MB read ``` am_ws_load · what the clients received ```text $ am_ws_load listen "ws://:38246/?x-am-authorization=&x-am-user-authorization=" 100 clients subscribed, 64.78s ┌──────────┬───────────┬───────────┬───────────┬───────────┬───────────┬──────────┬───────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├──────────┼───────────┼───────────┼───────────┼───────────┼───────────┼──────────┼───────────┤ │ Delivery │ 494.85 ms │ 693.25 ms │ 820.74 ms │ 845.31 ms │ 673.74 ms │ 94.63 ms │ 998.91 ms │ └──────────┴───────────┴───────────┴───────────┴───────────┴───────────┴──────────┴───────────┘ ┌───────────┬────┬──────┬────────┬────────┬──────────┬──────────┬─────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────┼──────┼────────┼────────┼──────────┼──────────┼─────┤ │ Notif/Sec │ 0 │ 0 │ 48,312 │ 63,866 │ 48,140.6 │ 11758.63 │ 0 │ └───────────┴────┴──────┴────────┴────────┴──────────┴──────────┴─────┘ Delivery : from the moment the request is written to the moment the client reads the notification. # of samples: 65 3.05M notifications received in 64.78s (3,046,100 of 3,038,600 expected), 955.4 MB read ``` ## With every client connected The same runs once more, with 338,006 clients connected and waiting : what a server which is full of WebSocket clients still does. | | No client, or 10,000 | 338,006 clients connected | |---|--:|--:| | A notification for one person, 1,000 a second : delivery p50 | 1.74 ms | **1.76 ms** | | … p99 | 3.70 ms | **3.03 ms** | | A notification for one person, as many as the server takes | 6,618 / s | **5,705 / s** | | Get all with both tokens, 100 connections (autocannon) | 3,058 req/s | **2,614 req/s** | | … latency p50 / p99 | 32 ms / 55 ms | **33 ms / 170 ms** | **Notification, 1,000 a second** Calls / second (paced) 1,000 60,005 calls in 60 s, 10 connections Notifications / second 983 60,015 received by 338,006 clients Delivery p50 1.76ms average 1.79 ms Delivery p99 3.03ms max 41.1 ms Not delivered 0 0 calls failed (Server during this run) API Maker **0.4 vCPU** · Redis **0.0 vCPU** · whole server busy **28 %** · load generator busy **19 %** autocannon · A notification for one person, 1,000 a second, every client connected ```text $ autocannon -c 10 -d 60 -R 1000 -l -m POST -b '{"name":"order-updates","eventData":{"recipientId":"","t":,"type":"order-updated","order_no":1001,"status":"SHIPPED"}}' -H "x-am-authorization: " -H "x-am-user-authorization: " -H "content-type: application/json" "http://:38246/api/system-api/bench/emit-event-ws" Running 60s test @ http://:38246/api/system-api/bench/emit-event-ws 10 connections ┌─────────┬──────┬──────┬───────┬──────┬─────────┬─────────┬───────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼──────┼──────┼───────┼──────┼─────────┼─────────┼───────┤ │ Latency │ 1 ms │ 1 ms │ 3 ms │ 5 ms │ 1.46 ms │ 1.79 ms │ 41 ms │ └─────────┴──────┴──────┴───────┴──────┴─────────┴─────────┴───────┘ ┌───────────┬────────┬────────┬────────┬────────┬──────────┬───────┬────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────────┼────────┼────────┼────────┼──────────┼───────┼────────┤ │ Req/Sec │ 998 │ 999 │ 1,000 │ 1,002 │ 1,000.09 │ 0.62 │ 998 │ ├───────────┼────────┼────────┼────────┼────────┼──────────┼───────┼────────┤ │ Bytes/Sec │ 356 kB │ 357 kB │ 357 kB │ 358 kB │ 357 kB │ 207 B │ 356 kB │ └───────────┴────────┴────────┴────────┴────────┴──────────┴───────┴────────┘ Req/Bytes counts sampled once per second. # of samples: 60 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 0 │ ├────────────┼──────────────┤ │ 0.01 │ 0 │ ├────────────┼──────────────┤ │ 0.1 │ 0 │ ├────────────┼──────────────┤ │ 1 │ 1 │ ├────────────┼──────────────┤ │ 2.5 │ 1 │ ├────────────┼──────────────┤ │ 10 │ 1 │ ├────────────┼──────────────┤ │ 25 │ 1 │ ├────────────┼──────────────┤ │ 50 │ 1 │ ├────────────┼──────────────┤ │ 75 │ 2 │ ├────────────┼──────────────┤ │ 90 │ 2 │ ├────────────┼──────────────┤ │ 97.5 │ 3 │ ├────────────┼──────────────┤ │ 99 │ 5 │ ├────────────┼──────────────┤ │ 99.9 │ 32 │ ├────────────┼──────────────┤ │ 99.99 │ 40 │ ├────────────┼──────────────┤ │ 99.999 │ 41 │ └────────────┴──────────────┘ 61k requests in 60.04s, 21.4 MB read ``` am_ws_load · what the clients received ```text $ am_ws_load listen "ws://:38246/?x-am-authorization=&x-am-user-authorization=" 338,006 clients subscribed, 64.05s ┌──────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├──────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼──────────┤ │ Delivery │ 1.01 ms │ 1.76 ms │ 2.71 ms │ 3.03 ms │ 1.79 ms │ 0.54 ms │ 41.06 ms │ └──────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴──────────┘ ┌───────────┬────┬──────┬───────┬───────┬───────┬────────┬─────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────┼──────┼───────┼───────┼───────┼────────┼─────┤ │ Notif/Sec │ 0 │ 0 │ 1,000 │ 1,000 │ 951.9 │ 213.52 │ 0 │ └───────────┴────┴──────┴───────┴───────┴───────┴────────┴─────┘ Delivery : from the moment the request is written to the moment the client reads the notification. # of samples: 64 60k notifications received in 64.05s (60,015 of 60,005 expected), 10.8 MB read ``` **Notification, as many as the server takes** Calls / second 5,791 347,446 calls in 60 s, 100 connections Notifications / second 5,705 347,546 received by 338,006 clients Delivery p50 15.1ms average 17.1 ms Delivery p99 64.8ms max 263 ms Not delivered 0 0 calls failed (Server during this run) API Maker **1.6 vCPU** · Redis **0.1 vCPU** · whole server busy **100 %** · load generator busy **45 %** autocannon · A notification for one person, as many as the server takes, every client connected ```text $ autocannon -c 100 -d 60 -w 4 -l -m POST -b '{"name":"order-updates","eventData":{"recipientId":"","t":,"type":"order-updated","order_no":1001,"status":"SHIPPED"}}' -H "x-am-authorization: " -H "x-am-user-authorization: " -H "content-type: application/json" "http://:38246/api/system-api/bench/emit-event-ws" Running 60s test @ http://:38246/api/system-api/bench/emit-event-ws 100 connections 4 workers ┌─────────┬──────┬───────┬───────┬───────┬──────────┬──────────┬────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼──────┼───────┼───────┼───────┼──────────┼──────────┼────────┤ │ Latency │ 9 ms │ 15 ms │ 42 ms │ 66 ms │ 16.73 ms │ 11.86 ms │ 263 ms │ └─────────┴──────┴───────┴───────┴───────┴──────────┴──────────┴────────┘ ┌───────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤ │ Req/Sec │ 5,239 │ 5,379 │ 5,759 │ 6,279 │ 5,791 │ 221.85 │ 5,239 │ ├───────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤ │ Bytes/Sec │ 1.87 MB │ 1.92 MB │ 2.06 MB │ 2.24 MB │ 2.07 MB │ 79.2 kB │ 1.87 MB │ └───────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘ Req/Bytes counts sampled once per second. # of samples: 240 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 1 │ ├────────────┼──────────────┤ │ 0.01 │ 2 │ ├────────────┼──────────────┤ │ 0.1 │ 3 │ ├────────────┼──────────────┤ │ 1 │ 7 │ ├────────────┼──────────────┤ │ 2.5 │ 9 │ ├────────────┼──────────────┤ │ 10 │ 11 │ ├────────────┼──────────────┤ │ 25 │ 13 │ ├────────────┼──────────────┤ │ 50 │ 15 │ ├────────────┼──────────────┤ │ 75 │ 17 │ ├────────────┼──────────────┤ │ 90 │ 20 │ ├────────────┼──────────────┤ │ 97.5 │ 42 │ ├────────────┼──────────────┤ │ 99 │ 66 │ ├────────────┼──────────────┤ │ 99.9 │ 162 │ ├────────────┼──────────────┤ │ 99.99 │ 235 │ ├────────────┼──────────────┤ │ 99.999 │ 260 │ └────────────┴──────────────┘ 348k requests in 60.03s, 124 MB read ``` am_ws_load · what the clients received ```text $ am_ws_load listen "ws://:38246/?x-am-authorization=&x-am-user-authorization=" 338,006 clients subscribed, 63.93s ┌──────────┬─────────┬──────────┬──────────┬─────────┬──────────┬──────────┬───────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├──────────┼─────────┼──────────┼──────────┼─────────┼──────────┼──────────┼───────────┤ │ Delivery │ 9.13 ms │ 15.12 ms │ 41.76 ms │ 64.8 ms │ 17.06 ms │ 11.71 ms │ 263.42 ms │ └──────────┴─────────┴──────────┴──────────┴─────────┴──────────┴──────────┴───────────┘ ┌───────────┬────┬──────┬───────┬───────┬─────────┬─────────┬─────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼────┼──────┼───────┼───────┼─────────┼─────────┼─────┤ │ Notif/Sec │ 0 │ 0 │ 5,738 │ 6,268 │ 5,559.6 │ 1058.35 │ 0 │ └───────────┴────┴──────┴───────┴───────┴─────────┴─────────┴─────┘ Delivery : from the moment the request is written to the moment the client reads the notification. # of samples: 64 348k notifications received in 63.93s (347,546 of 347,446 expected), 62.9 MB read ``` **Get all with both tokens** Requests / second 2,614 156,776 requests in 60 s Latency p50 33ms average 37.77 ms Latency p99 170ms max 413 ms Errors 0 non-2xx, errors, timeouts (Server during this run) API Maker **1.1 vCPU** · Redis **0.0 vCPU** · MongoDB **0.6 vCPU** · whole server busy **100 %** · load generator busy **34 %** autocannon · get all with both tokens · 338,006 WebSocket clients connected ```text $ autocannon -c 100 -d 60 -w 4 -l -H "x-am-authorization: " -H "x-am-user-authorization: " "http://:38246/api/schema/bench/mongodb/am_bench/customers" Running 60s test @ http://:38246/api/schema/bench/mongodb/am_bench/customers 100 connections 4 workers ┌─────────┬───────┬───────┬────────┬────────┬──────────┬──────────┬────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼───────┼───────┼────────┼────────┼──────────┼──────────┼────────┤ │ Latency │ 14 ms │ 33 ms │ 103 ms │ 170 ms │ 37.77 ms │ 27.63 ms │ 413 ms │ └─────────┴───────┴───────┴────────┴────────┴──────────┴──────────┴────────┘ ┌───────────┬─────────┬────────┬─────────┬─────────┬──────────┬────────┬─────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────────┼────────┼─────────┼─────────┼──────────┼────────┼─────────┤ │ Req/Sec │ 2,459 │ 2,523 │ 2,607 │ 2,765 │ 2,613.47 │ 64.5 │ 2,459 │ ├───────────┼─────────┼────────┼─────────┼─────────┼──────────┼────────┼─────────┤ │ Bytes/Sec │ 6.73 MB │ 6.9 MB │ 7.13 MB │ 7.57 MB │ 7.15 MB │ 176 kB │ 6.73 MB │ └───────────┴─────────┴────────┴─────────┴─────────┴──────────┴────────┴─────────┘ Req/Bytes counts sampled once per second. # of samples: 240 ┌────────────┬──────────────┐ │ Percentile │ Latency (ms) │ ├────────────┼──────────────┤ │ 0.001 │ 0 │ ├────────────┼──────────────┤ │ 0.01 │ 1 │ ├────────────┼──────────────┤ │ 0.1 │ 3 │ ├────────────┼──────────────┤ │ 1 │ 10 │ ├────────────┼──────────────┤ │ 2.5 │ 14 │ ├────────────┼──────────────┤ │ 10 │ 21 │ ├────────────┼──────────────┤ │ 25 │ 27 │ ├────────────┼──────────────┤ │ 50 │ 33 │ ├────────────┼──────────────┤ │ 75 │ 40 │ ├────────────┼──────────────┤ │ 90 │ 49 │ ├────────────┼──────────────┤ │ 97.5 │ 103 │ ├────────────┼──────────────┤ │ 99 │ 170 │ ├────────────┼──────────────┤ │ 99.9 │ 322 │ ├────────────┼──────────────┤ │ 99.99 │ 383 │ ├────────────┼──────────────┤ │ 99.999 │ 413 │ └────────────┴──────────────┘ 157k requests in 60.04s, 429 MB read ``` ## A restart API Maker was restarted (`systemctl restart api_maker_be`) with 253,504 clients connected : the newest left first, down to 75 % of the 338,006 it held. Every client of a restarted server comes back within the same seconds and waits for its turn with its request in the kernel, about 4 KB each : a server needs that room. It takes what Redis keeps of them out first, then closes every client with the code `1001` (*Server is going away*) ; each client waits one to five seconds, as an application does, connects again with the same tokens and sends its `REGISTER` again. The server which comes back lets 256 clients in at a time and handles the messages of 256 at a time, the others wait for their turn : its memory comes back to what it was, not higher. Clients dropped 253,504 of 253,504 connected All back after 159.1s the restart itself : 29.6 s A client was away, p50 68.7s its own wait of 1 to 5 s counted A client was away, p99 96.5s the longest : 154 s After the restart, with the 253,504 clients back : API Maker 945 MB (1,108 MB before it), 1,125 MB of memory available. They were then told again, each for itself, 1,000 notifications a second for 60 seconds : **all 60,000 arrived**, in 1.75 ms (p50) and 2.94 ms (p99). A client whose subscription had not come back with it would have been told nothing. ### Every client leaves at once At the end of the run, the 253,504 clients closed their connection in the same second, as when a network or a load balancer in front of the server goes. The server had closed every one of their sockets after **15.1 s**, and Redis held nothing of them any more after **16.1 s**. A socket is closed at once, and Redis is told behind it. ## Questions ### How many WebSocket connections can API Maker hold on 2 dedicated vCPUs and 4 GB ? 338,006 clients were connected at the same time on this server, each with both tokens and a subscription of its own, with MongoDB, Redis and two sandboxes on the same machine. Memory is what ends first : a client costs about 8.5 KB of the server in all. ### How much memory does a WebSocket client cost ? About 8.5 KB : 3.1 KB in the API Maker process, 4.3 KB in the Linux kernel (its socket) and 1.0 KB in Redis (its subscription). Measured between 100,000 and 250,000 clients which wait, each connected with both tokens and one subscription. ### How many clients can connect each second ? 3,464 clients a second connected with both tokens, subscribed and left, 100 at a time ; 7,077 a second without a subscription. With a can user connect code on the event : 2,891 a second. ### How fast does a notification reach a client ? At 1,000 notifications a second, each for one person among 10,000 : 1.74 ms from the HTTP call to the client (p50), 3.70 ms for 99 % of them. With as many calls as the server takes : 6,618 notifications a second. ### Do connected WebSocket clients slow the APIs down ? Get all with both tokens answered 3,058 requests a second with no WebSocket client, and 2,614 with 338,006 clients connected and waiting. A client which waits costs memory ; its CPU is a ping a minute.