This page View Markdown Open in ChatGPT Open in Claude

How WebSockets are benchmarked¶

The WebSocket benchmarks answer four questions : how many clients one server holds, how much memory a client takes, how to serve N users, and whether one big server beats several small ones. This page shows how every number was measured, so you can check it, or measure it again.


Servers	The seven Linode servers of the API benchmarks, in Mumbai, from 1 to 8 vCPUs, with MongoDB and Redis on the same machine. Plus three clusters of the same price : 8 dedicated vCPUs and 16 GB as one, two or four API servers, with MongoDB and Redis on a server of their own.
Clients	Real WebSocket clients, from a second server in the same data centre. Each one connects with two tokens, subscribes to its own notifications, and answers the pings of the server.
HTTP calls	autocannon 8, as in the API benchmarks : the emit event WS API, the insert API of a table, get all.
Runs	60 seconds each, after 5 seconds of warm-up. All the servers were measured side by side on 14 October 2026, each with its own load generator.
Installation	The API Maker installer with its default answers and no reverse proxy : clients reach API Maker on its own port `38246`. One setting changed : the sandboxes keep running.
Data	The benchmark dataset : 10 customers with 10 columns each, in MongoDB.
Samples	Once a second, on every server and on the load generator : memory, CPU (with steal time), the API Maker process and its open files, every container, the sockets of the kernel and the connections its firewall tracks.

What a client is¶

Every client does what a real application does :

const ws = new WebSocket('ws://<server>:38246/'
    + '?x-am-authorization=' + apiUserToken             // the token of an API user
    + '&x-am-user-authorization=' + personToken);       // the token of a person : a row of `customers`
ws.onmessage = (e) => {
    const msg = JSON.parse(e.data);
    if (msg.type === 'CONNECTED') {
        ws.send(JSON.stringify({
            objType: 'REGISTER',
            onEvents: [{
                eventType: 'CUSTOM_WS_EVENTS', apiName: 'order-updates',
                condition: { conditionType: 'RESPONSE', criteria: { recipientId: myId } },   // its own id
                getEventData: true,
            }],
        }));
    } else if (msg.type === 'NOTIFICATION') {
        received(msg.response.eventData);
    }
};

Both tokens : x-am-authorization is the token of an API user, x-am-user-authorization the token of a row of the customers table, made by a DB token generator. API Maker checks both when the client connects, and closes the client when one of them runs out.
Its own subscription : every client waits for the notifications of one person, itself. This is the pattern of the WebSocket events page, and the most expensive one for Redis : a key, a place in a list and a channel for every client.
It answers the pings : API Maker pings every client once a minute, and takes a client which sent nothing for two and a half minutes for gone.

A browser or a Node.js process holds a few thousand such clients. To open millions from one machine, the benchmarks use am_ws_load, a small Rust program written for them : a client is a socket, a buffer of 512 bytes and a task, about 3 KB of the load generator. It speaks WebSocket the way a browser does (handshake, masked frames, pongs, close), counts every message, and measures how late each notification arrives.

The servers¶

	Server under test	Load generator
Runs	API Maker, its MongoDB and Redis, its two sandboxes	The WebSocket clients (`am_ws_load`) and autocannon
Plan	Shared or dedicated CPU, as named on each page	A dedicated Linode, big enough to hold as many sockets as the server it measures : 4 vCPUs for the servers of 1 and 2 vCPUs, 8 for 4 vCPUs, 16 for 16 GB and the clusters, 32 (64 GB) for 32 GB
System	Ubuntu 24.04 LTS	Ubuntu 24.04 LTS, Node.js 24.21.0, autocannon 8.0.0
Network	A VLAN of the data centre	The same VLAN, 0.4 ms away (ping)

The load generator samples itself every second too : every page says how busy it was at most.

More than 64,000 clients from one machine¶

A TCP connection is named by four numbers : the address and port it comes from, and the address and port it goes to. So one address can open only about 64,000 connections to one port of a server. The load generator takes more addresses on the VLAN, and keeps each one under about 25,000 connections to a server : 10 addresses for the server of 1 GB, 20 for 4 GB, 40 for 8 GB, 100 for 16 GB and 150 for 32 GB. (Past about 32,000 connections from one address, Linux 6 spends a lot of CPU finding the port of each new one.) It also gets room in its own tables :

sysctl -w net.ipv4.ip_local_port_range="1024 65535" net.ipv4.tcp_tw_reuse=1 fs.nr_open=8388608 net.netfilter.nf_conntrack_max=8388608
sysctl -w net.ipv4.tcp_mem="<a quarter> <three eighths> <half of the memory of the machine, in pages>"

The last line is for the moments when every client does something at once : unless told otherwise, Linux gives all the TCP connections of a machine together 9 % of its memory. (On the server under test, API Maker raises that memory itself, as its clients come.)

A reverse proxy in front of API Maker meets the same limit : it reaches one server from one address about 64,000 times, 28,000 as Linux sets it up. Scale to N users says what to do about it.

The installation¶

API Maker is installed on a fresh server with its installer, its default answers and no reverse proxy :

bash install.sh --default --primary --caddy=no --ssl=no --qa_uat_prod=no --package=api-maker-4.0.0-linux-x64.tar.gz

The installer sets up Docker, MongoDB 8.0.10 and Redis 7.0.5 in Docker, swap, and API Maker as a systemd service. It also makes room for many connections :

Open files : every client is an open file of the service. The service may open as many as the firewall tracks connections (256 for each MB of memory, 1,048,576 at least : LimitNOFILE is 8,216,320 on the server of 32 GB), and Linux lets a process have that many (fs.nr_open).

The network : for a server of 8 GB, for example :

/etc/sysctl.d/90-api-maker.conf, on a server of 8 GB

net.netfilter.nf_conntrack_max = 2032896
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.core.somaxconn = 16384
net.ipv4.tcp_max_syn_backlog = 16384
fs.nr_open = 4065792

with LimitNOFILE=2032896 in the service of API Maker.

While it runs, API Maker raises the memory Linux gives all its TCP connections together (net.ipv4.tcp_mem) as its clients come : two pages for each open connection, never above a quarter of the memory of the server. The page of each server says what it was at the end of the run.

One setting is changed, in ~/config/.env, as in the API benchmarks : am__sandbox__removeSandboxInactiveSinceMinutes=1440 keeps the two sandboxes of the account running for the whole run, so their memory is counted.

The table of tracked connections¶

Docker makes Linux track every connection of the server in a table (nf_conntrack), the ones of API Maker too. As Linux sets it up, that table holds 262,144 connections on a server of 4 GB or more (65,536 on 1 GB), and a connection which closed stays in it for two more minutes. Once the table is full, the kernel drops new connections without a word, except in its own log (nf_conntrack: table full, dropping packet).

So the installer gives the table room for 256 connections per MB of memory, twice the clients the memory holds (131,072 at least, 16,777,216 at most : an entry is about 320 bytes, taken only while its connection is there), and lets a closed connection go after 30 seconds. API Maker says at its start how many connections the table holds, and warns when that is fewer than its memory has room for. On a server installed before, run the installer again, or :

sysctl -w net.netfilter.nf_conntrack_max=1048576 net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

The account¶

The account of the API benchmarks (its tables in MongoDB, its API user, its DB token generator), with five events on its WebSocket Events page, each taking the token of the DB token generator :

Event	What it is
`order-updates`	A notification for one person : every client subscribes to it with its own id in the criteria.
`announcements`	A broadcast : clients subscribe to it without a condition, and one emit reaches all of them.
`checked-token`	`order-updates` with a can user connect code which reads the person of the token.
`checked-database`	The same, with a code which also reads the person from the database.
`checked-sandbox`	The same, with a code which reads the clock, so that it runs in a sandbox.
Insert of `inserts_public`	The event of an API of a table (`SCHEMA_POST_BULK_INSERT`) : every saved row is told to the clients which subscribed.

A system API answers nobody until its settings say who may call it. The emit event WS API takes both tokens :

System API settings of EMIT_EVENT_WS

import * as T from 'types';

let systemApi: T.ISystemApiSettingsTypes = {
    enableCaching: false,
    acceptOnlyEncryptedData: false,
    apiAccessType: T.EAPIAccessType.TOKEN_ACCESS,
    authProviders: ['bench_token_mongodb'],
};
module.exports = systemApi;

The three can user connect codes are on the page of each server, with where each of them ran. Before the runs, a real WebSocket client (the one of Node.js) checks every event : a client with both tokens subscribes to each one, a notification for one person reaches that person and nobody else, a broadcast reaches everybody, an insert reaches the clients of its table with its row, a client without the token of the person is refused by every event, and a client with a wrong token is told why and closed with 1000.

The runs¶

Every measured run lasts 60 seconds, and the server rests 10 seconds between two runs. When the server is only watched with its clients connected (at rest, at each level of clients), it is watched for 30 seconds.

At rest¶

API Maker is restarted, its two sandboxes are started, the swap is emptied back into memory, and the server waits 30 seconds with no client : this is the memory at rest of the pages. Then autocannon loads get all with both tokens (100 connections), with no WebSocket client connected.

New clients per second¶

100 clients at a time connect, are let in and leave, one after the other, for 60 seconds after 5 seconds of warm-up : what autocannon -c 100 -d 60 does for requests, for clients. A client which leaves sends a close and waits for the close of the server. Six runs :

Run	What each client does
Connect : the token of the API user	Connects with `x-am-authorization`, is told `CONNECTED`, leaves.
Connect : both tokens	The same, with `x-am-user-authorization` too.
Connect and subscribe	Both tokens, `CONNECTED`, `REGISTER` for `order-updates` with its own criteria, the answer, leaves.
… with a can user connect code	The same on `checked-token`, `checked-database` and `checked-sandbox`.

The times count from the start of the TCP connection : to the connection itself, to CONNECTED, to the answer of REGISTER, and to the end of the close.

Notifications¶

10,000 clients connect and stay (5,000 on the server of 1 GB), each with its subscription. autocannon then makes the HTTP calls which notify them, and the clients count what they receive :

Kind	The call	Who is told
For one person	`POST /api/system-api/<user path>/emit-event-ws`, both tokens, `{ name, eventData: { recipientId, … } }`	The one client whose criteria is that `recipientId` : every call names another one.
A broadcast	The same API, on `announcements`	1,000 more clients, which subscribed without a condition.
Rows of a table	`POST /api/schema/<user path>/mongodb/am_bench/inserts_public/save-single-or-multiple`, a row of the dataset	100 clients which subscribed to the insert API of the table, each told the row.

Each kind is measured twice :

At a pace far under what the server takes : 1,000 notifications a second for one person, 5 broadcasts a second, 50 inserts a second. It shows how long a notification takes.
With no pace : as many calls as the server answers (100 connections ; 10 for the broadcast). It shows how many notifications a second the server tells. When calls come faster than the server tells its clients, the notifications wait their turn inside API Maker and keep arriving after the calls stopped : the run waits for all of them, and notifications per second counts that whole time.

How a delivery is timed. autocannon writes the time into the body of every request, just before it sends it (t of the emitted data, phone of an inserted row : milliseconds since 1970). The client reads it back in the notification and compares it with its own clock. autocannon and the clients run on the same machine, so both times come from one clock. A delivery is the whole way : the HTTP call, API Maker, Redis, the WebSocket.

Calls and notifications. autocannon counts the answers it read before its 60 seconds end. The calls it had sent by then (one per connection at most) are answered too, and what they emit is told, so the clients receive a few more notifications than calls are counted : ten more with 10 connections, a thousand more when one broadcast to 1,000 clients was on its way. Not delivered counts the notifications of the counted calls which no client received.

As many clients as fit¶

More clients connect, level after level (for a server of 8 GB : 50,000, 100,000, then every 100,000), at a pace the server follows : 1,000 a second per vCPU, about four fifths of what it lets in when every client subscribes. At each level they wait 30 seconds while the server is sampled, and Redis is asked what it holds ; the last level is held 60 seconds. Clients stop being added when one of these happens :

The memory is full : what the server has available (MemAvailable), minus what Linux moved to the swap since the clients began, is under 5 % of its memory (64 MB at least). Linux moves memory nobody touches (of MongoDB, of the sandboxes) to the swap long before a server is short of it, which makes as much memory available again : that is counted as memory the clients took. So the clients of the pages fit in memory, not in the swap.
Nobody else is let in : no client is let in for 20 seconds, or more than 1 % of a level is refused. API Maker answers 503 past the connections it takes (its open files, minus a share kept for the calls of the APIs).
The most the run opens for that server, which stays under the table of the firewall : a full table drops every new connection, SSH included.

Per client is the slope of a line through the levels : what one more client costs of API Maker (the resident memory of its process, plus what Linux moved of it to the swap on a small server), of the kernel (its slab, which holds the sockets and the tracked connections), of Redis (the memory of its container, and of its data) and of the whole server (what it has less of available).

One more subscription, and a broadcast to everybody¶

At the second level (the first one on the server of 1 GB), with memory still left, every connected client sends a second REGISTER, for announcements. The server waits 30 seconds with two subscriptions per client : the difference with the level before is what a subscription costs. Then three calls of the emit event WS API tell everybody, one after the other, and the clients time the first notification, the median and the last. Every client then takes its second subscription back (UNREGISTER), and the levels go on.

With every client connected¶

Once the server is full : the notification for one person again, at a pace and with no pace, and get all with both tokens (autocannon, 100 connections), while every client waits.

A restart¶

API Maker is restarted (systemctl restart api_maker_be) with three quarters of the clients it holds : the newest ones leave first. Nobody runs a server full, and a restarted server needs room : every client comes back within the same seconds, and waits for its turn with its request in the kernel, about 4 KB each.

API Maker closes its clients with the code 1001 (Server is going away). Each client waits one second, plus up to four more of its own choosing, then connects again and sends its REGISTER again, as the WebSocket events page tells an application to do. The server which comes back lets 256 clients in at a time, and handles the messages of 256 at a time : the others wait for their turn. The run counts the clients until they are all back, and how long each one was away. Then every client is told again, each for itself, 1,000 notifications a second for 60 seconds : a client whose subscription had not come back would be told nothing. Redis is asked too, whether it holds the same channels and keys as before the restart.

Every client leaves at once¶

At the very end, every client closes its connection in the same second, as when a network or a load balancer in front of the server goes down. The run counts the seconds until the server has no socket of a client left open, and until Redis holds nothing of them any more : Redis is asked every two seconds, and once more five seconds after it was found empty of them.

Several servers¶

A cluster is one data server (the MongoDB and Redis of API Maker, installed with --primary ; its own API Maker is stopped) and API servers installed with --secondary, which hold the clients. The clients go to the API servers in turn, as a load balancer which counts connections hands them out, and autocannon calls every API server at the same time, its 100 connections shared out. The runs are the ones above, and clients are added until the memory of one of the servers is full.

Instead of a restart, one API server stops and stays down : its clients go to the servers which are left, as a load balancer spreads them. The servers which are left need room for those clients, so before the server stops, the newest clients leave, down to what the other servers hold when each of them is 80 % full (40 % of the clients a cluster of two held, 60 % with four).

No load balancer stands between the clients and the servers : the pages measure API Maker. A proxy adds its own memory per connection, and the ports it reaches a server from.

The data server takes MongoDB and Redis off the API servers, so an API server of a cluster has all its memory for clients : the 8 vCPU · 16 GB server holds 2,022,026 clients as a cluster of one, and 1,732,035 on its own page, where MongoDB and Redis share its memory.

Memory and CPU¶

A small script on every server reads /proc and the cgroups of the containers once a second :

Measure	What it is
API Maker	The resident memory (RSS) of the `api_maker_be` process, its CPU in percent of one core, its open files.
Containers	Redis, MongoDB and the sandboxes : memory as `docker stats` counts it, and CPU.
Linux kernel	Its slab : the sockets, the tracked connections, and everything else the kernel keeps. The TCP sockets in use and the connections the firewall tracks are counted too.
Whole server	Memory used (total − available), swap used, pages swapped in and out.
Pressure	The share of the last ten seconds in which tasks of the server waited for memory, and for the CPU (`/proc/pressure`) : a server short of memory waits for it, one which only moved what nobody touches to its swap does not.
CPU	User, system, softirq, idle and steal : the time the hypervisor gave to other servers.

Redis is asked too, after each level and never during a run : the memory of its data (used_memory), its keys, and the channels the servers listen on.

What these numbers are not¶

No HTTPS, no reverse proxy. Caddy (which the installer puts in front by default) and TLS take memory for every connection and CPU for every handshake, on the machine they run on.
Clients which wait. A client with one subscription, told a notification now and then, is what most applications have. A client told a hundred messages a second costs CPU and a larger buffer.
One data centre. The clients are 0.4 ms away. Clients on the internet have slower connections : more of what they are told waits in the buffers of the server.
One person behind every token. All clients connect with the tokens of the same API user and the same person. API Maker keeps what the tokens said for each client, so different people would cost the same.
Not a published version. The pages say API Maker 4.0.0 : the number of the build that was measured (commit 423a34c and the WebSocket core of 14 October 2026), before the beta releases were numbered.

Run it yourself¶

Make two servers in the same data centre, and install API Maker on the first one.
Add an event to WebSocket Events, give your group all WebSocket events, and give the emit event WS system API its settings, as above.

On the second server, open your clients with any WebSocket library and the code of What a client is, ask for the two tokens with the token API, and call the emit event WS API with autocannon :

autocannon -c 100 -d 60 -m POST \
    -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" -H "content-type: application/json" \
    -b '{"name":"order-updates","eventData":{"recipientId":"42","order_no":1001}}' \
    "http://<server>:38246/api/system-api/<user path>/emit-event-ws"

The results : WebSocket benchmarks.