This page View Markdown Open in ChatGPT Open in Claude

How API Maker is benchmarked¶

The benchmarks measure what you would get yourself : API Maker set up by its own installer on a plain cloud server, the databases on the same machine, and the load sent from a second server. This page shows every step, so you can check any number, or run the benchmarks again.


Servers	7 Linode servers in Mumbai, from 1 to 8 vCPUs, shared and dedicated CPU, all on the same CPU (AMD EPYC 7642). Each one runs API Maker and its databases.
Load	autocannon 8 on a second, dedicated server in the same data centre : 100 connections, 5 seconds of warm-up, then 15 seconds measured.
Rounds	Every API runs twice, the second time in reverse order. The pages show the average of the two.
Data	The benchmark dataset : 10 customers with 10 columns each, in MongoDB, PostgreSQL and MySQL.
Installation	The API Maker installer with its default answers and no reverse proxy. One setting changed : the sandboxes keep running.
Checks	Every API answers once before its load : the status and the rows are checked, and an API with tokens refuses a request with none, or only one. Every error, non-2xx answer and timeout is counted.

The servers¶

Every benchmark uses two servers in the same data centre :

	Server under test	Load generator
Runs	API Maker, its MongoDB and Redis, PostgreSQL, MySQL, the sandboxes	autocannon, nothing else
Plan	Shared or dedicated CPU, as named on each page	Dedicated CPU, four times the vCPUs of the server under test
System	Ubuntu 24.04 LTS	Ubuntu 24.04 LTS, Node.js 24.21.0, autocannon 8.0.0
Network	Private network of the data centre	The same network, 0.5 ms away (ping)

The load comes from its own machine, so autocannon never takes CPU away from API Maker. That machine has dedicated CPUs, so its speed does not depend on its neighbours, and it is big enough never to be the limit : it was never more than 29 % busy during a run (42 % for the 1 vCPU server). Every page shows how busy it was.

Server under test	Load generator	autocannon threads
1 vCPU	Dedicated 8 GB (g6-dedicated-4) : 4 vCPUs	4 (25 connections each)
2 vCPU, shared and dedicated	Dedicated 16 GB (g6-dedicated-8) : 8 vCPUs	5 (20 connections each)
4 vCPU, shared and dedicated	Dedicated 32 GB (g6-dedicated-16) : 16 vCPUs	10 (10 connections each)
8 vCPU, shared and dedicated	Dedicated 64 GB (g6-dedicated-32) : 32 vCPUs	20 (5 connections each)

Enough autocannon threads¶

One autocannon thread cannot keep up with the fastest APIs. On the 1 vCPU server, get by id with caching gave about 10,400 requests per second with one thread, and about 12,000 with four, at the same latency. So autocannon runs several threads, which share the 100 connections.

To be sure the threads were not the limit, the same API was loaded again after each run, with more threads. It gave the same requests per second :

Server	Threads of the run : req/s	More threads : req/s
2 vCPU, shared	5 : 19,709 and 19,524	10 : 19,766 and 20,058
2 vCPU, dedicated	5 : 16,442 and 16,466	10 : 16,176 and 16,412
4 vCPU, shared	10 : 26,932 and 27,201	20 : 27,304 and 26,271
4 vCPU, dedicated	10 : 32,748 and 32,841	20 : 32,998 and 32,970
8 vCPU, shared	20 : 41,740 and 41,994	25 : 42,075 and 42,190
8 vCPU, dedicated	20 : 48,650 and 48,364	25 : 47,617 and 47,700

The same CPU, at the same time¶

A Linode server runs on whatever CPU its host has. All seven servers under test run on the AMD EPYC 7642, so every size and both plans are compared on the same CPU. The load generators run on AMD EPYC 7642 or 7713 ; each page names its own.

All seven servers were measured at the same time, each with its own load generator. Linode placement groups (strict anti-affinity) kept the three shared CPU servers of 2, 4 and 8 vCPUs on three different hosts, and the three dedicated servers too. Every page shows the steal time of each run : the CPU the hypervisor gave to other servers.

Why 100 connections¶

The fastest server (8 dedicated vCPUs) was loaded with 100, 200 and 400 connections, 20 seconds each :

API	100 connections	200	400	Server busy
Get by id with caching, public, MongoDB	45,742	46,061	44,016	87 to 90 %
Get all, public, MongoDB	11,059	11,114	11,049	95 %
Custom API with both tokens, PostgreSQL	5,395	5,962	6,128	99 to 100 %

At 100 connections the server already works at full speed. More connections mostly add waiting : the latency doubles at each step, for few more requests. The more connections runs of every page, from 10 to 1,000, say the same for get all : above 100 connections it gained 2.0 % at most. Only get by id with caching on the smallest server gained more : on 1 shared vCPU, 250 connections gave 15,248 requests per second against 12,096 with 100, at twice the latency (16 ms p50 against 8 ms).

The installation¶

API Maker is installed on a fresh server with its own installer, its default answers and no reverse proxy, so nothing stands between autocannon and API Maker :

bash install.sh --default --primary --caddy=no --ssl=no --qa_uat_prod=no --package=api-maker-4.0.0-linux-x64.tar.gz

The installer sets up Docker, MongoDB 8.0.10 and Redis 7.0.5 in Docker, swap, and API Maker as a systemd service. API Maker keeps its default settings :

one worker thread per vCPU ;
the Redis of the installation for cached answers, kept two hours (redisValueExpireInSeconds 7200) ;
two sandboxes per environment of an account (its sandbox count, set to 2), each opening up to one runner per vCPU while it is busy ;
no request logs : a fresh installation has no database for them.

One setting is changed, in ~/config/.env :

am__sandbox__removeSandboxInactiveSinceMinutes=1440

By default, a sandbox nobody used for 5 minutes stops, and the next custom API starts it again. Here the two sandboxes of the account keep running while a database is measured, so every API runs with them, and their memory is counted.

PostgreSQL and MySQL¶

PostgreSQL 18 and MySQL 8.4 run from their official images. On the servers of 2 to 8 vCPUs they allow 500 connections (100 and 151 by default) ; the 1 vCPU server keeps their defaults :

docker run -d --name bench_postgres -e POSTGRES_PASSWORD=… -p 5432:5432 postgres:18 -c max_connections=500
docker run -d --name bench_mysql -e MYSQL_ROOT_PASSWORD=… -p 3306:3306 mysql:8.4 --max-connections=500

Code that runs in a sandbox opens connection pools of its own, one per runner. On 8 vCPUs, a custom API in the sandbox on PostgreSQL can hold 161 connections : 2 sandboxes × 8 runners × a pool of 10, plus the pool of API Maker itself. That is more than the 100 PostgreSQL allows by default. If your server has many vCPUs and runs code on a SQL database, raise max_connections, or lower Max pool connections of the instance.

The MongoDB data of the benchmarks lives in the MongoDB of API Maker itself (database am_bench), the way a small installation keeps everything in one MongoDB.

The data¶

Every table holds the ten rows of the benchmark dataset. One row :

{
    "_id": "60ae2488c37cd955cc144161",
    "customer_id": 2,
    "first_name": "NICK",
    "last_name": "WAHLBERG",
    "phone": 1123456789,
    "last_update": "2006-02-15T04:34:00.330Z",
    "pincode": 111111,
    "shipping_id": 2,
    "dob": "2022-10-30T03:25:03.000Z",
    "country": "India"
}

Each database (am_bench in MongoDB, PostgreSQL and MySQL) has six tables. API Maker made the schema of each from the table :

Table	Its APIs	Caching
`customers_public`	Public	Off
`customers`	Both tokens	Off
`customers_public_cached`	Public	On
`customers_cached`	Both tokens	On
`customers_pre_hook`	Both tokens, a pre hook on its get all	Off
`inserts_public`	Public : the insert benchmark writes here	Off

In SQL, customer_id is the primary key (SERIAL / AUTO_INCREMENT in inserts_public) and the dates are TIMESTAMP(3) / DATETIME(3). In PostgreSQL the URLs name the tables with their schema : public.customers_public.

The APIs¶

The settings of the APIs of a table (__COLLECTION_SETTINGS__), for each kind of table :

PublicBoth tokensCached

import * as T from 'types';
let settings: T.IInstanceApiSettingsTypes = {
    enableCaching: false,
    apiAccessType: T.EAPIAccessType.IS_PUBLIC,
};
module.exports = settings;

import * as T from 'types';
let settings: T.IInstanceApiSettingsTypes = {
    enableCaching: false,
    apiAccessType: T.EAPIAccessType.TOKEN_ACCESS,
    authProviders: ["bench_token_mongodb"],
};
module.exports = settings;

import * as T from 'types';
let settings: T.IInstanceApiSettingsTypes = {
    enableCaching: true,
    apiAccessType: T.EAPIAccessType.IS_PUBLIC,
};
module.exports = settings;

The custom APIs are TypeScript :

import * as T from 'types';
async function main(g: T.IAMGlobal) {
    return await g.sys.db.getAll({ instance: 'mongodb', database: 'am_bench', collection: 'customers_public' });
}
module.exports = main;

The custom API with both tokens is the same, with apiAccessType: TOKEN_ACCESS and the token generator of its database in authProviders, and reads customers. The custom API without a database returns { message: 'Hello from API Maker', time: 1 }.

In the server or in the sandbox¶

The code above is lean : it only reads what it is given and calls g.sys. API Maker runs lean code inside the server, with QuickJS, on the thread that answers the request : the request never leaves the server. The pre hook below is lean too. Code that needs more (a package, g.shared, the clock, a timer, try) runs in a sandbox of the account : a container with Node.js 24.

To measure both, every database also has a custom API that runs in the sandbox : the same code with one more line, which reads Date.now() and so is not lean. The custom API without a database has a sandbox twin too ({ message: 'Hello from the sandbox', time: Date.now() }).

The custom API which runs in the sandbox

import * as T from 'types';
async function main(g: T.IAMGlobal) {
    const started = Date.now();       // the time of the machine : this code runs in the sandbox of the account
    return await g.sys.db.getAll({ instance: 'mongodb', database: 'am_bench', collection: 'customers_public' });
}
module.exports = main;

The pre hook¶

The table customers_pre_hook has the settings of both tokens, and a pre hook on its get all API (SCHEMA_GET_ALL) : row scoping, the pre hook most applications write. It reads the signed-in person from x-am-user-authorization and keeps the rows of their country :

Pre hook of SCHEMA_GET_ALL on customers_pre_hook

import * as T from 'types';

async function main(g: T.IAMGlobal) {
    if (!g.req.isApiRequestFromUser) return;
    const person = g.req.auth.authAMDB;               // the person, from x-am-user-authorization
    if (!person) throw [{ code: T.EStatusCode.UNAUTHORIZED, message: 'Sign in first.' }];
    const find = g.req.query.find ? (typeof g.req.query.find === 'string' ? JSON.parse(g.req.query.find) : g.req.query.find) : {};
    g.req.query.find = { $and: [find, { country: person.country }] };
}
module.exports = main;

Every row of the dataset is in India, like the person of the token, so the answer is the same ten rows as get all, after the hook ran. Before the benchmarks, a first hook adds a warning to the answer, to prove that the hooks of the table run ; then the hook above takes its place.

The two tokens¶

Both tokens means what a real application sends : the token of an API user and the token of a user of the application.

x-am-authorization : the token of an API user, from the token API. Its groups say which APIs it may call ; here one group allows everything.
x-am-user-authorization : the token of a row of the customers table, made by a DB token generator of the database (first_name as the user name, last_name as the password). Its select puts customer_id and country in g.req.auth.authAMDB, for the pre hook : a column which is not selected is not there.

import * as T from 'types';
let dbTokenGenerator: T.IAuthTokenAMDB = {
    name: 'bench_token_mongodb',
    instance: 'mongodb',
    database: 'am_bench',
    collection: 'customers',
    usernameColumn: 'first_name',
    passwordColumn: 'last_name',
    select: 'customer_id,country',
    expiresInSeconds: 2592000,
    runOnNativeProcess: false,
};
module.exports = dbTokenGenerator;

Both tokens are asked for again before every run, and sent with every request. API Maker checks both on every request.

The load¶

For every API, on every database :

Check : one request checks the answer. Status 200 (201 for the insert), ten rows for get all, the row for get by id, the saved row for the insert. Before the first run, every API with tokens was also checked to answer 401 without its tokens, and with only one of them.
Warm-up : 5 seconds, so the cache fills, the connections to the database open, and the code of the custom API or the hook is loaded.

Measure : a pause of 3 seconds, then 15 seconds of load, no pipelining :

autocannon -c 100 -d 15 -w 4 -l "http://<server>:38246/api/schema/bench/mongodb/am_bench/customers_public"
autocannon -c 100 -d 15 -w 4 -l \
    -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" \
    "http://<server>:38246/api/schema/bench/mongodb/am_bench/customers"

Rest : 15 seconds before the next API.

All the APIs of a database run once (round 1), then again in reverse order (round 2), so a slow moment of a shared CPU does not fall on the same API twice. The number shown is the average of the two rounds, and the autocannon output shown is the round closer to it. The page of each server also shows both rounds and how far apart they are.

More connections is a separate run of get all, public and get by id with caching, public on MongoDB : 15 seconds for each number of connections (10, 50, 100, 250, 500 and 1,000).

Nothing else runs¶

Before the APIs of a database are loaded :

the other benchmark databases are stopped (the MongoDB and Redis of API Maker keep running : API Maker needs them) ;
API Maker is restarted, so every database starts with a fresh process, and its two sandboxes are started and checked ;
the swap is emptied back into memory, when it fits. On the 1 vCPU server it did not fit before MySQL : MySQL 8.4, MongoDB, Redis, API Maker and its sandboxes need more than the 961 MB of that server, so its MySQL runs started with swap in use ;
the server rests 60 seconds : this is the memory at rest of the pages.

Nothing else runs on the server : no reverse proxy, no admin panel open, no other load. The periodic jobs of Ubuntu (apt, man-db, sysstat, fstrim and the others) are stopped on both servers for the benchmarks. The job which rotates the log of API Maker, part of its installation, stays on.

Memory and CPU¶

A small script reads /proc and the cgroups of the containers once a second, on the server and on the load generator. It costs a few milliseconds of CPU a second.

Measure	What it is
API Maker	The resident memory (RSS) of the `api_maker_be` process, and its CPU in percent of one core.
Containers	MongoDB, Redis, PostgreSQL, MySQL and the sandboxes : memory as `docker stats` counts it (without the inactive file cache), and CPU.
Whole server	Memory used (total − available), swap used, pages swapped in and out.
CPU	User, system, softirq, idle and steal : the time the hypervisor gave to other servers. On a shared CPU, steal shows whether a neighbour took the CPU during a run.

Under load on the pages is the most a part held while the APIs of its database ran, leaving out the first two seconds of each run, while autocannon opens its connections. At rest is the average of the 60 seconds before, with API Maker and its two sandboxes started and no request.

What these numbers are not¶

No HTTPS, no reverse proxy. Caddy (which the installer puts in front by default) and TLS cost CPU of their own, on the same cores.
A small table. Ten rows of ten columns measure what API Maker adds to each request. Bigger answers cost more bytes, and more time in the database.
Not your server. Your CPU, your neighbours on a shared plan, your network and your data change the numbers. The method is here so you can measure your own.
Not a published version. The pages say API Maker 4.0.0 : the number of the build that was measured (commit 864bb64, 12 October 2026), before the beta releases were numbered.
Not a rule about shared and dedicated CPU. Get all reached 4,692 req/s on 2 shared vCPUs and 3,712 on 2 dedicated vCPUs, on the same CPU model. A shared vCPU can run faster while its host is quiet, the two load generators differed (an EPYC 7713 and an EPYC 7642), and each server was measured once. Read a gap of this size as the spread of one run. Choose dedicated vCPUs for a speed which does not depend on the neighbours.

Run it yourself¶

Make two servers in the same data centre. Install API Maker on the first one, and the databases you want to measure.
Load the dataset into the tables above. Add an instance for each database, then the schemas, the settings, the token generators and the custom APIs of this page.
On the second server : npm install autocannon@8, ask for the two tokens with the token API, and run the commands of The load for each API.

The results : API benchmarks.