# How API Maker is benchmarked

> How API Maker is benchmarked - Linode servers of 1 to 8 vCPUs, autocannon with 100 connections for 15 s, two rounds, the dataset, every API setting and the commands to run it yourself.

Source: https://docs.apimaker.dev/v4/docs/benchmarks/methodology.html

The benchmarks measure what you would get yourself : API Maker set up by its own installer on a plain cloud server, the databases on the same machine, and the load sent from a second server. This page shows every step, so you can check any number, or run the benchmarks again.

| | |
|---|---|
| Servers | 7 [Linode](https://www.linode.com) servers in Mumbai, from 1 to 8 vCPUs, shared and dedicated CPU, all on the same CPU (AMD EPYC 7642). Each one runs API Maker and its databases. |
| Load | autocannon 8 on a second, dedicated server in the same data centre : 100 connections, 5 seconds of warm-up, then 15 seconds measured. |
| Rounds | Every API runs twice, the second time in reverse order. The pages show the average of the two. |
| Data | The [benchmark dataset](https://apimaker.dev/json/benchmark_dataset.json) : 10 customers with 10 columns each, in MongoDB, PostgreSQL and MySQL. |
| Installation | The API Maker installer with its default answers and no reverse proxy. One setting changed : the sandboxes keep running. |
| Checks | Every API answers once before its load : the status and the rows are checked, and an API with tokens refuses a request with none, or only one. Every error, non-2xx answer and timeout is counted. |

## The servers

Every benchmark uses two servers in the same data centre :

| | Server under test | Load generator |
|---|---|---|
| Runs | API Maker, its MongoDB and Redis, PostgreSQL, MySQL, the sandboxes | autocannon, nothing else |
| Plan | Shared or dedicated CPU, as named on each page | Dedicated CPU, four times the vCPUs of the server under test |
| System | Ubuntu 24.04 LTS | Ubuntu 24.04 LTS, Node.js 24.21.0, autocannon 8.0.0 |
| Network | Private network of the data centre | The same network, 0.5 ms away (ping) |

The load comes from its own machine, so autocannon never takes CPU away from API Maker. That machine has dedicated CPUs, so its speed does not depend on its neighbours, and it is big enough never to be the limit : it was never more than 29 % busy during a run (42 % for the 1 vCPU server). Every page shows how busy it was.

| Server under test | Load generator | autocannon threads |
|---|---|--:|
| 1 vCPU | Dedicated 8 GB (g6-dedicated-4) : 4 vCPUs | 4 (25 connections each) |
| 2 vCPU, shared and dedicated | Dedicated 16 GB (g6-dedicated-8) : 8 vCPUs | 5 (20 connections each) |
| 4 vCPU, shared and dedicated | Dedicated 32 GB (g6-dedicated-16) : 16 vCPUs | 10 (10 connections each) |
| 8 vCPU, shared and dedicated | Dedicated 64 GB (g6-dedicated-32) : 32 vCPUs | 20 (5 connections each) |

### Enough autocannon threads

One autocannon thread cannot keep up with the fastest APIs. On the 1 vCPU server, *get by id with caching* gave about 10,400 requests per second with one thread, and about 12,000 with four, at the same latency. So autocannon runs several threads, which share the 100 connections.

To be sure the threads were not the limit, the same API was loaded again after each run, with more threads. It gave the same requests per second :

| Server | Threads of the run : req/s | More threads : req/s |
|---|--:|--:|
| [2 vCPU, shared](https://docs.apimaker.dev/v4/docs/benchmarks/linode-shared-2-vcpu-4gb.html) | 5 : 19,709 and 19,524 | 10 : 19,766 and 20,058 |
| [2 vCPU, dedicated](https://docs.apimaker.dev/v4/docs/benchmarks/linode-dedicated-2-vcpu-4gb.html) | 5 : 16,442 and 16,466 | 10 : 16,176 and 16,412 |
| [4 vCPU, shared](https://docs.apimaker.dev/v4/docs/benchmarks/linode-shared-4-vcpu-8gb.html) | 10 : 26,932 and 27,201 | 20 : 27,304 and 26,271 |
| [4 vCPU, dedicated](https://docs.apimaker.dev/v4/docs/benchmarks/linode-dedicated-4-vcpu-8gb.html) | 10 : 32,748 and 32,841 | 20 : 32,998 and 32,970 |
| [8 vCPU, shared](https://docs.apimaker.dev/v4/docs/benchmarks/linode-shared-8-vcpu-32gb.html) | 20 : 41,740 and 41,994 | 25 : 42,075 and 42,190 |
| [8 vCPU, dedicated](https://docs.apimaker.dev/v4/docs/benchmarks/linode-dedicated-8-vcpu-16gb.html) | 20 : 48,650 and 48,364 | 25 : 47,617 and 47,700 |

### The same CPU, at the same time

A Linode server runs on whatever CPU its host has. All seven servers under test run on the **AMD EPYC 7642**, so every size and both plans are compared on the same CPU. The load generators run on AMD EPYC 7642 or 7713 ; each page names its own.

All seven servers were measured at the same time, each with its own load generator. Linode placement groups (strict *anti-affinity*) kept the three shared CPU servers of 2, 4 and 8 vCPUs on three different hosts, and the three dedicated servers too. Every page shows the steal time of each run : the CPU the hypervisor gave to other servers.

### Why 100 connections

The fastest server (8 dedicated vCPUs) was loaded with 100, 200 and 400 connections, 20 seconds each :

| API | 100 connections | 200 | 400 | Server busy |
|---|--:|--:|--:|--:|
| Get by id with caching, public, MongoDB | 45,742 | 46,061 | 44,016 | 87 to 90 % |
| Get all, public, MongoDB | 11,059 | 11,114 | 11,049 | 95 % |
| Custom API with both tokens, PostgreSQL | 5,395 | 5,962 | 6,128 | 99 to 100 % |

At 100 connections the server already works at full speed. More connections mostly add waiting : the latency doubles at each step, for few more requests. The *more connections* runs of every page, from 10 to 1,000, say the same for *get all* : above 100 connections it gained 2.0 % at most. Only *get by id with caching* on the smallest server gained more : on 1 shared vCPU, 250 connections gave 15,248 requests per second against 12,096 with 100, at twice the latency (16 ms p50 against 8 ms).

## The installation

API Maker is installed on a fresh server with its own [installer](https://docs.apimaker.dev/v4/docs/getting-started/install-on-server.html), its default answers and no reverse proxy, so nothing stands between autocannon and API Maker :

```sh
bash install.sh --default --primary --caddy=no --ssl=no --qa_uat_prod=no --package=api-maker-4.0.0-linux-x64.tar.gz
```

The installer sets up Docker, MongoDB 8.0.10 and Redis 7.0.5 in Docker, swap, and API Maker as a systemd service. API Maker keeps its default settings :

- one worker thread per vCPU ;
- the Redis of the installation for cached answers, kept two hours (`redisValueExpireInSeconds` 7200) ;
- two sandboxes per environment of an account (its [sandbox count](https://docs.apimaker.dev/v4/docs/settings/sandboxSettings.html#sandboxes-on-the-servers), set to 2), each opening up to one runner per vCPU while it is busy ;
- no request logs : a fresh installation has no database for them.

One setting is changed, in `~/config/.env` :

```sh
am__sandbox__removeSandboxInactiveSinceMinutes=1440
```

By default, a sandbox nobody used for 5 minutes stops, and the next custom API starts it again. Here the two sandboxes of the account keep running while a database is measured, so every API runs with them, and their memory is counted.

### PostgreSQL and MySQL

PostgreSQL 18 and MySQL 8.4 run from their official images. On the servers of 2 to 8 vCPUs they allow 500 connections (100 and 151 by default) ; the 1 vCPU server keeps their defaults :

```sh
docker run -d --name bench_postgres -e POSTGRES_PASSWORD=… -p 5432:5432 postgres:18 -c max_connections=500
docker run -d --name bench_mysql -e MYSQL_ROOT_PASSWORD=… -p 3306:3306 mysql:8.4 --max-connections=500
```

Code that runs in a sandbox opens connection pools of its own, one per runner. On 8 vCPUs, a custom API in the sandbox on PostgreSQL can hold **161 connections** : 2 sandboxes × 8 runners × a pool of 10, plus the pool of API Maker itself. That is more than the 100 PostgreSQL allows by default. If your server has many vCPUs and runs code on a SQL database, raise `max_connections`, or lower *Max pool connections* of the instance.

The MongoDB data of the benchmarks lives in the MongoDB of API Maker itself (database `am_bench`), the way a small installation keeps everything in one MongoDB.

## The data

Every table holds the ten rows of the benchmark dataset. One row :

```json
{
    "_id": "60ae2488c37cd955cc144161",
    "customer_id": 2,
    "first_name": "NICK",
    "last_name": "WAHLBERG",
    "phone": 1123456789,
    "last_update": "2006-02-15T04:34:00.330Z",
    "pincode": 111111,
    "shipping_id": 2,
    "dob": "2022-10-30T03:25:03.000Z",
    "country": "India"
}
```

Each database (`am_bench` in MongoDB, PostgreSQL and MySQL) has six tables. API Maker made the schema of each from the table :

| Table | Its APIs | Caching |
|---|---|---|
| `customers_public` | Public | Off |
| `customers` | Both tokens | Off |
| `customers_public_cached` | Public | On |
| `customers_cached` | Both tokens | On |
| `customers_pre_hook` | Both tokens, a pre hook on its get all | Off |
| `inserts_public` | Public : the insert benchmark writes here | Off |

In SQL, `customer_id` is the primary key (`SERIAL` / `AUTO_INCREMENT` in `inserts_public`) and the dates are `TIMESTAMP(3)` / `DATETIME(3)`. In PostgreSQL the URLs name the tables with their schema : `public.customers_public`.

## The APIs

The settings of the APIs of a table (`__COLLECTION_SETTINGS__`), for each kind of table :

**Public**

```typescript
import * as T from 'types';
let settings: T.IInstanceApiSettingsTypes = {
    enableCaching: false,
    apiAccessType: T.EAPIAccessType.IS_PUBLIC,
};
module.exports = settings;
```

**Both tokens**

```typescript
import * as T from 'types';
let settings: T.IInstanceApiSettingsTypes = {
    enableCaching: false,
    apiAccessType: T.EAPIAccessType.TOKEN_ACCESS,
    authProviders: ["bench_token_mongodb"],
};
module.exports = settings;
```

**Cached**

```typescript
import * as T from 'types';
let settings: T.IInstanceApiSettingsTypes = {
    enableCaching: true,
    apiAccessType: T.EAPIAccessType.IS_PUBLIC,
};
module.exports = settings;
```

The custom APIs are TypeScript :

```typescript
import * as T from 'types';
async function main(g: T.IAMGlobal) {
    return await g.sys.db.getAll({ instance: 'mongodb', database: 'am_bench', collection: 'customers_public' });
}
module.exports = main;
```

The custom API with both tokens is the same, with `apiAccessType: TOKEN_ACCESS` and the token generator of its database in `authProviders`, and reads `customers`. The custom API without a database returns `{ message: 'Hello from API Maker', time: 1 }`.

### In the server or in the sandbox

The code above is **lean** : it only reads what it is given and calls `g.sys`. API Maker runs lean code inside the server, with QuickJS, on the thread that answers the request : the request never leaves the server. The pre hook below is lean too. Code that needs more (a package, `g.shared`, the clock, a timer, `try`) runs in a sandbox of the account : a container with Node.js 24.

To measure both, every database also has a custom API that runs in the sandbox : the same code with one more line, which reads `Date.now()` and so is not lean. The custom API without a database has a sandbox twin too (`{ message: 'Hello from the sandbox', time: Date.now() }`).

**The custom API which runs in the sandbox**

```typescript
import * as T from 'types';
async function main(g: T.IAMGlobal) {
    const started = Date.now();       // the time of the machine : this code runs in the sandbox of the account
    return await g.sys.db.getAll({ instance: 'mongodb', database: 'am_bench', collection: 'customers_public' });
}
module.exports = main;
```

### The pre hook

The table `customers_pre_hook` has the settings of *both tokens*, and a pre hook on its get all API (`SCHEMA_GET_ALL`) : row scoping, the pre hook most applications write. It reads the signed-in person from `x-am-user-authorization` and keeps the rows of their country :

**Pre hook of SCHEMA_GET_ALL on customers_pre_hook**

```typescript
import * as T from 'types';

async function main(g: T.IAMGlobal) {
    if (!g.req.isApiRequestFromUser) return;
    const person = g.req.auth.authAMDB;               // the person, from x-am-user-authorization
    if (!person) throw [{ code: T.EStatusCode.UNAUTHORIZED, message: 'Sign in first.' }];
    const find = g.req.query.find ? (typeof g.req.query.find === 'string' ? JSON.parse(g.req.query.find) : g.req.query.find) : {};
    g.req.query.find = { $and: [find, { country: person.country }] };
}
module.exports = main;
```

Every row of the dataset is in India, like the person of the token, so the answer is the same ten rows as *get all*, after the hook ran. Before the benchmarks, a first hook adds a warning to the answer, to prove that the hooks of the table run ; then the hook above takes its place.

### The two tokens

*Both tokens* means what a real application sends : the token of an API user and the token of a user of the application.

- **`x-am-authorization`** : the token of an [API user](https://docs.apimaker.dev/v4/docs/apis-security/api-user-permission.html), from the [token API](https://docs.apimaker.dev/v4/docs/apis-all/system-apis/system-generated-token-api.html). Its groups say which APIs it may call ; here one group allows everything.
- **`x-am-user-authorization`** : the token of a row of the `customers` table, made by a [DB token generator](https://docs.apimaker.dev/v4/docs/authorization/AMDB.html) of the database (`first_name` as the user name, `last_name` as the password). Its `select` puts `customer_id` and `country` in `g.req.auth.authAMDB`, for the pre hook : a column which is not selected is not there.

```typescript
import * as T from 'types';
let dbTokenGenerator: T.IAuthTokenAMDB = {
    name: 'bench_token_mongodb',
    instance: 'mongodb',
    database: 'am_bench',
    collection: 'customers',
    usernameColumn: 'first_name',
    passwordColumn: 'last_name',
    select: 'customer_id,country',
    expiresInSeconds: 2592000,
    runOnNativeProcess: false,
};
module.exports = dbTokenGenerator;
```

Both tokens are asked for again before every run, and sent with every request. API Maker checks both on every request.

## The load

For every API, on every database :

1. **Check** : one request checks the answer. Status 200 (201 for the insert), ten rows for *get all*, the row for *get by id*, the saved row for the insert. Before the first run, every API with tokens was also checked to answer 401 without its tokens, and with only one of them.
2. **Warm-up** : 5 seconds, so the cache fills, the connections to the database open, and the code of the custom API or the hook is loaded.
3. **Measure** : a pause of 3 seconds, then **15 seconds** of load, no pipelining :

    ```sh
    autocannon -c 100 -d 15 -w 4 -l "http://<server>:38246/api/schema/bench/mongodb/am_bench/customers_public"
    autocannon -c 100 -d 15 -w 4 -l \
        -H "x-am-authorization: <token>" -H "x-am-user-authorization: <token>" \
        "http://<server>:38246/api/schema/bench/mongodb/am_bench/customers"
    ```

4. **Rest** : 15 seconds before the next API.

All the APIs of a database run once (round 1), then again in reverse order (round 2), so a slow moment of a shared CPU does not fall on the same API twice. **The number shown is the average of the two rounds**, and the autocannon output shown is the round closer to it. The page of each server also shows both rounds and how far apart they are.

*More connections* is a separate run of *get all, public* and *get by id with caching, public* on MongoDB : 15 seconds for each number of connections (10, 50, 100, 250, 500 and 1,000).

## Nothing else runs

Before the APIs of a database are loaded :

- the other benchmark databases are stopped (the MongoDB and Redis of API Maker keep running : API Maker needs them) ;
- API Maker is restarted, so every database starts with a fresh process, and its two sandboxes are started and checked ;
- the swap is emptied back into memory, when it fits. On the 1 vCPU server it did not fit before MySQL : MySQL 8.4, MongoDB, Redis, API Maker and its sandboxes need more than the 961 MB of that server, so its MySQL runs started with swap in use ;
- the server rests 60 seconds : this is the memory *at rest* of the pages.

Nothing else runs on the server : no reverse proxy, no admin panel open, no other load. The periodic jobs of Ubuntu (apt, man-db, sysstat, fstrim and the others) are stopped on both servers for the benchmarks. The job which rotates the log of API Maker, part of its installation, stays on.

## Memory and CPU

A small script reads `/proc` and the cgroups of the containers once a second, on the server and on the load generator. It costs a few milliseconds of CPU a second.

| Measure | What it is |
|---|---|
| API Maker | The resident memory (RSS) of the `api_maker_be` process, and its CPU in percent of one core. |
| Containers | MongoDB, Redis, PostgreSQL, MySQL and the sandboxes : memory as `docker stats` counts it (without the inactive file cache), and CPU. |
| Whole server | Memory used (total − available), swap used, pages swapped in and out. |
| CPU | User, system, softirq, idle and **steal** : the time the hypervisor gave to other servers. On a shared CPU, steal shows whether a neighbour took the CPU during a run. |

*Under load* on the pages is the most a part held while the APIs of its database ran, leaving out the first two seconds of each run, while autocannon opens its connections. *At rest* is the average of the 60 seconds before, with API Maker and its two sandboxes started and no request.

## What these numbers are not

- **No HTTPS, no reverse proxy.** Caddy (which the installer puts in front by default) and TLS cost CPU of their own, on the same cores.
- **A small table.** Ten rows of ten columns measure what API Maker adds to each request. Bigger answers cost more bytes, and more time in the database.
- **Not your server.** Your CPU, your neighbours on a shared plan, your network and your data change the numbers. The method is here so you can measure your own.
- **Not a published version.** The pages say API Maker 4.0.0 : the number of the build that was measured (commit `864bb64`, 12 October 2026), before the beta releases were numbered.
- **Not a rule about shared and dedicated CPU.** Get all reached 4,692 req/s on 2 shared vCPUs and 3,712 on 2 dedicated vCPUs, on the same CPU model. A shared vCPU can run faster while its host is quiet, the two load generators differed (an EPYC 7713 and an EPYC 7642), and each server was measured once. Read a gap of this size as the spread of one run. Choose dedicated vCPUs for a speed which does not depend on the neighbours.

## Run it yourself

1. Make two servers in the same data centre. Install API Maker on the first one, and the databases you want to measure.
2. Load the [dataset](https://apimaker.dev/json/benchmark_dataset.json) into the tables above. Add an [instance](https://docs.apimaker.dev/v4/docs/apis-all/overview.html#instances) for each database, then the schemas, the settings, the token generators and the custom APIs of this page.
3. On the second server : `npm install autocannon@8`, ask for the two tokens with the [token API](https://docs.apimaker.dev/v4/docs/apis-all/system-apis/system-generated-token-api.html), and run the commands of [The load](https://docs.apimaker.dev/v4/docs/benchmarks/methodology.html#the-load) for each API.

The results : [API benchmarks](https://docs.apimaker.dev/v4/docs/benchmarks/index.html).