How fast is API Maker ? Measured, not promised.
API Maker installed by its own installer on 7 Linode servers, from 1 to 8 vCPUs, shared and dedicated CPU, its databases on the same machine. Every API loaded for 15 seconds by autocannon from a second server, two rounds each, on MongoDB, PostgreSQL and MySQL : 136,821,729 requests, 0 errors. Every page shows the full autocannon output and what each part of the server held in memory.
The servers¶
Shared CPU¶
Dedicated CPU¶
Results at a glance¶
A whole table of ten rows, read from the database on every request.
One row, answered from Redis.
| Server | CPU · price / month | Get all, MongoDB | Get all, PostgreSQL | Get all, MySQL | Get by id, cached | Custom API | API Maker at rest |
|---|---|---|---|---|---|---|---|
| 1 vCPU · 1 GB | Shared · $5 | 2,435 | 3,787 | 4,623 | 12,140 | 1,909 | 84 MB |
| 2 vCPU · 4 GB | Shared · $24 | 4,692 | 5,577 | 6,660 | 18,970 | 3,178 | 101 MB |
| 4 vCPU · 8 GB | Shared · $48 | 7,423 | 8,395 | 9,375 | 26,444 | 5,178 | 109 MB |
| 8 vCPU · 32 GB | Shared · $192 | 11,002 | 13,054 | 13,816 | 44,564 | 8,062 | 112 MB |
| 2 vCPU · 4 GB | Dedicated · $36 | 3,712 | 4,782 | 5,418 | 16,455 | 2,591 | 99 MB |
| 4 vCPU · 8 GB | Dedicated · $72 | 7,614 | 8,912 | 10,000 | 32,588 | 5,228 | 104 MB |
| 8 vCPU · 16 GB | Dedicated · $144 | 12,640 | 12,641 | 13,748 | 47,980 | 8,893 | 111 MB |
Requests per second : the average of two rounds of 15 seconds with 100 connections, public APIs ; get by id cached and the custom API on MongoDB. All results puts every API of every server side by side, with the latencies and the memory. The page of each server gives every API public and with both tokens, on the three databases, with the full autocannon output and the memory and CPU of every part.
WebSockets¶
API Maker serves WebSockets on the port of its APIs. The same servers were filled with WebSocket clients, each with both tokens and a subscription : 26,484 clients connected at once on 1 vCPU and 1 GB, 3,662,564 on 8 vCPUs and 32 GB, 8.5 to 8.8 KB per client.
What is measured¶
Every API is measured on MongoDB, PostgreSQL and MySQL, public and with both tokens (x-am-authorization + x-am-user-authorization, checked on every request) :
- Get all : The schema API of a table answers all its rows.
- Get all with caching : The same request on a table whose API settings say
enableCaching: true. - Get all by stream :
GET …/<table>/stream. - Custom API calling get all : A custom API written in TypeScript,
return await g.sys.db.getAll({ instance, database, collection }). - Custom API in the sandbox : The same custom API, with one line more.
- Get all with a pre hook : Get all, with both tokens, on a table whose get all API has a pre hook written in TypeScript.
- Get by id : One row by its primary key.
- Get by id with caching : One row by its primary key, on a table with
enableCaching: true. - Insert one row :
POST …/<table>/save-single-or-multiplewith one row of 9 columns in the body. - Custom API without a database : what running code costs, in the server (lean code) and in the sandbox.
- More connections : from 10 to 1,000 clients at once.
The data is the API Maker benchmark dataset (https://apimaker.dev/json/benchmark_dataset.json) : ten rows of customers, ten columns each, in every table.
How it is measured¶
- A real installation API Maker is installed by its installer with its default answers, MongoDB and Redis in Docker on the same server, PostgreSQL and MySQL from their official images.
- Only what is needed runs While a database is measured, the other ones are stopped. API Maker is restarted, its two sandboxes started and the swap emptied before each database.
- A second server makes the load autocannon runs on a dedicated server of the same data centre, on the private network, sized to the server under test and sampled too : it never comes near its limit.
- Every answer is checked Before its load, the answer of each API is checked (status, ten rows, the row), and the APIs with tokens are checked to refuse a request without them. Errors, non-2xx answers and timeouts are counted : 0 in 136,821,729 requests.
- Two rounds, the average 5 seconds of warm-up, then 15 seconds measured, two times, the second time in the reverse order. The page shows the average and the autocannon output of the round closer to it.
- Memory and CPU every second The server samples its CPU (with steal time), its memory, the API Maker process and every container while the load runs.
The whole method, with the settings of every API and the commands to run it yourself : How API Maker is benchmarked.
Questions¶
How many requests per second can API Maker serve ?¶
On the servers measured, with MongoDB, Redis and the database under test on the same machine, a whole table of ten rows (get all, public, MongoDB) was answered 2,435 times a second on 1 shared vCPU, 4,692 times a second on 2 shared vCPUs, 7,423 times a second on 4 shared vCPUs, 11,002 times a second on 8 shared vCPUs, 3,712 times a second on 2 dedicated vCPUs, 7,614 times a second on 4 dedicated vCPUs, 12,640 times a second on 8 dedicated vCPUs. A cached row reached 47,980 requests per second on 8 dedicated vCPUs.
Is API Maker fast on a small server ?¶
On the smallest Linode server (1 vCPU, 1 GB, $5 a month), API Maker answered 2,435 requests per second for a whole table on MongoDB (PostgreSQL 3,787, MySQL 4,623) and 12,140 for a cached row.
How much memory does API Maker need ?¶
The API Maker process held 84 to 112 MB at rest on these servers, with its two sandboxes running beside it. Most of the memory of a server goes to the databases and to the sandboxes which run your code : each server page shows every part, at rest and under load.
Why is the database on the same server ?¶
It is the cheapest way to run API Maker, and the hardest test : API Maker, its sandboxes and the database share the same vCPUs, so every number counts the work of all of them.
How are the benchmarks made ?¶
API Maker is installed by its own installer on a fresh server, the load comes from autocannon on a second server of the same data centre, every API runs two rounds of 15 seconds and the average is shown with the full autocannon output of a round.