Benchmarks · All results · 7 servers
Every API, every server : side by side.
The average requests per second of every API Maker benchmark, on 7 Linode servers from 1 to 8 vCPUs, shared and dedicated CPU, on MongoDB, PostgreSQL and MySQL. 136,821,729 requests, 0 errors. Each number links to its run, with the full autocannon output.
Servers
7
4 shared CPU · 3 dedicated CPU
APIs per server
46
15 on each of 3 databases + a custom API without one
Requests
137million
2 rounds of 15 s per API, plus the sweeps of connections
Errors
0
non-2xx, errors, timeouts
Every API on every server¶
Requests per second (the average of two rounds of 15 seconds, 100 connections) :
| API | 1 vCPU shared · 1 GB |
2 vCPU shared · 4 GB |
4 vCPU shared · 8 GB |
8 vCPU shared · 32 GB |
2 vCPU dedicated · 4 GB |
4 vCPU dedicated · 8 GB |
8 vCPU dedicated · 16 GB |
|---|---|---|---|---|---|---|---|
| Get all | 2,435 | 4,692 | 7,423 | 11,002 | 3,712 | 7,614 | 12,640 |
| Get all · both tokens | 2,439 | 4,425 | 7,021 | 11,882 | 3,564 | 7,220 | 12,106 |
| Get all with caching | 11,054 | 18,195 | 26,115 | 41,093 | 15,152 | 29,762 | 43,980 |
| Get all with caching · both tokens | 8,709 | 14,144 | 20,550 | 34,364 | 11,750 | 23,619 | 36,123 |
| Get all by stream | 2,545 | 4,548 | 6,724 | 10,646 | 3,661 | 7,342 | 11,898 |
| Get all by stream · both tokens | 2,548 | 4,297 | 6,737 | 11,025 | 3,462 | 6,933 | 11,516 |
| Custom API calling get all QuickJS | 1,909 | 3,178 | 5,178 | 8,062 | 2,591 | 5,228 | 8,893 |
| Custom API calling get all · both tokens QuickJS | 1,699 | 2,957 | 4,910 | 8,393 | 2,420 | 4,933 | 8,426 |
| Custom API in the sandbox Sandbox | 1,389 | 3,015 | 4,649 | 7,636 | 2,269 | 4,491 | 7,569 |
| Get all with a pre hook · both tokens QuickJS | 1,984 | 3,339 | 5,327 | 8,512 | 2,713 | 5,460 | 9,279 |
| Get by id | 3,100 | 5,397 | 8,076 | 12,297 | 4,323 | 8,777 | 14,224 |
| Get by id · both tokens | 3,044 | 4,947 | 7,513 | 12,654 | 4,044 | 8,182 | 13,443 |
| Get by id with caching | 12,140 | 18,970 | 26,444 | 44,564 | 16,455 | 32,588 | 47,980 |
| Get by id with caching · both tokens | 9,481 | 14,695 | 20,887 | 36,219 | 12,553 | 25,516 | 38,521 |
| Insert one row | 1,156 | 2,352 | 3,198 | 4,620 | 1,735 | 3,574 | 5,490 |
Latency in milliseconds, p50 / p99, of the same runs :
| API | 1 vCPU shared · 1 GB |
2 vCPU shared · 4 GB |
4 vCPU shared · 8 GB |
8 vCPU shared · 32 GB |
2 vCPU dedicated · 4 GB |
4 vCPU dedicated · 8 GB |
8 vCPU dedicated · 16 GB |
|---|---|---|---|---|---|---|---|
| Get all | 39 / 46 | 20 / 40 | 11 / 28 | 8 / 20 | 26 / 68 | 12 / 28 | 7 / 17 |
| Get all · both tokens | 39 / 52 | 22 / 38 | 14 / 28 | 7 / 17 | 27 / 48 | 13 / 27 | 7 / 17 |
| Get all with caching | 9 / 13 | 5 / 8 | 3 / 5 | 2 / 3 | 6 / 10 | 3 / 5 | 2 / 3 |
| Get all with caching · both tokens | 11 / 17 | 7 / 10 | 4 / 7 | 2 / 4 | 8 / 12 | 4 / 6 | 2 / 4 |
| Get all by stream | 38 / 46 | 21 / 39 | 14 / 33 | 8 / 21 | 26 / 50 | 13 / 27 | 8 / 17 |
| Get all by stream · both tokens | 39 / 54 | 23 / 39 | 14 / 29 | 8 / 18 | 29 / 48 | 13 / 26 | 8 / 18 |
| Custom API calling get all QuickJS | 53 / 69 | 31 / 50 | 19 / 34 | 12 / 23 | 38 / 62 | 18 / 32 | 11 / 20 |
| Custom API calling get all · both tokens QuickJS | 61 / 125 | 33 / 54 | 19 / 34 | 11 / 21 | 41 / 67 | 20 / 35 | 11 / 21 |
| Custom API in the sandbox Sandbox | 75 / 97 | 31 / 64 | 20 / 42 | 12 / 28 | 42 / 77 | 21 / 41 | 13 / 24 |
| Get all with a pre hook · both tokens QuickJS | 52 / 66 | 29 / 50 | 18 / 34 | 11 / 24 | 36 / 61 | 18 / 31 | 10 / 20 |
| Get by id | 31 / 37 | 18 / 33 | 11 / 26 | 7 / 18 | 22 / 42 | 11 / 22 | 6 / 14 |
| Get by id · both tokens | 33 / 44 | 20 / 35 | 12 / 23 | 7 / 15 | 24 / 41 | 12 / 22 | 7 / 14 |
| Get by id with caching | 7 / 12 | 5 / 8 | 3 / 5 | 2 / 3 | 6 / 9 | 3 / 4 | 2 / 3 |
| Get by id with caching · both tokens | 10 / 16 | 6 / 10 | 4 / 7 | 2 / 4 | 8 / 12 | 3 / 6 | 2 / 4 |
| Insert one row | 83 / 110 | 38 / 83 | 30 / 63 | 21 / 38 | 58 / 99 | 27 / 52 | 17 / 32 |
Requests per second (the average of two rounds of 15 seconds, 100 connections) :
| API | 1 vCPU shared · 1 GB |
2 vCPU shared · 4 GB |
4 vCPU shared · 8 GB |
8 vCPU shared · 32 GB |
2 vCPU dedicated · 4 GB |
4 vCPU dedicated · 8 GB |
8 vCPU dedicated · 16 GB |
|---|---|---|---|---|---|---|---|
| Get all | 3,787 | 5,577 | 8,395 | 13,054 | 4,782 | 8,912 | 12,641 |
| Get all · both tokens | 3,560 | 5,290 | 7,680 | 12,013 | 4,384 | 8,245 | 11,913 |
| Get all with caching | 11,064 | 17,836 | 26,420 | 39,947 | 14,883 | 30,205 | 43,173 |
| Get all with caching · both tokens | 8,797 | 14,083 | 20,113 | 32,966 | 11,615 | 24,179 | 35,403 |
| Get all by stream | 3,989 | 5,866 | 8,226 | 12,914 | 4,837 | 9,098 | 13,073 |
| Get all by stream · both tokens | 3,461 | 5,189 | 7,486 | 12,049 | 4,427 | 8,490 | 12,162 |
| Custom API calling get all QuickJS | 2,178 | 3,806 | 5,658 | 9,350 | 3,059 | 5,886 | 9,176 |
| Custom API calling get all · both tokens QuickJS | 2,016 | 3,461 | 5,158 | 8,433 | 2,801 | 5,442 | 8,561 |
| Custom API in the sandbox Sandbox | 1,871 | 3,423 | 5,126 | 8,682 | 2,389 | 4,673 | 7,916 |
| Get all with a pre hook · both tokens QuickJS | 2,318 | 3,884 | 6,151 | 9,668 | 3,246 | 6,280 | 9,524 |
| Get by id | 4,105 | 6,325 | 9,446 | 13,656 | 5,090 | 9,655 | 13,571 |
| Get by id · both tokens | 3,887 | 5,739 | 8,233 | 12,548 | 4,626 | 8,790 | 12,656 |
| Get by id with caching | 12,083 | 19,622 | 26,617 | 43,050 | 16,517 | 33,015 | 47,888 |
| Get by id with caching · both tokens | 9,402 | 14,989 | 21,238 | 34,322 | 12,500 | 25,875 | 38,045 |
| Insert one row | 1,818 | 3,021 | 3,969 | 5,754 | 2,335 | 4,236 | 5,874 |
Latency in milliseconds, p50 / p99, of the same runs :
| API | 1 vCPU shared · 1 GB |
2 vCPU shared · 4 GB |
4 vCPU shared · 8 GB |
8 vCPU shared · 32 GB |
2 vCPU dedicated · 4 GB |
4 vCPU dedicated · 8 GB |
8 vCPU dedicated · 16 GB |
|---|---|---|---|---|---|---|---|
| Get all | 25 / 51 | 18 / 22 | 11 / 14 | 7 / 9 | 20 / 24 | 11 / 13 | 7 / 10 |
| Get all · both tokens | 28 / 36 | 18 / 22 | 12 / 15 | 8 / 10 | 22 / 27 | 12 / 14 | 8 / 10 |
| Get all with caching | 8 / 13 | 5 / 8 | 3 / 5 | 2 / 3 | 6 / 10 | 3 / 5 | 2 / 3 |
| Get all with caching · both tokens | 11 / 17 | 7 / 12 | 4 / 8 | 3 / 4 | 8 / 12 | 4 / 6 | 2 / 4 |
| Get all by stream | 24 / 31 | 17 / 20 | 12 / 14 | 7 / 9 | 20 / 25 | 10 / 13 | 7 / 9 |
| Get all by stream · both tokens | 29 / 37 | 19 / 23 | 13 / 15 | 8 / 10 | 22 / 26 | 11 / 13 | 8 / 10 |
| Custom API calling get all QuickJS | 46 / 57 | 26 / 30 | 17 / 20 | 10 / 13 | 32 / 37 | 16 / 19 | 10 / 13 |
| Custom API calling get all · both tokens QuickJS | 48 / 57 | 28 / 32 | 19 / 22 | 11 / 14 | 35 / 41 | 18 / 21 | 11 / 14 |
| Custom API in the sandbox Sandbox | 52 / 100 | 28 / 51 | 19 / 35 | 11 / 20 | 40 / 72 | 21 / 38 | 12 / 22 |
| Get all with a pre hook · both tokens QuickJS | 42 / 53 | 25 / 31 | 16 / 18 | 10 / 12 | 30 / 36 | 15 / 18 | 10 / 13 |
| Get by id | 24 / 31 | 15 / 19 | 10 / 12 | 7 / 8 | 19 / 22 | 10 / 12 | 7 / 9 |
| Get by id · both tokens | 25 / 32 | 17 / 21 | 11 / 17 | 7 / 10 | 20 / 40 | 11 / 13 | 7 / 11 |
| Get by id with caching | 8 / 12 | 5 / 7 | 3 / 5 | 2 / 3 | 5 / 9 | 3 / 4 | 2 / 3 |
| Get by id with caching · both tokens | 10 / 16 | 6 / 10 | 4 / 7 | 2 / 4 | 7 / 12 | 3 / 5 | 2 / 4 |
| Insert one row | 54 / 64 | 33 / 38 | 24 / 28 | 17 / 19 | 42 / 49 | 23 / 28 | 16 / 19 |
Requests per second (the average of two rounds of 15 seconds, 100 connections) :
| API | 1 vCPU shared · 1 GB |
2 vCPU shared · 4 GB |
4 vCPU shared · 8 GB |
8 vCPU shared · 32 GB |
2 vCPU dedicated · 4 GB |
4 vCPU dedicated · 8 GB |
8 vCPU dedicated · 16 GB |
|---|---|---|---|---|---|---|---|
| Get all | 4,623 | 6,660 | 9,375 | 13,816 | 5,418 | 10,000 | 13,748 |
| Get all · both tokens | 4,126 | 5,902 | 8,338 | 12,645 | 4,939 | 9,236 | 12,712 |
| Get all with caching | 11,245 | 18,227 | 25,630 | 40,335 | 15,123 | 30,217 | 44,027 |
| Get all with caching · both tokens | 8,870 | 13,394 | 20,763 | 33,000 | 11,626 | 24,025 | 35,915 |
| Get all by stream | 4,213 | 5,815 | 8,355 | 12,489 | 5,034 | 9,458 | 13,095 |
| Get all by stream · both tokens | 3,662 | 5,317 | 7,469 | 11,587 | 4,518 | 8,733 | 12,340 |
| Custom API calling get all QuickJS | 2,419 | 4,090 | 6,016 | 9,698 | 3,357 | 6,513 | 9,820 |
| Custom API calling get all · both tokens QuickJS | 2,231 | 3,666 | 5,536 | 8,962 | 3,058 | 6,032 | 9,129 |
| Custom API in the sandbox Sandbox | 2,170 | 3,834 | 5,486 | 9,181 | 2,714 | 5,174 | 8,657 |
| Get all with a pre hook · both tokens QuickJS | 2,688 | 4,389 | 6,768 | 10,192 | 3,524 | 6,820 | 10,204 |
| Get by id | 5,090 | 7,155 | 9,893 | 14,402 | 5,874 | 10,654 | 14,510 |
| Get by id · both tokens | 4,526 | 6,551 | 8,892 | 13,222 | 5,304 | 9,758 | 13,517 |
| Get by id with caching | 12,193 | 19,858 | 27,185 | 43,160 | 16,379 | 32,897 | 48,818 |
| Get by id with caching · both tokens | 9,217 | 15,385 | 22,684 | 34,667 | 12,515 | 25,837 | 38,544 |
| Insert one row | 1,651 | 2,371 | 3,181 | 3,532 | 2,081 | 3,462 | 3,542 |
Latency in milliseconds, p50 / p99, of the same runs :
| API | 1 vCPU shared · 1 GB |
2 vCPU shared · 4 GB |
4 vCPU shared · 8 GB |
8 vCPU shared · 32 GB |
2 vCPU dedicated · 4 GB |
4 vCPU dedicated · 8 GB |
8 vCPU dedicated · 16 GB |
|---|---|---|---|---|---|---|---|
| Get all | 21 / 27 | 14 / 17 | 10 / 12 | 6 / 8 | 18 / 22 | 9 / 11 | 7 / 9 |
| Get all · both tokens | 23 / 31 | 16 / 20 | 12 / 17 | 7 / 10 | 20 / 24 | 10 / 12 | 7 / 10 |
| Get all with caching | 8 / 13 | 5 / 8 | 3 / 6 | 2 / 3 | 6 / 9 | 3 / 5 | 2 / 3 |
| Get all with caching · both tokens | 11 / 17 | 7 / 10 | 4 / 7 | 3 / 4 | 8 / 13 | 4 / 6 | 2 / 4 |
| Get all by stream | 22 / 29 | 17 / 21 | 12 / 14 | 8 / 10 | 19 / 23 | 10 / 12 | 7 / 9 |
| Get all by stream · both tokens | 26 / 34 | 18 / 22 | 13 / 15 | 8 / 10 | 21 / 26 | 11 / 13 | 8 / 10 |
| Custom API calling get all QuickJS | 41 / 51 | 24 / 28 | 16 / 19 | 10 / 12 | 30 / 35 | 15 / 17 | 10 / 12 |
| Custom API calling get all · both tokens QuickJS | 44 / 53 | 27 / 31 | 17 / 20 | 11 / 14 | 32 / 38 | 16 / 19 | 10 / 13 |
| Custom API in the sandbox Sandbox | 43 / 70 | 25 / 46 | 17 / 33 | 10 / 19 | 35 / 64 | 19 / 35 | 11 / 20 |
| Get all with a pre hook · both tokens QuickJS | 37 / 47 | 22 / 27 | 14 / 17 | 9 / 13 | 27 / 59 | 14 / 17 | 9 / 13 |
| Get by id | 19 / 25 | 13 / 16 | 9 / 11 | 6 / 8 | 16 / 20 | 9 / 11 | 6 / 8 |
| Get by id · both tokens | 21 / 28 | 15 / 19 | 10 / 13 | 7 / 9 | 18 / 22 | 10 / 12 | 7 / 9 |
| Get by id with caching | 8 / 12 | 4 / 7 | 3 / 5 | 2 / 3 | 5 / 9 | 2 / 4 | 2 / 3 |
| Get by id with caching · both tokens | 10 / 16 | 6 / 9 | 4 / 6 | 2 / 4 | 7 / 12 | 3 / 5 | 2 / 4 |
| Insert one row | 59 / 71 | 41 / 49 | 31 / 35 | 28 / 35 | 48 / 54 | 29 / 33 | 28 / 38 |
The custom API without a database, requests per second : in the server (lean code) 1 vCPU shared 8,096 · 2 vCPU shared 14,731 · 4 vCPU shared 26,046 · 8 vCPU shared 47,967 · 2 vCPU dedicated 10,790 · 4 vCPU dedicated 21,532 · 8 vCPU dedicated 39,192 ; in the sandbox 1 vCPU shared 9,258 · 2 vCPU shared 12,669 · 4 vCPU shared 17,702 · 8 vCPU shared 27,302 · 2 vCPU dedicated 9,953 · 4 vCPU dedicated 18,562 · 8 vCPU dedicated 27,920.
More vCPUs¶
API Maker is one process which uses every vCPU of its server : the same APIs, from the smallest server to the largest.
1 vCPUshared 1 GB $5
2 vCPUsshared 4 GB $24 · dedicated 4 GB $36
4 vCPUsshared 8 GB $48 · dedicated 8 GB $72
8 vCPUsshared 32 GB $192 · dedicated 16 GB $144
1 vCPUshared 1 GB $5
2 vCPUsshared 4 GB $24 · dedicated 4 GB $36
4 vCPUsshared 8 GB $48 · dedicated 8 GB $72
8 vCPUsshared 32 GB $192 · dedicated 16 GB $144
1 vCPUshared 1 GB $5
2 vCPUsshared 4 GB $24 · dedicated 4 GB $36
4 vCPUsshared 8 GB $48 · dedicated 8 GB $72
8 vCPUsshared 32 GB $192 · dedicated 16 GB $144
| API · database | 1 vCPU shared |
2 vCPU shared |
4 vCPU shared |
8 vCPU shared |
2 vCPU dedicated |
4 vCPU dedicated |
8 vCPU dedicated |
|---|---|---|---|---|---|---|---|
| Get all · MongoDB | 2,435 | 4,692 × 1.9 | 7,423 × 3.0 | 11,002 × 4.5 | 3,712 × 1.5 | 7,614 × 3.1 | 12,640 × 5.2 |
| Get all · PostgreSQL | 3,787 | 5,577 × 1.5 | 8,395 × 2.2 | 13,054 × 3.4 | 4,782 × 1.3 | 8,912 × 2.4 | 12,641 × 3.3 |
| Get all · MySQL | 4,623 | 6,660 × 1.4 | 9,375 × 2.0 | 13,816 × 3.0 | 5,418 × 1.2 | 10,000 × 2.2 | 13,748 × 3.0 |
| Get by id with caching · MongoDB | 12,140 | 18,970 × 1.6 | 26,444 × 2.2 | 44,564 × 3.7 | 16,455 × 1.4 | 32,588 × 2.7 | 47,980 × 4.0 |
| Get by id with caching · PostgreSQL | 12,083 | 19,622 × 1.6 | 26,617 × 2.2 | 43,050 × 3.6 | 16,517 × 1.4 | 33,015 × 2.7 | 47,888 × 4.0 |
| Get by id with caching · MySQL | 12,193 | 19,858 × 1.6 | 27,185 × 2.2 | 43,160 × 3.5 | 16,379 × 1.3 | 32,897 × 2.7 | 48,818 × 4.0 |
| Custom API calling get all · MongoDB QuickJS | 1,909 | 3,178 × 1.7 | 5,178 × 2.7 | 8,062 × 4.2 | 2,591 × 1.4 | 5,228 × 2.7 | 8,893 × 4.7 |
| Custom API in the sandbox · MongoDB Sandbox | 1,389 | 3,015 × 2.2 | 4,649 × 3.3 | 7,636 × 5.5 | 2,269 × 1.6 | 4,491 × 3.2 | 7,569 × 5.5 |
| Get all with a pre hook · MongoDB QuickJS | 1,984 | 3,339 × 1.7 | 5,327 × 2.7 | 8,512 × 4.3 | 2,713 × 1.4 | 5,460 × 2.8 | 9,279 × 4.7 |
× n : the requests per second against the 1 vCPU server (1 GB of memory : it also used its swap). From 1 to 8 shared vCPUs : get all ×4.5 on MongoDB, ×3.4 on PostgreSQL and ×3.0 on MySQL, get by id with caching ×3.7, the custom API ×4.2. On every server the database, Redis and the sandboxes run beside API Maker and take their share of the same vCPUs.
Memory¶
The API Maker process at rest (with its two sandboxes started, before any request) and the most it held under load, the two sandboxes at rest, and the whole server : at rest with MongoDB, and the most it used while any database was loaded.
| Server | CPU | API Maker at rest | API Maker under load | Sandboxes at rest | Server at rest | Server under load |
|---|---|---|---|---|---|---|
| 1 vCPU · 1 GB | shared | 84 MB | 111 MB | 50 MB | 686 MB | 807 MB of 961 MB |
| 2 vCPU · 4 GB | shared | 101 MB | 125 MB | 50 MB | 803 MB | 1658 MB of 3,915 MB |
| 4 vCPU · 8 GB | shared | 109 MB | 142 MB | 55 MB | 872 MB | 2048 MB of 7,941 MB |
| 8 vCPU · 32 GB | shared | 112 MB | 154 MB | 55 MB | 1102 MB | 2710 MB of 32,095 MB |
| 2 vCPU · 4 GB | dedicated | 99 MB | 127 MB | 50 MB | 773 MB | 1600 MB of 3,915 MB |
| 4 vCPU · 8 GB | dedicated | 104 MB | 140 MB | 55 MB | 872 MB | 2024 MB of 7,941 MB |
| 8 vCPU · 16 GB | dedicated | 111 MB | 176 MB | 55 MB | 946 MB | 2602 MB of 15,991 MB |
WebSockets¶
API Maker serves WebSockets on the port of its APIs. The same servers were filled with WebSocket clients, each with both tokens and a subscription : 26,484 clients connected at once on 1 vCPU and 1 GB, 3,662,564 on 8 vCPUs and 32 GB, 8.5 to 8.8 KB per client.
WWebSocket benchmarksHow many connections one server holds, the memory when it is full, new clients and notifications a second.
NScale to N usersOne big server or several small ones, the picture of N users, and the rule to count the servers.
ΣAll WebSocket resultsEvery server side by side : clients, memory per client, delivery times.
Questions¶
Does API Maker scale with more vCPUs ?¶
Yes : API Maker is one process which uses every vCPU of its server. Get all on MongoDB went from 2,435 requests per second on 1 vCPU to 4,692 on 2 shared, 7,423 on 4 shared, 11,002 on 8 shared, 3,712 on 2 dedicated, 7,614 on 4 dedicated, 12,640 on 8 dedicated. The database runs on the same server, so its own work grows with the load too.