# Get All by Stream API

> Stream the rows of a table with the get all by stream API of API Maker - the same filters as get all, sent as they are read, for exports and large result sets without holding everything in memory.

Source: https://docs.apimaker.dev/v4/docs/apis-all/schema-apis/auto-generated-schema-based-get-all-by-stream-api.html

The same query as [get all](https://docs.apimaker.dev/v4/docs/apis-all/schema-apis/auto-generated-schema-based-get-all-api.html), answered as a stream : rows go to the client as the database produces them, so a million rows cost the server a few kilobytes of memory, not the whole result.

| | |
|---|---|
| Method | GET |
| URL | `/api/schema/admin/mysql8/inventory/customers/stream` |
| Body | none |
| Query params | the same as get all : find, skip, limit, sort, select, deep, getTotalCount, any column |
| Answer | `data` : an array of rows, streamed |
| Databases | MongoDB, MySQL, MariaDB, PostgreSQL, SQL Server, Oracle, TiDB, Percona |
| Cached | no : streams always come from the database |
| From code | [`g.sys.db.getAllByStream`](https://docs.apimaker.dev/v4/examples/sys/db/getAllByStream.html) |
| API id | `SCHEMA_GET_ALL_STREAM` (groups, settings, hooks, WebSocket subscriptions) |
| Without a schema | `/api/gen/admin/mysql8/inventory/customers/stream` : [below](https://docs.apimaker.dev/v4/docs/apis-all/schema-apis/auto-generated-schema-based-get-all-by-stream-api.html#without-a-schema) |

> **Generated (`/api/gen`) or schema (`/api/schema`) ?**
>
> Both families offer the same operations on the same URL pattern. The **schema APIs** read the [schema](https://docs.apimaker.dev/v4/docs/schema/schema.html) of the table : values are converted to their types, validated, encrypted, defaulted and numbered before they reach the database. The **generated APIs** are schemaless : they pass your data as it is, so a number sent as `"22"` stays a string. Use the schema APIs for the tables your application writes to, and the generated APIs for quick reads and for tables without a schema.

## The URL

`/api/schema/admin/mysql8/inventory/customers/stream` : replace `admin` with the user path of your account, `mysql8` with your instance, `inventory` with the database and `customers` with the table. The examples use this table :

| customer_id | first_name | last_name | last_update           | pincode | isActive |
|-------------|------------|-----------|-----------------------|---------|----------|
| 1           | Bob        | lin       | 2022-11-14 04: 34: 58 | 382345  | 1        |
| 2           | Alice      | Page      | 2022-10-15 02: 10: 40 | 382346  | 1        |
| 3           | Mallory    | Brown     | 2022-09-13 03: 44: 05 | 382347  | 1        |
| 4           | Eve        | Mathly    | 2022-11-12 01: 59: 33 | 382348  | 1        |
| 5           | Eve        | Page      | 2022-11-12 01: 59: 33 | 382349  | 1        |

## The call

```text
GET /api/schema/admin/mysql8/inventory/customers?find={isActive:1}&sort=customer_id&select=customer_id,first_name
```

```json
{ "data": [
    { "customer_id": 1, "first_name": "Bob" },
    { "customer_id": 2, "first_name": "Alice" }
], "success": true, "statusCode": 200, "meta": {} }
```

- The envelope is the same as [get all](https://docs.apimaker.dev/v4/docs/apis-all/schema-apis/auto-generated-schema-based-get-all-api.html), but `data` comes first and its rows are written in chunks while the query runs. `success`, `statusCode`, `meta` and `totalCount` close the answer. Parse it as JSON when it is complete, or read it with a streaming JSON parser as it arrives.
- If the database fails while rows were already sent, the answer still ends as valid JSON : `], "success": false, "statusCode": 500, "errors": [...] }`. Read `success` at the end.

## When to use it

- Exports, reports, synchronisations, copies of a table : anything which reads many rows and would not fit a normal answer.
- Every filter of get all works : `find`, `sort`, `select`, `deep`, `skip`, `limit`.
- The answer is never cached, and the rows are never compressed in memory : `x-am-data-source` is always `api`.

## Hooks on a stream

- Pre hooks run before the query and can change it.
- Post hooks run when the stream ends. They can not change what was sent, and `g.res.output` holds only the last chunk of rows, so do not use a post hook to reshape a stream : reshape in a custom API around `g.sys.db.getAllByStream`.

## From your code

In a custom API the same call hands you every row as it arrives, without building the whole array :

**TypeScript**

```typescript
let count = 0;
await g.sys.db.getAllByStream({
    instance: 'mysql8', database: 'inventory', collection: 'customers',
    queryParams: { find: { isActive: 1 } },
}, (row) => {
    count++; // one object at a time, in order
});
return { count };
```

**Python**

```python
count = 0


def take(row):
    nonlocal count
    count += 1  # one object at a time, in order


await g.sys.db.get_all_by_stream(GetAll('mysql8', 'inventory', 'customers', find={'isActive': 1}), take)
return {'count': count}
```

**PHP**

```php
$count = 0;
$g->sys->db->getAllByStream(new GetAll('mysql8', 'inventory', 'customers', find: ['isActive' => 1]), function ($row) use (&$count) {
    $count++; // one object at a time, in order
});
return ['count' => $count];
```

**C#**

```csharp
var count = 0;
await g.Sys.Db.GetAllByStream(new GetAll("mysql8", "inventory", "customers") { Find = new JsonObject { ["isActive"] = 1 } }, row =>
{
    count++; // one object at a time, in order
});
return new { count };
```

**Go**

```go
count := 0
err := g.Sys.Db.GetAllByStream(am.GetAll{Instance: "mysql8", Database: "inventory", Collection: "customers", Find: am.M{"isActive": 1}}, func(row am.M) error {
	count++ // one object at a time, in order
	return nil
})
if err != nil {
	return nil, err
}
return am.M{"count": count}, nil
```

**Rust**

```rust
let mut count = 0;
g.sys.db.get_all_by_stream(GetAll::new("mysql8", "inventory", "customers").find(json!({ "isActive": 1 })), |_row| {
    count += 1; // one object at a time, in order
    Ok(())
})?;
Ok(json!({ "count": count }))
```

**Java**

```java
int[] count = { 0 };
g.sys.db.getAllByStream(new GetAll("mysql8", "inventory", "customers").find(Map.of("isActive", 1)), row -> {
    count[0]++; // one object at a time, in order
});
return Map.of("count", count[0]);
```

**Kotlin**

```kotlin
var count = 0
g.sys.db.getAllByStream(GetAll("mysql8", "inventory", "customers").find(mapOf("isActive" to 1))) { row ->
    count++ // one object at a time, in order
}
return mapOf("count" to count)
```

## Without a schema : /api/gen

Every table also answers this operation under `/api/gen`, with or without a schema :

| | |
|---|---|
| URL | `/api/gen/admin/mysql8/inventory/customers/stream` |
| API id | `GEN_GET_ALL_STREAM` (groups, settings, hooks, WebSocket subscriptions) |
| From code | [`g.sys.db.gen.getAllByStreamGen`](https://docs.apimaker.dev/v4/examples/sys/db/getAllByStream.html#without-a-schema) |

The data passes as it is : nothing is converted, validated, defaulted, encrypted or numbered, and a filter matches the type you send (`"22"` does not match the number `22`). On MongoDB, strings which look like ObjectIds are converted. The query params, headers and errors are the ones of this page.

## Headers

Every call takes the [request headers](https://docs.apimaker.dev/v4/docs/apis-all/header/requestHeader.html) : the tokens, `x-am-response-case`, `x-am-content-type-response` (JSON, XML, YAML…), `x-am-response-object-type: make_flat`, `x-am-cache-control`, `x-am-get-encrypted-data`, `x-am-internationalization`, `x-am-tenant-username`, `x-am-meta`.

## Errors

| Code | When |
|---|---|
| `401` | The token in `x-am-authorization` is missing or invalid. |
| `403` | No group of the API user grants this API of this table, or a field of the request. |
| `404` | The instance, database or table of the URL does not exist, or the table has no schema (use `/api/gen`). |
| `400` | The query or the body is wrong : the [error messages](https://docs.apimaker.dev/v4/docs/apis-all/error-codes.html) say which key and why. |

## Related

- [All APIs at a glance](https://docs.apimaker.dev/v4/docs/apis-all/overview.html) · [Query params](https://docs.apimaker.dev/v4/docs/apis-all/query-params/query-params.html) · [Response format](https://docs.apimaker.dev/v4/docs/apis-all/response-format.html) · [Pre hooks](https://docs.apimaker.dev/v4/docs/apis-all/hooks/preHook-api.html) and [post hooks](https://docs.apimaker.dev/v4/docs/apis-all/hooks/postHook-api.html) · [Automatic caching](https://docs.apimaker.dev/v4/docs/features/automatic-caching.html)
- [Table schema](https://docs.apimaker.dev/v4/docs/schema/schema.html) : what the conversions and validations do to every write.