Stream
Category: Data
Node id: stream
Processes data streams with filter, map, reduce, aggregate, window, buffer, deduplicate, and sort operations
What it does
- Eight stream operations: filter, map, reduce, aggregate, window, buffer, deduplicate, sort
- Time-based windowing with configurable duration
- Buffered processing with configurable buffer size
- Field-level deduplication for stream dedup
- Group-by aggregation with sum, avg, min, max, count
Use cases
- Filter real-time HL7 message streams by event type
- Aggregate vital sign readings over time windows
- Deduplicate duplicate clinical events from multiple sources
- Sort incoming lab results by timestamp
Runtime
- Performance tier:
fast
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
stream | json | Yes | Array of data items to process |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
result | json | No | Full stream processing result with output array, counts, aggregations, windows, and timing |
count | number | No | Number of items in output |
success | boolean | No | Whether stream processing completed |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Operation | operation | enum | Yes | filter | filter, map, reduce, aggregate, window, buffer, deduplicate, sort | Stream processing operation |
| Filter Field | filter_field | string | No | Field name to filter on | ||
| Filter Value | filter_value | any | No | Value to compare against | ||
| Filter Operator | filter_op | enum | No | eq | eq, ne, gt, lt, gte, lte | Comparison operator for filtering |
| Map Field | map_field | string | No | Field to apply map function to | ||
| Map Function | map_function | enum | No | identity | identity, extract, uppercase | Transformation function |
| Reduce Operation | reduce_op | enum | No | sum | sum, avg, min, max, count | Reduction operation |
| Reduce Field | reduce_field | string | No | value | Field to reduce on | |
| Group By | group_by | string | No | Field to group by for aggregation | ||
| Aggregate Field | aggregate_field | string | No | value | Field to aggregate | |
| Aggregate Operation | aggregate_op | enum | No | sum | sum, avg, min, max, count | Aggregation operation |
| Sort Field | sort_field | string | No | Field to sort by | ||
| Sort Order | sort_order | enum | No | asc | asc, desc | Sort direction |
| Dedupe Field | dedupe_field | string | No | Field to deduplicate on |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Timestamp Field | timestamp_field | string | No | timestamp | Field containing timestamps for windowing | |
| Buffer Size | buffer_size | number | No | 1000 | min 1; max 100000 | Maximum buffer size |
| Window Duration (ms) | window_duration_ms | number | No | 1000 | min 1 | Time window duration in milliseconds |
Errors
| Code | Description |
|---|---|
INVALID_OPERATION | The specified stream operation is not recognized |
MISSING_FILTER_FIELD | Filter field is required for filter operation |
INVALID_STREAM_DATA | Input stream is not a valid array |
Runtime notes
- Output shape varies by operation: filter adds 'filtered', reduce adds 'aggregated', window adds 'windows' and 'window_count'
- The 'result' output nests the full StreamResult; wire 'count' for simple downstream checks
- Buffer operation is stateful across invocations — items persist in memory until buffer_size is exceeded
- Input accepts []interface{} or []map[string]interface{}; single items are wrapped in an array
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
Quelque chose n'est pas clair ? Envoyez vos commentaires sur la documentation par courriel.