Aller au contenu principal

Stream

Category: Data
Node id: stream

Processes data streams with filter, map, reduce, aggregate, window, buffer, deduplicate, and sort operations

What it does​

  • Eight stream operations: filter, map, reduce, aggregate, window, buffer, deduplicate, sort
  • Time-based windowing with configurable duration
  • Buffered processing with configurable buffer size
  • Field-level deduplication for stream dedup
  • Group-by aggregation with sum, avg, min, max, count

Use cases​

  • Filter real-time HL7 message streams by event type
  • Aggregate vital sign readings over time windows
  • Deduplicate duplicate clinical events from multiple sources
  • Sort incoming lab results by timestamp

Runtime​

  • Performance tier: fast

Input ports​

NameTypeRequiredDescription
streamjsonYesArray of data items to process

Output ports​

NameTypeRequiredDescription
resultjsonNoFull stream processing result with output array, counts, aggregations, windows, and timing
countnumberNoNumber of items in output
successbooleanNoWhether stream processing completed

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
OperationoperationenumYesfilterfilter, map, reduce, aggregate, window, buffer, deduplicate, sortStream processing operation
Filter Fieldfilter_fieldstringNoField name to filter on
Filter Valuefilter_valueanyNoValue to compare against
Filter Operatorfilter_openumNoeqeq, ne, gt, lt, gte, lteComparison operator for filtering
Map Fieldmap_fieldstringNoField to apply map function to
Map Functionmap_functionenumNoidentityidentity, extract, uppercaseTransformation function
Reduce Operationreduce_openumNosumsum, avg, min, max, countReduction operation
Reduce Fieldreduce_fieldstringNovalueField to reduce on
Group Bygroup_bystringNoField to group by for aggregation
Aggregate Fieldaggregate_fieldstringNovalueField to aggregate
Aggregate Operationaggregate_openumNosumsum, avg, min, max, countAggregation operation
Sort Fieldsort_fieldstringNoField to sort by
Sort Ordersort_orderenumNoascasc, descSort direction
Dedupe Fielddedupe_fieldstringNoField to deduplicate on

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Timestamp Fieldtimestamp_fieldstringNotimestampField containing timestamps for windowing
Buffer Sizebuffer_sizenumberNo1000min 1; max 100000Maximum buffer size
Window Duration (ms)window_duration_msnumberNo1000min 1Time window duration in milliseconds

Errors​

CodeDescription
INVALID_OPERATIONThe specified stream operation is not recognized
MISSING_FILTER_FIELDFilter field is required for filter operation
INVALID_STREAM_DATAInput stream is not a valid array

Runtime notes​

  • Output shape varies by operation: filter adds 'filtered', reduce adds 'aggregated', window adds 'windows' and 'window_count'
  • The 'result' output nests the full StreamResult; wire 'count' for simple downstream checks
  • Buffer operation is stateful across invocations — items persist in memory until buffer_size is exceeded
  • Input accepts []interface{} or []map[string]interface{}; single items are wrapped in an array

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Quelque chose n'est pas clair ? Envoyez vos commentaires sur la documentation par courriel.