Zum Hauptinhalt springen

Batch

Category: Integration
Node id: batch

Processes arrays of items in configurable batches with parallel execution, validation, and transformation

What it does​

  • Six batch operations: process, validate, transform, load, chunk, parallel
  • Parallel worker execution with configurable concurrency
  • Continue-on-error for fault-tolerant batch processing
  • Batch statistics with success/failure counts and processing rate
  • Chunked processing for memory-efficient large datasets

Use cases​

  • Batch-validate thousands of FHIR resources before import
  • Parallel-process patient records for bulk data transformation
  • Chunk large datasets for incremental loading into FHIR servers

Runtime​

  • Performance tier: variable

Input ports​

NameTypeRequiredDescription
batchjsonYesArray of items to process in batches
metadatajsonNoMetadata to attach to the batch job

Output ports​

NameTypeRequiredDescription
resultjsonNoFull batch processing result with output, job metadata, and errors
statisticsjsonNoProcessing statistics including counts and success rate
successbooleanNoWhether all items were processed without errors

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
OperationoperationenumYesprocessprocess, validate, transform, load, chunk, parallelBatch operation to perform
Batch Sizebatch_sizenumberNo100min 1; max 10000Number of items per batch
ParallelismparallelismnumberNo1min 1; max 100Number of parallel workers
Continue On Errorcontinue_on_errorbooleanNotrueContinue processing when individual items fail

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Processing Functionprocessing_functionenumNoidentityidentity, uppercaseItem processing function
Transform Functiontransform_functionenumNoidentityidentity, extract_fieldItem transformation function
Extract Fieldextract_fieldstringNoField name to extract (for extract_field transform)
DestinationdestinationstringNoDestination identifier for load operation
Validation Rulesvalidation_rulesjsonNoValidation rules for validate operation

Errors​

CodeDescription
INVALID_OPERATIONThe specified batch operation is not recognized
INVALID_BATCH_DATAInput batch is not a valid array
BATCH_PROCESSING_ERROROne or more items failed during batch processing

Runtime notes​

  • success is false if any item failed — check statistics.success_rate and result.errors for partial failure details
  • chunk operation returns an array of chunk objects (not flat items) — each has chunk_number, items, and size
  • With continue_on_error=true, failed items are skipped and reported in result.errors; output contains only successful items
  • parallelism > 1 triggers concurrent processing — output order may differ from input order

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Etwas unklar? Senden Sie Feedback zur Dokumentation per E-Mail..