Batch
Category: Integration
Node id: batch
Processes arrays of items in configurable batches with parallel execution, validation, and transformation
What it does
- Six batch operations: process, validate, transform, load, chunk, parallel
- Parallel worker execution with configurable concurrency
- Continue-on-error for fault-tolerant batch processing
- Batch statistics with success/failure counts and processing rate
- Chunked processing for memory-efficient large datasets
Use cases
- Batch-validate thousands of FHIR resources before import
- Parallel-process patient records for bulk data transformation
- Chunk large datasets for incremental loading into FHIR servers
Runtime
- Performance tier:
variable
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
batch | json | Yes | Array of items to process in batches |
metadata | json | No | Metadata to attach to the batch job |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
result | json | No | Full batch processing result with output, job metadata, and errors |
statistics | json | No | Processing statistics including counts and success rate |
success | boolean | No | Whether all items were processed without errors |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Operation | operation | enum | Yes | process | process, validate, transform, load, chunk, parallel | Batch operation to perform |
| Batch Size | batch_size | number | No | 100 | min 1; max 10000 | Number of items per batch |
| Parallelism | parallelism | number | No | 1 | min 1; max 100 | Number of parallel workers |
| Continue On Error | continue_on_error | boolean | No | true | Continue processing when individual items fail |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Processing Function | processing_function | enum | No | identity | identity, uppercase | Item processing function |
| Transform Function | transform_function | enum | No | identity | identity, extract_field | Item transformation function |
| Extract Field | extract_field | string | No | Field name to extract (for extract_field transform) | ||
| Destination | destination | string | No | Destination identifier for load operation | ||
| Validation Rules | validation_rules | json | No | Validation rules for validate operation |
Errors
| Code | Description |
|---|---|
INVALID_OPERATION | The specified batch operation is not recognized |
INVALID_BATCH_DATA | Input batch is not a valid array |
BATCH_PROCESSING_ERROR | One or more items failed during batch processing |
Runtime notes
- success is false if any item failed — check statistics.success_rate and result.errors for partial failure details
- chunk operation returns an array of chunk objects (not flat items) — each has chunk_number, items, and size
- With continue_on_error=true, failed items are skipped and reported in result.errors; output contains only successful items
- parallelism > 1 triggers concurrent processing — output order may differ from input order
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
Quelque chose n'est pas clair ? Envoyez vos commentaires sur la documentation par courriel.