Parquet Writer
Category: Data
Node id: parquet-writer
Standards: EHDS
Writes data to Apache Parquet columnar format with auto-schema inference and configurable compression (zstd, snappy, gzip)
What it does
- Auto-schema inference from input data
- Four compression codecs: snappy, gzip, zstd, none
- Multiple input sources: data, records, or FHIR resources
- Automatic parent directory creation
- Path security validation
Use cases
- Export FHIR resources to columnar format for analytics
- Write clinical data to Parquet for data warehouse ingestion
- Archive healthcare datasets in compressed columnar format
Runtime
- Performance tier:
medium
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
data | json | No | Array of records to write |
records | json | No | Array of records (alternative input) |
resources | json | No | Array of FHIR resources (alternative input) |
instances | json | No | A dicom-deid/dicom array envelope ({instances:[...]}); flattened to one row per instance. With embed_file_column set, each row also carries the whole file (base64). |
file_path | text | No | Output file path (overrides config) |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
result | json | No | Write result containing file_path, rows_written, file_size_bytes, compression, and schema |
success | boolean | No | Whether the operation succeeded |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| File Path | file_path | string | Yes | Output Parquet file path | ||
| Compression | compression | enum | No | zstd | snappy, gzip, zstd, none | Compression codec |
| Exclude Columns | exclude_columns | string | No | Comma-separated field names left OUT of the written file. The records the upstream node produced are not touched, so a sibling node still sees every field. Use it to keep an identifier out of a published dataset while a restricted sink still receives it. | ||
| Embed as binary | embed_file_binary | boolean | No | false | Store the embedded file as a raw BYTE_ARRAY instead of base64 text. base64 puts a 1.33x floor on the payload that compression does not recover -- on already-compressed pixel data zstd and no compression produce the SAME size. Changes the column type from STRING to BINARY for every reader of the file, so it is opt-in. |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Page Buffer (bytes) | page_buffer_size | number | No | Parquet page buffer. Defaults to 8 MiB when embed_file_column is set, otherwise parquet-go's 256 KiB. A value close to the page size makes the writer degrade badly: a few-hundred-KiB DICOM instance per row cost 6.73x the payload at 256 KiB pages and 1.93x at 8 MiB, and peak memory was lower with the larger buffer, not higher. | ||
| Create Directory | create_directory | boolean | No | true | Create parent directories if they do not exist | |
| Base Directory | base_directory | string | No | Base directory for path security validation | ||
| Name By Field | name_by_field | string | No | When set, name the output file by this record field ("key" or "parent.child", e.g. regenerated_uids.study_instance_uid) from the first record that has it, keeping the configured directory and extension. Lets each study write its own part-file instead of overwriting a fixed name. | ||
| Embed File Column | embed_file_column | string | No | When set, read each row's file (path in embed_file_from, sandboxed to the DICOM temp cache) and store its bytes base64 in this column. Turns a de-id envelope into a self-contained Parquet with the whole .dcm per row. | ||
| Embed File Source Field | embed_file_from | string | No | file_path | Record field holding the local file path to embed (default file_path). | |
| Drop Source Path | embed_file_drop_source | boolean | No | true | Remove the source path field after embedding so the archive does not leak container-local temp paths. |
Errors
| Code | Description |
|---|---|
WRITE_FAILED | Failed to write Parquet data to disk |
EMPTY_DATA | No data provided to write |
SCHEMA_INFERENCE_FAILED | Could not infer Parquet schema from input data |
PATH_SECURITY_VIOLATION | Output path is outside the allowed base directory |
Runtime notes
- Schema is inferred from the first record; all records must share the same top-level keys
- Nested objects and arrays are automatically serialized to JSON string columns
- Accepts data from 'data', 'records', 'resources', or 'input' keys (checked in that order)
- Bad rows that fail encoding are silently skipped; rows_written may be less than input length
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
Quelque chose n'est pas clair ? Envoyez vos commentaires sur la documentation par courriel.