Saltar al contenido principal

Parquet Writer

Category: Data
Node id: parquet-writer

Standards: EHDS

Writes data to Apache Parquet columnar format with auto-schema inference and configurable compression (zstd, snappy, gzip)

What it does​

  • Auto-schema inference from input data
  • Four compression codecs: snappy, gzip, zstd, none
  • Multiple input sources: data, records, or FHIR resources
  • Automatic parent directory creation
  • Path security validation

Use cases​

  • Export FHIR resources to columnar format for analytics
  • Write clinical data to Parquet for data warehouse ingestion
  • Archive healthcare datasets in compressed columnar format

Runtime​

  • Performance tier: medium

Input ports​

NameTypeRequiredDescription
datajsonNoArray of records to write
recordsjsonNoArray of records (alternative input)
resourcesjsonNoArray of FHIR resources (alternative input)
instancesjsonNoA dicom-deid/dicom array envelope ({instances:[...]}); flattened to one row per instance. With embed_file_column set, each row also carries the whole file (base64).
file_pathtextNoOutput file path (overrides config)

Output ports​

NameTypeRequiredDescription
resultjsonNoWrite result containing file_path, rows_written, file_size_bytes, compression, and schema
successbooleanNoWhether the operation succeeded

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
File Pathfile_pathstringYesOutput Parquet file path
CompressioncompressionenumNozstdsnappy, gzip, zstd, noneCompression codec
Exclude Columnsexclude_columnsstringNoComma-separated field names left OUT of the written file. The records the upstream node produced are not touched, so a sibling node still sees every field. Use it to keep an identifier out of a published dataset while a restricted sink still receives it.
Embed as binaryembed_file_binarybooleanNofalseStore the embedded file as a raw BYTE_ARRAY instead of base64 text. base64 puts a 1.33x floor on the payload that compression does not recover -- on already-compressed pixel data zstd and no compression produce the SAME size. Changes the column type from STRING to BINARY for every reader of the file, so it is opt-in.

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Page Buffer (bytes)page_buffer_sizenumberNoParquet page buffer. Defaults to 8 MiB when embed_file_column is set, otherwise parquet-go's 256 KiB. A value close to the page size makes the writer degrade badly: a few-hundred-KiB DICOM instance per row cost 6.73x the payload at 256 KiB pages and 1.93x at 8 MiB, and peak memory was lower with the larger buffer, not higher.
Create Directorycreate_directorybooleanNotrueCreate parent directories if they do not exist
Base Directorybase_directorystringNoBase directory for path security validation
Name By Fieldname_by_fieldstringNoWhen set, name the output file by this record field ("key" or "parent.child", e.g. regenerated_uids.study_instance_uid) from the first record that has it, keeping the configured directory and extension. Lets each study write its own part-file instead of overwriting a fixed name.
Embed File Columnembed_file_columnstringNoWhen set, read each row's file (path in embed_file_from, sandboxed to the DICOM temp cache) and store its bytes base64 in this column. Turns a de-id envelope into a self-contained Parquet with the whole .dcm per row.
Embed File Source Fieldembed_file_fromstringNofile_pathRecord field holding the local file path to embed (default file_path).
Drop Source Pathembed_file_drop_sourcebooleanNotrueRemove the source path field after embedding so the archive does not leak container-local temp paths.

Errors​

CodeDescription
WRITE_FAILEDFailed to write Parquet data to disk
EMPTY_DATANo data provided to write
SCHEMA_INFERENCE_FAILEDCould not infer Parquet schema from input data
PATH_SECURITY_VIOLATIONOutput path is outside the allowed base directory

Runtime notes​

  • Schema is inferred from the first record; all records must share the same top-level keys
  • Nested objects and arrays are automatically serialized to JSON string columns
  • Accepts data from 'data', 'records', 'resources', or 'input' keys (checked in that order)
  • Bad rows that fail encoding are silently skipped; rows_written may be less than input length

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

¿Algo no está claro? Envíe sus comentarios sobre la documentación por correo electrónico..