Skip to main content

Parquet Reader

Category: Data
Node id: parquet-reader

Standards: EHDS

Reads data from Apache Parquet files — a local path or bytes handed over by another step — with pagination, schema inspection, and metadata extraction

What it does​

  • Three read modes: data, schema inspection, metadata extraction
  • Pagination with row limit and offset for large files
  • Parquet schema-to-JSON type mapping
  • Row group and key-value metadata extraction
  • Path security validation

Use cases​

  • Read columnar healthcare analytics data from Parquet files
  • Inspect Parquet file schema before data transformation
  • Extract metadata from Parquet exports for audit logging

Runtime​

  • Performance tier: medium

Input ports​

NameTypeRequiredDescription
file_pathtextNoParquet file path (overrides config)
contentanyNoParquet bytes when source is content: []byte, base64 text, an object-store download record (download with format: raw, which keeps data_base64) or a read_file result

Output ports​

NameTypeRequiredDescription
datajsonNoArray of record objects (read operation)
schemajsonNoParquet schema as field-type map
row_countnumberNoTotal number of rows in the file
returnednumberNoNumber of rows returned after pagination
columnsjsonNoColumn detail array (read_schema)
num_columnsnumberNoNumber of columns
num_rowsnumberNoTotal rows in file metadata
num_row_groupsnumberNoNumber of row groups (read_metadata)
key_value_metajsonNoFile key-value metadata (read_metadata)
successbooleanNoWhether the operation succeeded

Settings​

Set at least one of: file_path, source.

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
OperationoperationenumNoreadread, read_schema, read_metadataRead operation mode
SourcesourceenumNofilefile, contentWhere the Parquet bytes come from: file (file_path on this host) or content (the content input, e.g. an S3 download)
File Pathfile_pathstringNoParquet file path (source: file)

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Content Encodingcontent_encodingenumNoautoauto, text, base64How a string content input is encoded; auto = base64 (a read_file result's own encoding field always wins)
Row Limitrow_limitnumberNo0min 0Maximum rows to return (0 = unlimited)
Row Offsetrow_offsetnumberNo0min 0Number of rows to skip
Base Directorybase_directorystringNoBase directory for path security validation

Errors​

CodeDescription
FILE_NOT_FOUNDThe specified Parquet file does not exist
INVALID_PARQUETFile is not a valid Parquet file or is corrupted
PATH_SECURITY_VIOLATIONFile path is outside the allowed base directory

Runtime notes​

  • Output keys differ by operation: read returns data/schema/row_count/returned; read_schema returns schema/columns/num_columns; read_metadata returns num_rows/row_groups/key_value_meta etc.
  • file_path input overrides the config value, allowing dynamic path injection from upstream nodes
  • source: content reads the content input (bytes, base64, an S3/GCS download record or a read_file result) in memory, so no local hop is needed; a truncated read_file result is refused
  • Row offset/limit operate at the read level; row_count always reflects the total rows in the file
  • schema is always included regardless of operation and maps column names to Parquet type strings

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Something unclear? Send documentation feedback by email.