Skip to main content

NDJSON Reader

Category: Data
Node id: ndjson-reader

Standards: EHDS, FHIR-Bulk-Data

Reads FHIR resources from NDJSON files with resource type filtering, pagination, bulk directory reading, and schema inference

What it does​

  • Three read modes: single file, bulk directory, schema inference
  • FHIR resource type filtering
  • Pagination with row limit and offset
  • Gzip-compressed NDJSON file support
  • Resource type counting across files

Use cases​

  • Read FHIR Bulk Data Export files for pipeline processing
  • Infer schema from NDJSON files before transformation
  • Bulk-read all patient resources from an export directory

Runtime​

  • Performance tier: medium

Output ports​

NameTypeRequiredDescription
resourcesjsonNoArray of parsed FHIR resources
total_readnumberNoTotal lines read from the file
total_matchednumberNoTotal lines matching the resource type filter
resource_countsjsonNoMap of resourceType to count
files_readjsonNoList of files read (read_bulk)
schemajsonNoInferred schema from first resource (read_schema)
resource_typetextNoResource type from first resource (read_schema)
successbooleanNoWhether the operation succeeded

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
OperationoperationenumYesreadread, read_bulk, read_schemaRead operation mode
File Pathfile_pathstringNoNDJSON file path (for read and read_schema)
Input Directoryinput_directorystringNoDirectory containing NDJSON files (for read_bulk)
Resource Type Filterresource_type_filterstringNoOnly return resources of this FHIR type (e.g., Patient)

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Row Limitrow_limitnumberNo0min 0Maximum number of resources to return (0 = unlimited)
Row Offsetrow_offsetnumberNo0min 0Number of matching resources to skip
Base Directorybase_directorystringNoBase directory for path security validation

Errors​

CodeDescription
FILE_NOT_FOUNDThe specified NDJSON file does not exist
INVALID_JSON_LINEA line in the NDJSON file is not valid JSON
PATH_SECURITY_VIOLATIONFile path is outside the allowed base directory

Runtime notes​

  • Output keys vary by operation: read returns resources/total_read/total_matched, read_bulk adds files_read, read_schema returns schema/resource_type
  • Supports gzip-compressed .ndjson.gz files transparently
  • Malformed JSON lines are silently skipped; total_read still counts them
  • resource_counts tallies ALL resource types in the file, even those excluded by resource_type_filter

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Something unclear? Send documentation feedback by email.