Aller au contenu principal

File

Category: Data
Node id: file

Performs file system operations (read, write, append, delete, list, exists) and parses file content handed over by another step (parse); CSV and Excel (.xlsx) become records keyed by the header row

What it does​

  • Eight operations: read, write, append, delete, list, exists, read_all (a folder in one run) and parse (content from another step)
  • Multi-format support: text, JSON, CSV, Excel (.xlsx read), binary
  • CSV/xlsx to records: delimiter auto-detection, UTF-8/Latin-1/Windows-1252, header row, sheet selection, row and size caps
  • Path security validation with base directory restriction
  • Automatic parent directory creation
  • Glob pattern filtering for directory listings

Use cases​

  • Read FHIR export files for pipeline processing
  • Write transformed data to JSON or CSV files
  • List files in a bulk export directory for batch ingestion
  • Turn a CSV read over SFTP, or an Excel workbook downloaded from a bucket, into records for a database load

Runtime​

  • Performance tier: medium

Input ports​

NameTypeRequiredDescription
file_pathtextNoFile path (overrides config)
contentanyNoContent to write or append; for parse, the file to parse: text, bytes, base64, or a read_file / object-store download result

Output ports​

NameTypeRequiredDescription
resultjsonNoFull file operation result with path, type, size, and metadata
contentanyNoFile content (read and parse), or for read_all an array of {path, name, size, modified, content} in name order
recordsjsonNoCSV / xlsx only: one object per data row, keyed by the header row
columnsjsonNoCSV / xlsx only: the record keys in file order
successbooleanNoWhether the operation succeeded

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
OperationoperationenumYesreadread, write, append, delete, list, exists, read_all, parseFile operation to perform; parse reads the content input instead of a local file
File Pathfile_pathstringNoPath to the file
File Typefile_typeenumNotexttext, json, csv, xlsx, binaryType of file for read/write/parse parsing

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Base Directorybase_directorystringNoDirectory every path must stay inside (symbolic links included); a relative file_path resolves against it
Create Directorycreate_directorybooleanNofalseCreate parent directories if they do not exist
Glob PatternpatternstringNoFile name pattern filter for list operation
Only Filesonly_filesbooleanNofalseExclude directories from list results
Add Newlineadd_newlinebooleanNotrueAppend a newline after content in append operation
Max Filesmax_filesnumberNo1000min 1; max 10000read_all: hard cap on files read per run
Max File Bytesmax_file_bytesnumberNo10485760min 1; max 67108864read_all: a larger file is skipped (listed in result.skipped), never cut
Max Total Bytesmax_total_bytesnumberNo67108864min 1; max 268435456read_all: hard cap on bytes read per run
Content Encodingcontent_encodingenumNoautoauto, text, base64parse: how a string content input is encoded; auto = text for csv/json/text, base64 for xlsx (a read_file result's own encoding field always wins)
CSV DelimiterdelimiterstringNoautoauto (detects , ; tab |) or one character; "tab" for tab
CSV EncodingencodingenumNoutf-8utf-8, latin-1, windows-1252Character encoding of CSV bytes; utf-8 must be valid UTF-8 (a BOM is dropped)
Header Rowheader_rownumberNo1min 01-based row holding the column names (rows above are skipped); 0 = no header, columns are column_1..column_N
Sheet NamesheetstringNoxlsx: sheet to read by name (default: the sheet at sheet_index)
Sheet Indexsheet_indexnumberNo1min 1xlsx: 1-based sheet position when sheet is not set
Max Rowsmax_rowsnumberNo100000min 1CSV/xlsx: fail when the file has more data rows than this (memory guard; hard limit 2,000,000)
Max Bytesmax_bytesnumberNo67108864min 1CSV/xlsx: fail when the file is larger than this many bytes (hard limit 512 MiB)
Empty As Nullempty_as_nullbooleanNofalseCSV/xlsx: empty cells become null instead of an empty string
Normalise Headersnormalize_headersbooleanNofalseCSV/xlsx: lower-case column names, strip accents, replace other characters with _ ("Data Naixement" -> data_naixement)
Raw Cell Valuesraw_valuesbooleanNofalsexlsx: return unformatted cell values (date serial numbers, full-precision numbers)
Source Fieldsource_fieldstringNoCSV/xlsx: add this field to every record holding the file path (read) or the upstream path/key (parse), for lineage

Errors​

CodeDescription
FILE_NOT_FOUNDThe specified file does not exist
PERMISSION_DENIEDInsufficient permissions to access the file
PATH_SECURITY_VIOLATIONFile path is outside the allowed base directory
INVALID_OPERATIONThe specified file operation is not recognized

Runtime notes​

  • content output is only populated for read, read_all and parse; write/append/delete/list/exists return nil
  • read_all reads every regular file directly in file_path (no subdirectories, no symlinks) matching pattern, in name order, bounded by max_files, max_file_bytes and max_total_bytes; result.truncated/truncated_reason report a cap that stopped it, result.skipped lists files not read, and success is false unless every matching file was read
  • result shape varies by operation — read includes size/modified_time, write includes bytes_written, list includes files array
  • Path traversal (..) is rejected; use base_directory config for sandboxed access
  • With base_directory, a relative file_path resolves against it; containment uses filepath.Rel (base /x/in does not admit /x/inbox) and symbolic links may not lead outside it
  • CSV read returns [][]string (rows of columns, header included) in content, as it always has, and the records in records; xlsx read and every parse return the records in content too
  • CSV/xlsx values are never type-guessed: every cell is text as the file holds it (leading zeros kept); empty_as_null turns empty cells into null
  • A CSV row with more or fewer fields than the header, bytes invalid in the declared encoding, an xlsx value right of the last header column, or more than max_rows rows fails the node — nothing is truncated or dropped silently
  • xlsx: cells with Excel's built-in short-date format come back as YYYY-MM-DD; custom date formats come back as displayed (normalise them with transform parse_date)
  • parse with a read_file result whose truncated flag is set fails: raise the reader's size cap instead of parsing a partial file

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Quelque chose n'est pas clair ? Envoyez vos commentaires sur la documentation par courriel.