Saltar al contenido principal

Intake Verify

Category: Data
Node id: intake-verify
Maturity: beta
Risk tier: B

Verify file and dump deliveries in the intake volume against their SHA-256 manifest and the intake profile (checksums, row counts, volume ranges, schema drift, types, required nulls); accept or quarantine each delivery whole.

What it does​

  • phi

Use cases​

  • Verify every HGUGM extraction delivery before any mapping reads it
  • Quarantine a truncated, corrupted or column-reordered export with a precise reason
  • Register accepted tables in the catalogue so schema drift stays visible across deliveries

Runtime​

  • Performance tier: slow

Input ports​

NameTypeRequiredDescription
triggeranyNoOptional upstream signal (a scheduler tick or an intake-sql-export result); its content is not read.

Output ports​

NameTypeRequiredDescription
verdictsjsonNoOne entry per delivery processed: attempt id, verdict, sorted reasons, warnings and final path. Counts and names only.
acceptedjsonNoAccepted deliveries with their settled path and tables, for downstream readers.
assetsjsonNoCatalogue assets for every accepted table, in the estate-register input shape (fqn <prefix>.<source>.<table>).
accepted_countnumberNoHow many deliveries this run accepted. Gate estate-register on accepted_count > 0: it rejects an empty assets list.
quarantined_countnumberNoHow many deliveries this run quarantined.
errorsjsonNoPresent only when a delivery could not be verified or settled while others were: one entry per delivery with its code (INTAKE_SETTLE or STORE). That delivery stays in incoming/ for the next run.
successbooleanNoTrue only when no delivery was quarantined or left unsettled in this run.

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Intake Rootintake_rootstringYesAbsolute path of the intake volume holding incoming/, accepted/ and quarantine/ (pilot: /srv/hgugm/intake).
Intake ProfileprofilestringNohgugm-cardiology-v1Id of the versioned intake profile holding table contracts, row ranges and required columns. Embedded unless HYDRA_INTAKE_PROFILE_DIR is set.
Delivery IDdelivery_idstringNoVerify only this delivery. Empty verifies every delivery directory waiting in incoming/.

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Parallel Filesmax_parallel_filesnumberNo2min 1; max 16Files hashed and scanned at once within a delivery. Each holds one 1 MiB buffer.
Catalogue Prefixcatalog_prefixstringNohgugmFirst segment of the catalogue FQN emitted for accepted tables.

Errors​

CodeDescription
INTAKE_ROOT_INVALIDintake_root is not an absolute directory with an incoming/ subdirectory
PROFILE_NOT_FOUNDThe intake profile does not exist or the override directory is missing or empty
STORENo verdict store (platform database) is configured, or recording the verdict failed
INTAKE_SETTLEThe delivery could not be verified or moved (unreadable directory, destination already exists); it stays in incoming/

Runtime notes​

  • All-or-nothing per delivery: any reason quarantines the whole directory; range_action: warn findings go to reasons_warn and do not quarantine.
  • The verdict is recorded before the directory moves. If the database write fails the delivery stays in incoming/; it is reported under errors, and the node fails only when no delivery settled in the run.
  • The shipped hgugm-cardiology-v1 profile is a placeholder until the extraction spec is agreed, so every delivery is quarantined with profile_not_approved until then.
  • Dump files are checked for integrity only; their tables are verified after restore, when intake-sql-export produces a Parquet delivery.
  • Reasons, verdict.json and the database carry counts, hashes and names only, never a cell value.
  • A delivery id is used once: a second delivery under a settled id is quarantined beside it with delivery_id_reused.

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

¿Algo no está claro? Envíe sus comentarios sobre la documentación por correo electrónico..