Intake Verify
Category: Data
Node id: intake-verify
Maturity: beta
Risk tier: B
Verify file and dump deliveries in the intake volume against their SHA-256 manifest and the intake profile (checksums, row counts, volume ranges, schema drift, types, required nulls); accept or quarantine each delivery whole.
What it does
- phi
Use cases
- Verify every HGUGM extraction delivery before any mapping reads it
- Quarantine a truncated, corrupted or column-reordered export with a precise reason
- Register accepted tables in the catalogue so schema drift stays visible across deliveries
Runtime
- Performance tier:
slow
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
trigger | any | No | Optional upstream signal (a scheduler tick or an intake-sql-export result); its content is not read. |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
verdicts | json | No | One entry per delivery processed: attempt id, verdict, sorted reasons, warnings and final path. Counts and names only. |
accepted | json | No | Accepted deliveries with their settled path and tables, for downstream readers. |
assets | json | No | Catalogue assets for every accepted table, in the estate-register input shape (fqn <prefix>.<source>.<table>). |
accepted_count | number | No | How many deliveries this run accepted. Gate estate-register on accepted_count > 0: it rejects an empty assets list. |
quarantined_count | number | No | How many deliveries this run quarantined. |
errors | json | No | Present only when a delivery could not be verified or settled while others were: one entry per delivery with its code (INTAKE_SETTLE or STORE). That delivery stays in incoming/ for the next run. |
success | boolean | No | True only when no delivery was quarantined or left unsettled in this run. |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Intake Root | intake_root | string | Yes | Absolute path of the intake volume holding incoming/, accepted/ and quarantine/ (pilot: /srv/hgugm/intake). | ||
| Intake Profile | profile | string | No | hgugm-cardiology-v1 | Id of the versioned intake profile holding table contracts, row ranges and required columns. Embedded unless HYDRA_INTAKE_PROFILE_DIR is set. | |
| Delivery ID | delivery_id | string | No | Verify only this delivery. Empty verifies every delivery directory waiting in incoming/. |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Parallel Files | max_parallel_files | number | No | 2 | min 1; max 16 | Files hashed and scanned at once within a delivery. Each holds one 1 MiB buffer. |
| Catalogue Prefix | catalog_prefix | string | No | hgugm | First segment of the catalogue FQN emitted for accepted tables. |
Errors
| Code | Description |
|---|---|
INTAKE_ROOT_INVALID | intake_root is not an absolute directory with an incoming/ subdirectory |
PROFILE_NOT_FOUND | The intake profile does not exist or the override directory is missing or empty |
STORE | No verdict store (platform database) is configured, or recording the verdict failed |
INTAKE_SETTLE | The delivery could not be verified or moved (unreadable directory, destination already exists); it stays in incoming/ |
Runtime notes
- All-or-nothing per delivery: any reason quarantines the whole directory; range_action: warn findings go to reasons_warn and do not quarantine.
- The verdict is recorded before the directory moves. If the database write fails the delivery stays in incoming/; it is reported under errors, and the node fails only when no delivery settled in the run.
- The shipped hgugm-cardiology-v1 profile is a placeholder until the extraction spec is agreed, so every delivery is quarantined with profile_not_approved until then.
- Dump files are checked for integrity only; their tables are verified after restore, when intake-sql-export produces a Parquet delivery.
- Reasons, verdict.json and the database carry counts, hashes and names only, never a cell value.
- A delivery id is used once: a second delivery under a settled id is quarantined beside it with delivery_id_reused.
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
Etwas unklar? Senden Sie Feedback zur Dokumentation per E-Mail..