Intake SQL Export
Category: Data
Node id: intake-sql-export
Maturity: beta
Risk tier: B
Export every table of a restored Oracle dump delivery from the read-only staging database into Parquet, reconciling COUNT(*) against the manifest and the exported rows, and hand the generated delivery to intake-verify.
What it does
- phi
- database
Use cases
- Turn an Oracle Data Pump delivery restored on the hospital staging container into a verifiable Parquet delivery
- Prove every restored table holds exactly the rows the hospital manifest declared
Runtime
- Performance tier:
slow
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
trigger | any | No | Optional upstream signal; its content is not read. |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
generated_delivery_id | text | No | Id of the generated delivery now waiting in incoming/: <delivery_id>-export. |
tables | json | No | Per table: staging COUNT(*), exported rows and the manifest row count. |
success | boolean | No | True when every table reconciled and the delivery reached incoming/. |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Staging Connector | connector_id | string | Yes | Saved oracle, mssql or postgres connector for the hospital-side staging database, connecting as the read-only user. | ||
| Source Delivery ID | delivery_id | string | Yes | The accepted dump delivery whose tables were restored by the runbook. | ||
| Intake Root | intake_root | string | Yes | Absolute path of the intake volume holding incoming/, accepted/ and quarantine/ (pilot: /srv/hgugm/intake). | ||
| Intake Profile | profile | string | No | hgugm-cardiology-v1 | Intake profile whose table contracts give the columns to export and their declared types. |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Expected DB User | expected_user | string | No | HYDRA_RO | The staging read-only user. The export refuses to run when the connection is any other user. | |
| Row Group Size | batch_size | number | No | 5000 | min 100; max 100000 | Rows per Parquet row group. |
Errors
| Code | Description |
|---|---|
COUNT_MISMATCH | Staging COUNT(*), exported rows and the manifest row_count are not all equal |
IDENTIFIER_INVALID | A manifest or profile identifier failed the allowlist; no SQL was executed |
NOT_READ_ONLY | The connector is not an oracle/mssql/postgres connector configured read_only and connected as expected_user |
DELIVERY_NOT_ACCEPTED | The source dump delivery is not accepted |
EXPORT_EXISTS | The generated delivery id is already in incoming/, accepted/ or quarantine/ |
EXPORT_LOCKED | Another run is exporting the same dump delivery; retry after it finishes |
STORE | No verdict store or connector resolver is configured |
Runtime notes
- Reads only: COUNT(*) goes through the connector's guarded sql_query service, rows stream through a read-only transaction, and the connector must be configured read_only and connect as expected_user.
- Every identifier is checked against the allowlist before any SQL is built; nothing runs when one fails.
- Output is written under staging-export/ and renamed into incoming/ only when every table reconciled; on failure nothing appears in incoming/.
- Values are exported as text in the profile's declared layouts; intake-verify then parses every one against the declared type.
- The dump itself is restored by scripts/hgugm/restore_oracle_dump.sh, never by Hydra.
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
Quelque chose n'est pas clair ? Envoyez vos commentaires sur la documentation par courriel.