Impala (Direct)
Category: Data
Node id: impala-direct
Standards: HIPAA, GDPR
Run SQL directly against Cloudera Impala over Arrow Flight SQL — the node opens its own connection to the coordinator with no Hydra Connect proxy. SELECT/SHOW/DESCRIBE return result.columns+result.rows; DDL/DML (CREATE/INSERT/UPDATE/DELETE/…) return result.rows_affected. Auth is LDAP (username/password, e.g. Cloudera Data Warehouse) or Kerberos, configured on the node.
What it does
- Direct Arrow Flight SQL (no Connect proxy)
- LDAP (user/password) or Kerberos auth
- Inline or saved-connection server config
Runtime
- Performance tier:
variable
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
sql | text | No | Optional SQL overriding config.sql |
params | json | No | Optional bind parameters for the SQL '?' placeholders |
tables | json | No | Optional {table: [rows]} payload (or a dicom-to-omop result with omop_tables). When present, the node CREATEs a Parquet table per table and INSERTs the rows — landing them as Parquet on HDFS. Wire map_omop:result here. |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
result | json | No | Parsed JSON response payload from the vendor API |
metadata | json | No | Operation, HTTP status code, and content length |
success | boolean | No | True if HTTP 2xx |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| SQL | sql | text | No | SQL to run (raw-SQL mode). SELECT/SHOW/DESCRIBE return rows; CREATE/INSERT/… return rows affected. Ignored when a tables payload is wired in. | ||
| Max Rows | max_rows | number | No | 0 | Cap rows returned for queries (0 = no cap). |
Load
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Target Database | target_database | string | No | Database for table-load mode (defaults to impala_database). Tables are created STORED AS PARQUET → written to HDFS. | ||
| Table Prefix | table_prefix | string | No | Optional prefix prepended to each loaded table name (e.g. 'dicom_'). | ||
| Drop Existing | drop_existing | boolean | No | false | DROP each target table before creating it (full reload). For EXTERNAL tables this drops only the metadata; the HDFS files remain. | |
| Replace By Key | replace_keys | json | No | {table: column} naming the column that identifies a row across runs. Before inserting, the node asks which of those keys the table already holds and inserts only the rest, so a retried study leaves one copy instead of two - and an insert that failed halfway is completed rather than repeated. It does NOT update a row whose key is already there: Impala only modifies Kudu and Iceberg tables, so there is no delete to do it with. Absent = append. | ||
| External Table | external | boolean | No | false | Create EXTERNAL tables at an explicit HDFS LOCATION instead of managed warehouse tables. Requires External Location. | |
| External Location (HDFS) | external_location | string | No | Base HDFS dir for EXTERNAL tables, e.g. /warehouse/tablespace/external/hive/edi_pax_dicom. Each table is created at <location>/<table>; Impala writes the Parquet files there on INSERT. |
Connection
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Impala Host | impala_host | string | No | Impala coordinator host (Flight SQL / HS2). Required for Impala operations. | ||
| Impala Port | impala_port | number | No | Impala port (Flight SQL default 443/tls; HS2 21050). | ||
| Impala Protocol | impala_protocol | enum | No | flightsql | flightsql, hs2 | Wire protocol: Arrow Flight SQL (preferred) or HiveServer2 Thrift. |
| Database | impala_database | string | No | Default Impala database/schema for unqualified table names. | ||
| Impala TLS | impala_tls | boolean | No | true | Use TLS for the Impala connection (grpc+tls for Flight SQL, https for HS2). | |
| HS2 HTTP Path | impala_http_path | string | No | cliservice | HiveServer2 HTTP path for the hs2 protocol (Cloudera Data Warehouse uses 'cliservice'). | |
| HDFS / httpFS URL | hdfs_url | string | No | WebHDFS/httpFS base URL, e.g. https://httpfs.cluster:14000. Required for HDFS operations. | ||
| HDFS Base Path | hdfs_base | string | No | Base path that relative HDFS paths resolve against, e.g. /data/dicom. |
Auth
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Impala Username (LDAP) | impala_username | string | No | LDAP user for Impala basic-auth (Cloudera Data Warehouse). Leave blank to use Kerberos. | ||
| Impala Password (LDAP) | impala_password | secret | No | LDAP password for the Impala user. Stored as a secret. | ||
| Kerberos Principal | kerberos_principal | string | No | Service principal, e.g. tietai@REALM. Leave blank for a non-Kerberized cluster. | ||
| Kerberos Realm | kerberos_realm | string | No | Realm (defaults to the part after @ in the principal). | ||
| KDC Host | kerberos_kdc | string | No | Key Distribution Center host, e.g. kdc.cluster. | ||
| Keytab (base64) | keytab_base64 | secret | No | Base64-encoded keytab for the principal. Stored as a secret; sent to Connect over the call. |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Mock (dev) | mock | boolean | No | false | Use an in-memory mock backend (no cluster/Kerberos) for local development and testing. |
Errors
| Code | Description |
|---|---|
SERVER_REQUIRED | No Impala host configured (and mock disabled) |
Runtime notes
- Independent of Hydra Connect — connects straight to the Impala coordinator.
- Build requires -tags cloudera_flightsql for the Flight SQL driver.
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
¿Algo no está claro? Envíe sus comentarios sobre la documentación por correo electrónico..