Zum Hauptinhalt springen

Impala (Direct)

Category: Data
Node id: impala-direct

Standards: HIPAA, GDPR

Run SQL directly against Cloudera Impala over Arrow Flight SQL — the node opens its own connection to the coordinator with no Hydra Connect proxy. SELECT/SHOW/DESCRIBE return result.columns+result.rows; DDL/DML (CREATE/INSERT/UPDATE/DELETE/…) return result.rows_affected. Auth is LDAP (username/password, e.g. Cloudera Data Warehouse) or Kerberos, configured on the node.

What it does​

  • Direct Arrow Flight SQL (no Connect proxy)
  • LDAP (user/password) or Kerberos auth
  • Inline or saved-connection server config

Runtime​

  • Performance tier: variable

Input ports​

NameTypeRequiredDescription
sqltextNoOptional SQL overriding config.sql
paramsjsonNoOptional bind parameters for the SQL '?' placeholders
tablesjsonNoOptional {table: [rows]} payload (or a dicom-to-omop result with omop_tables). When present, the node CREATEs a Parquet table per table and INSERTs the rows — landing them as Parquet on HDFS. Wire map_omop:result here.

Output ports​

NameTypeRequiredDescription
resultjsonNoParsed JSON response payload from the vendor API
metadatajsonNoOperation, HTTP status code, and content length
successbooleanNoTrue if HTTP 2xx

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
SQLsqltextNoSQL to run (raw-SQL mode). SELECT/SHOW/DESCRIBE return rows; CREATE/INSERT/… return rows affected. Ignored when a tables payload is wired in.
Max Rowsmax_rowsnumberNo0Cap rows returned for queries (0 = no cap).

Load​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Target Databasetarget_databasestringNoDatabase for table-load mode (defaults to impala_database). Tables are created STORED AS PARQUET → written to HDFS.
Table Prefixtable_prefixstringNoOptional prefix prepended to each loaded table name (e.g. 'dicom_').
Drop Existingdrop_existingbooleanNofalseDROP each target table before creating it (full reload). For EXTERNAL tables this drops only the metadata; the HDFS files remain.
Replace By Keyreplace_keysjsonNo{table: column} naming the column that identifies a row across runs. Before inserting, the node asks which of those keys the table already holds and inserts only the rest, so a retried study leaves one copy instead of two - and an insert that failed halfway is completed rather than repeated. It does NOT update a row whose key is already there: Impala only modifies Kudu and Iceberg tables, so there is no delete to do it with. Absent = append.
External TableexternalbooleanNofalseCreate EXTERNAL tables at an explicit HDFS LOCATION instead of managed warehouse tables. Requires External Location.
External Location (HDFS)external_locationstringNoBase HDFS dir for EXTERNAL tables, e.g. /warehouse/tablespace/external/hive/edi_pax_dicom. Each table is created at <location>/<table>; Impala writes the Parquet files there on INSERT.

Connection​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Impala Hostimpala_hoststringNoImpala coordinator host (Flight SQL / HS2). Required for Impala operations.
Impala Portimpala_portnumberNoImpala port (Flight SQL default 443/tls; HS2 21050).
Impala Protocolimpala_protocolenumNoflightsqlflightsql, hs2Wire protocol: Arrow Flight SQL (preferred) or HiveServer2 Thrift.
Databaseimpala_databasestringNoDefault Impala database/schema for unqualified table names.
Impala TLSimpala_tlsbooleanNotrueUse TLS for the Impala connection (grpc+tls for Flight SQL, https for HS2).
HS2 HTTP Pathimpala_http_pathstringNocliserviceHiveServer2 HTTP path for the hs2 protocol (Cloudera Data Warehouse uses 'cliservice').
HDFS / httpFS URLhdfs_urlstringNoWebHDFS/httpFS base URL, e.g. https://httpfs.cluster:14000. Required for HDFS operations.
HDFS Base Pathhdfs_basestringNoBase path that relative HDFS paths resolve against, e.g. /data/dicom.

Auth​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Impala Username (LDAP)impala_usernamestringNoLDAP user for Impala basic-auth (Cloudera Data Warehouse). Leave blank to use Kerberos.
Impala Password (LDAP)impala_passwordsecretNoLDAP password for the Impala user. Stored as a secret.
Kerberos Principalkerberos_principalstringNoService principal, e.g. tietai@REALM. Leave blank for a non-Kerberized cluster.
Kerberos Realmkerberos_realmstringNoRealm (defaults to the part after @ in the principal).
KDC Hostkerberos_kdcstringNoKey Distribution Center host, e.g. kdc.cluster.
Keytab (base64)keytab_base64secretNoBase64-encoded keytab for the principal. Stored as a secret; sent to Connect over the call.

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Mock (dev)mockbooleanNofalseUse an in-memory mock backend (no cluster/Kerberos) for local development and testing.

Errors​

CodeDescription
SERVER_REQUIREDNo Impala host configured (and mock disabled)

Runtime notes​

  • Independent of Hydra Connect — connects straight to the Impala coordinator.
  • Build requires -tags cloudera_flightsql for the Flight SQL driver.

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Etwas unklar? Senden Sie Feedback zur Dokumentation per E-Mail..