HDFS Put (WebHDFS / httpFS)
Category: Data
Node id: cloudera-webhdfs-put
Standards: HIPAA, GDPR
Upload files straight to HDFS over WebHDFS/httpFS with its own connection (no Hydra Connect proxy). Direct + Kerberos (SPNEGO) to the cluster httpFS/NameNode when Knox is token-gated, or Knox Bearer token / HTTP Basic (LDAP). Single-file (file_path/data) or batch: wire a dicom-deid result (instances) to land one de-identified .dcm per instance.
What it does
- Direct WebHDFS/httpFS + Kerberos (SPNEGO)
- Knox Bearer / HTTP Basic (LDAP)
- Batch: one de-identified .dcm per instance
- Batch: per-study series index (_series.json)
- Handles the CREATE datanode redirect + re-auth
Runtime
- Performance tier:
variable
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
instances | json | No | A dicom-deid result envelope ({instances:[{file_path|deid_data, regenerated_uids,...}]}). Each emitted instance is written to <hdfs_path>/<deidStudyUID>/<deidSopUID>.dcm. Wire dicom_deid:result here. |
file_path | text | No | Single-file mode: local file to upload (e.g. a parquet-writer result.file_path on the same container). |
data | json | No | Single-file mode: inline bytes/string to write when no file_path is given. |
hdfs_path | text | No | Destination HDFS path (single-file) or directory (batch); overrides config. A trailing '/' is treated as a directory and the source basename appended. |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
result | json | No | Parsed JSON response payload from the vendor API |
metadata | json | No | Operation, HTTP status code, and content length |
success | boolean | No | True if HTTP 2xx |
Settings
Set at least one of: hdfs_url, knox_url.
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Username (LDAP) | username | string | No | LDAP / basic-auth user (Knox Basic mode). | ||
| Password (LDAP) | password | password | No | LDAP / basic-auth password. | ||
| Knox Token (Bearer) | knox_token | password | No | Knox JWT access token, sent as Bearer (Knox token topologies). | ||
| Knox Token URL | knox_token_url | string | No | Optional knoxtoken endpoint to fetch a Bearer token using the LDAP creds. | ||
| HDFS Destination | hdfs_path | string | No | Target HDFS path (single-file) or base directory (batch), e.g. /data/pandora/radiologia/edi_pax_images | ||
| Overwrite | overwrite | boolean | No | true | Overwrite an existing file at the destination. | |
| Remove the local file after upload | delete_after_put | boolean | No | true | Single-file mode: remove file_path once the upload has succeeded. Only ever removes a file this pipeline produced (resolved under the temp root, in a hydra-owned directory) -- an arbitrary path is uploaded but kept. A FAILED upload never deletes. Set false when another node also consumes the same file: sibling nodes run in parallel, so a second reader can still have it open. | |
| Write series index (batch) | write_series_manifest | boolean | No | true | Batch mode: also write <hdfs_path>/<deidStudyUID>/_series.json listing each series and its instances. The HDFS layout is flat, so without it a reader cannot tell which series an instance belongs to. |
Server
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Direct WebHDFS URL | hdfs_url | string | No | Direct WebHDFS/httpFS base (skips Knox), e.g. https://httpfs.cluster:14000 or http://namenode:9870. Use with Kerberos. | ||
| Knox Gateway URL | knox_url | string | No | Knox gateway base, e.g. https://console-cdp.apps.pvc-prod.salud.madrid.org (used when hdfs_url is not set). | ||
| Knox Topology | knox_topology | string | No | cdp-proxy-api | WebHDFS topology under the Knox gateway (default cdp-proxy-api). | |
| Skip TLS Verify | tls_skip_verify | boolean | No | false | Skip TLS certificate verification (internal / self-signed cert, e.g. SERMAS CA). |
Kerberos
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Kerberos Principal | kerberos_principal | string | No | e.g. ufpandoraradiologia@SALUD.MADRID.ORG | ||
| Kerberos Realm | kerberos_realm | string | No | e.g. SALUD.MADRID.ORG (defaults to the principal's realm). | ||
| KDC (host:port) | kerberos_kdc | string | No | A KDC address, e.g. dc01.salud.madrid.org:88. | ||
| Keytab (base64) | kerberos_keytab_base64 | password | No | base64-encoded keytab for the principal. Presence of this enables Kerberos/SPNEGO auth. | ||
| Service SPN (override) | kerberos_spn | string | No | Override the SPNEGO service principal (default HTTP/<endpoint-host>). | ||
| Enctype pin | kerberos_enctypes | string | No | Optional space-separated enctype pin, e.g. rc4-hmac — when an AD keytab only carries RC4 but the KDC advertises AES first. |
Errors
| Code | Description |
|---|---|
ENDPOINT_REQUIRED | Neither hdfs_url nor knox_url configured |
Runtime notes
- Single-file mode removes the local file_path once the upload succeeds (delete_after_put, default true): the copy is redundant once HDFS has it, and a multi-GB export left behind fills the disk. A FAILED upload never deletes, so the file stays to retry from.
- Kerberos mode skips Knox — point hdfs_url at the cluster httpFS (:14000) or NameNode WebHDFS (:9870) and supply the keytab.
- Batch mode reads each de-id instance's spilled file_path (same container FS) and PUTs it under hdfs_path.
- Batch mode also writes a per-study _series.json index (write_series_manifest, default true). The layout is flat -- <study>/<sop>.dcm -- so the series level exists nowhere in the paths; rebuilding it later would mean one ranged read per instance through a gateway that serialises them per user. An instance whose upload failed is left out, and a study retrieved without per-series UIDs gets no index at all rather than an empty one.
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
¿Algo no está claro? Envíe sus comentarios sobre la documentación por correo electrónico..