Zum Hauptinhalt springen

HDFS Put (WebHDFS / httpFS)

Category: Data
Node id: cloudera-webhdfs-put

Standards: HIPAA, GDPR

Upload files straight to HDFS over WebHDFS/httpFS with its own connection (no Hydra Connect proxy). Direct + Kerberos (SPNEGO) to the cluster httpFS/NameNode when Knox is token-gated, or Knox Bearer token / HTTP Basic (LDAP). Single-file (file_path/data) or batch: wire a dicom-deid result (instances) to land one de-identified .dcm per instance.

What it does​

  • Direct WebHDFS/httpFS + Kerberos (SPNEGO)
  • Knox Bearer / HTTP Basic (LDAP)
  • Batch: one de-identified .dcm per instance
  • Batch: per-study series index (_series.json)
  • Handles the CREATE datanode redirect + re-auth

Runtime​

  • Performance tier: variable

Input ports​

NameTypeRequiredDescription
instancesjsonNoA dicom-deid result envelope ({instances:[{file_path|deid_data, regenerated_uids,...}]}). Each emitted instance is written to <hdfs_path>/<deidStudyUID>/<deidSopUID>.dcm. Wire dicom_deid:result here.
file_pathtextNoSingle-file mode: local file to upload (e.g. a parquet-writer result.file_path on the same container).
datajsonNoSingle-file mode: inline bytes/string to write when no file_path is given.
hdfs_pathtextNoDestination HDFS path (single-file) or directory (batch); overrides config. A trailing '/' is treated as a directory and the source basename appended.

Output ports​

NameTypeRequiredDescription
resultjsonNoParsed JSON response payload from the vendor API
metadatajsonNoOperation, HTTP status code, and content length
successbooleanNoTrue if HTTP 2xx

Settings​

Set at least one of: hdfs_url, knox_url.

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Username (LDAP)usernamestringNoLDAP / basic-auth user (Knox Basic mode).
Password (LDAP)passwordpasswordNoLDAP / basic-auth password.
Knox Token (Bearer)knox_tokenpasswordNoKnox JWT access token, sent as Bearer (Knox token topologies).
Knox Token URLknox_token_urlstringNoOptional knoxtoken endpoint to fetch a Bearer token using the LDAP creds.
HDFS Destinationhdfs_pathstringNoTarget HDFS path (single-file) or base directory (batch), e.g. /data/pandora/radiologia/edi_pax_images
OverwriteoverwritebooleanNotrueOverwrite an existing file at the destination.
Remove the local file after uploaddelete_after_putbooleanNotrueSingle-file mode: remove file_path once the upload has succeeded. Only ever removes a file this pipeline produced (resolved under the temp root, in a hydra-owned directory) -- an arbitrary path is uploaded but kept. A FAILED upload never deletes. Set false when another node also consumes the same file: sibling nodes run in parallel, so a second reader can still have it open.
Write series index (batch)write_series_manifestbooleanNotrueBatch mode: also write <hdfs_path>/<deidStudyUID>/_series.json listing each series and its instances. The HDFS layout is flat, so without it a reader cannot tell which series an instance belongs to.

Server​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Direct WebHDFS URLhdfs_urlstringNoDirect WebHDFS/httpFS base (skips Knox), e.g. https://httpfs.cluster:14000 or http://namenode:9870. Use with Kerberos.
Knox Gateway URLknox_urlstringNoKnox gateway base, e.g. https://console-cdp.apps.pvc-prod.salud.madrid.org (used when hdfs_url is not set).
Knox Topologyknox_topologystringNocdp-proxy-apiWebHDFS topology under the Knox gateway (default cdp-proxy-api).
Skip TLS Verifytls_skip_verifybooleanNofalseSkip TLS certificate verification (internal / self-signed cert, e.g. SERMAS CA).

Kerberos​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Kerberos Principalkerberos_principalstringNoe.g. ufpandoraradiologia@SALUD.MADRID.ORG
Kerberos Realmkerberos_realmstringNoe.g. SALUD.MADRID.ORG (defaults to the principal's realm).
KDC (host:port)kerberos_kdcstringNoA KDC address, e.g. dc01.salud.madrid.org:88.
Keytab (base64)kerberos_keytab_base64passwordNobase64-encoded keytab for the principal. Presence of this enables Kerberos/SPNEGO auth.
Service SPN (override)kerberos_spnstringNoOverride the SPNEGO service principal (default HTTP/<endpoint-host>).
Enctype pinkerberos_enctypesstringNoOptional space-separated enctype pin, e.g. rc4-hmac — when an AD keytab only carries RC4 but the KDC advertises AES first.

Errors​

CodeDescription
ENDPOINT_REQUIREDNeither hdfs_url nor knox_url configured

Runtime notes​

  • Single-file mode removes the local file_path once the upload succeeds (delete_after_put, default true): the copy is redundant once HDFS has it, and a multi-GB export left behind fills the disk. A FAILED upload never deletes, so the file stays to retry from.
  • Kerberos mode skips Knox — point hdfs_url at the cluster httpFS (:14000) or NameNode WebHDFS (:9870) and supply the keytab.
  • Batch mode reads each de-id instance's spilled file_path (same container FS) and PUTs it under hdfs_path.
  • Batch mode also writes a per-study _series.json index (write_series_manifest, default true). The layout is flat -- <study>/<sop>.dcm -- so the series level exists nowhere in the paths; rebuilding it later would mean one ranged read per instance through a gateway that serialises them per user. An instance whose upload failed is left out, and a study retrieved without per-series UIDs gets no index at all rather than an empty one.

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Etwas unklar? Senden Sie Feedback zur Dokumentation per E-Mail..