Zum Hauptinhalt springen

Catalog Profile

Category: Analytics
Node id: estate-profile
Maturity: beta
Risk tier: none

Compute per-column statistics from sampled rows: null ratio, distinct count, min/max, percentiles, dominant value shapes and placeholder detection. Emits statistics only — no sampled value is ever carried out.

What it does​

  • Kind inference independent of the source's declared type
  • Abstracted value shapes instead of values
  • Placeholder detection (1900-01-01, 9999, UNKNOWN)
  • Sampled versus exact recorded on every result

Use cases​

  • Profile a hospital table before proposing a FHIR mapping
  • Detect that a birth-date column carries three different formats
  • Find columns that are mostly null before relying on them

Runtime​

  • Performance tier: medium

Input ports​

NameTypeRequiredDescription
recordsjsonYesRows to profile: an array of objects keyed by column name

Output ports​

NameTypeRequiredDescription
profilesjsonNoOne profile per column: kind, null ratio, distinct, statistics, shapes and sentinels
recordsjsonNoThe input rows, passed through so a downstream step can still use them
column_countnumberNoColumns profiled

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
PercentilespercentilesstringNo0.25,0.5,0.75Comma-separated percentiles in [0,1] to compute for numeric columns.
Top Shapestop_shapesnumberNo5How many distinct value shapes to report per column. Shapes are abstracted (\d{2}/\d{2}/\d{4}), never values.

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Sentinel Thresholdsentinel_min_rationumberNo0.02Share of non-missing values a repeated placeholder must reach before it is reported. Higher means fewer false accusations about the data.
Rows Are CompleteexactbooleanNofalseDeclare that the input is the whole column rather than a sample. Affects how distinct counts may be read; leave off unless it is genuinely exhaustive.

Errors​

CodeDescription
no_recordsThe input carried no rows to profile
bad_recordsThe input was not an array of objects

Runtime notes​

  • Profiles the rows on its input port; it does not read the source again, so it adds no load to the system being catalogued.
  • Emits statistics and abstracted shapes only — no sampled value leaves this node, which is what makes the output safe to store and to send to a model.
  • Kind is inferred from the values, not from the source's declared type: a HIS that types everything as VARCHAR tells you nothing.
  • A two-valued 0/1 column is reported as boolean, because calling it numeric produces a mean that means nothing.
  • Every result records whether it came from a sample or the whole column, so a sampled figure is never read as a census.

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Etwas unklar? Senden Sie Feedback zur Dokumentation per E-Mail..