Saltar al contenido principal

Catalog Profile

Category: Analytics
Node id: estate-profile
Maturity: beta
Risk tier: none

Compute per-column statistics from sampled rows: null ratio, distinct count, min/max, percentiles, dominant value shapes and placeholder detection. Emits statistics only — no sampled value is ever carried out.

What it does​

  • Kind inference independent of the source's declared type
  • Abstracted value shapes instead of values
  • Placeholder detection (1900-01-01, 9999, UNKNOWN)
  • Sampled versus exact recorded on every result

Use cases​

  • Profile a hospital table before proposing a FHIR mapping
  • Detect that a birth-date column carries three different formats
  • Find columns that are mostly null before relying on them

Runtime​

  • Performance tier: medium

Input ports​

NameTypeRequiredDescription
recordsjsonYesRows to profile: an array of objects keyed by column name

Output ports​

NameTypeRequiredDescription
profilesjsonNoOne profile per column: kind, null ratio, distinct, statistics, shapes and sentinels
recordsjsonNoThe input rows, passed through so a downstream step can still use them
column_countnumberNoColumns profiled

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
PercentilespercentilesstringNo0.25,0.5,0.75Comma-separated percentiles in [0,1] to compute for numeric columns.
Top Shapestop_shapesnumberNo5How many distinct value shapes to report per column. Shapes are abstracted (\d{2}/\d{2}/\d{4}), never values.

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Sentinel Thresholdsentinel_min_rationumberNo0.02Share of non-missing values a repeated placeholder must reach before it is reported. Higher means fewer false accusations about the data.
Rows Are CompleteexactbooleanNofalseDeclare that the input is the whole column rather than a sample. Affects how distinct counts may be read; leave off unless it is genuinely exhaustive.

Errors​

CodeDescription
no_recordsThe input carried no rows to profile
bad_recordsThe input was not an array of objects

Runtime notes​

  • Profiles the rows on its input port; it does not read the source again, so it adds no load to the system being catalogued.
  • Emits statistics and abstracted shapes only — no sampled value leaves this node, which is what makes the output safe to store and to send to a model.
  • Kind is inferred from the values, not from the source's declared type: a HIS that types everything as VARCHAR tells you nothing.
  • A two-valued 0/1 column is reported as boolean, because calling it numeric produces a mean that means nothing.
  • Every result records whether it came from a sample or the whole column, so a sampled figure is never read as a census.

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

¿Algo no está claro? Envíe sus comentarios sobre la documentación por correo electrónico..