Skip to main content

Catalog DB Introspect

Category: Data
Node id: estate-db-introspect
Maturity: beta
Risk tier: none

Enumerate the schemas, tables and columns of any supported database through a saved connector, emitting catalogue assets ready for estate-register. Pages large schemas and reports truncation rather than silently stopping.

What it does​

  • Thirteen SQL dialects including Oracle, SQL Server, HANA and Db2
  • Read-only: every query is gated and run in a read-only transaction
  • Keyset paging with a resumable cursor
  • Explicit truncation reporting

Use cases​

  • Catalogue a hospital information system before an integration project
  • Enumerate one schema of a data warehouse for FHIR mapping
  • Re-run nightly to detect new or removed tables

Runtime​

  • Performance tier: slow

Input ports​

NameTypeRequiredDescription
cursorjsonNoOptional {cursor} from a previous run's next_cursor, to continue a paged sweep

Output ports​

NameTypeRequiredDescription
assetsjsonNoAssets shaped for estate-register: {fqn, kind, system, name, fields[]}
next_cursortextNoOpaque cursor when more tables remain; empty when the sweep is complete
truncatedbooleanNoTrue when more tables remain after this page
table_countnumberNoTables enumerated in this page

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Connectorconnector_idconnector-selectYesSaved database connector supplying the connection and its credentials. Credentials never appear in the pipeline definition.
Schemaschema_filterstringNoRestrict the sweep to one schema. Strongly recommended on a large database: it is the difference between a targeted catalogue and a full crawl.
Tabletable_filterstringNoRestrict the sweep to a single table.
Tables Per Pagepage_sizenumberNo500Maximum tables to enumerate per run. Bounds both the query and the memory one page holds.

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
CursorcursorstringNoContinue a previous sweep. Usually fed from the cursor input port inside a loop rather than set by hand.
Timeouttimeout_secondsnumberNo20Per-page query timeout. Raise it for a slow or heavily loaded source.
System NamesystemstringNoName used in generated FQNs (table://<system>/<schema>.<table>). Defaults to the connector id, which keeps FQNs stable across runs.

Errors​

CodeDescription
connector_unavailableConnector registry not configured on this server
no_connectorconnector_id is missing or does not resolve
unsupported_driverThe connector's driver does not expose schema exploration

Runtime notes​

  • Reads catalogue metadata only — no table rows are read, so it is safe against a production system.
  • Emits assets shaped for estate-register; wire the two together and register before profiling so a failure part-way through loses nothing.
  • When truncated is true, feed next_cursor back into the cursor input to continue; the sweep is complete when truncated is false.
  • FQNs are table://<system>/<schema>.<table> and are stable across runs, which is what makes registration idempotent.
  • Without schema_filter this enumerates every non-system schema, which on a large HIS is a long sweep — set it unless a full crawl is genuinely intended.

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Something unclear? Send documentation feedback by email.