Google Cloud Storage
Category: Cloud Storage
Node id: gcs-connector
Read and write objects in Google Cloud Storage buckets.
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
records | json | No | Records to send when operation=write; ignored by read-only drivers |
service_params | json | No | Parameters for operation=service; merged over the service_params config, so a pipeline can page a sweep by feeding back a cursor |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
records | json | No | Records returned by the read operation; the record shape depends on the connector driver |
count | number | No | Number of records read (operation=read) |
cursor | string | No | Continuation cursor (operation=read) |
done | boolean | No | Whether reading has finished (operation=read); with paginate, true only when the source's last page was read |
pages | number | No | Pages read (operation=read with paginate) |
bytes | number | No | JSON-encoded size of the records returned (operation=read with paginate) |
truncated | boolean | No | A cap or a source defect stopped the walk before the last page (operation=read with paginate); the records are NOT the complete set |
truncated_reason | string | No | Why the walk stopped early: max_pages, max_items, max_page_bytes, timeout, cursor_repeated or no_cursor |
cursor_skip | number | No | Records at the start of the page cursor fetches that were already returned; skip them when resuming |
written | number | No | Number of records written (operation=write) |
errors | json | No | Per-record failures (operation=write); inspect before acknowledging completion |
service_id | string | No | Invoked service identifier (operation=service) |
result | json | No | Result of operation=service; the shape depends on the service (e.g. schema_explorer returns schemas, or a page with next_cursor when limit is set) |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Connector | connector_id | connector-select | No | Use a saved connector for the connection + credentials. Leave empty to configure inline below. | ||
| Operation | operation | enum | Yes | read, write, service | read/write move records; service calls one of the driver's named services (e.g. schema_explorer, table_preview, sql_query). | |
| Service | service_id | string | No | Service to call when operation=service. Available: file_stats, list_directory, read_file. | ||
| Service Parameters | service_params | json | No | Parameters for the service, as JSON. May also be supplied on the service_params input port. |
Connection
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| billing_project | billing_project | string | No | Billing project for requester-pays buckets (defaults to project) Ignored when connector_id is set. | ||
| bucket | bucket | string | Yes | GCS bucket name Ignored when connector_id is set. | ||
| endpoint | endpoint | string | No | Custom storage endpoint (e.g. a fake-gcs emulator) Ignored when connector_id is set. | ||
| prefix | prefix | string | No | Object prefix for scoping Ignored when connector_id is set. | ||
| project | project | string | Yes | GCP project ID Ignored when connector_id is set. | ||
| service_account_json | service_account_json | secret | No | Service account key JSON (leave blank for ADC / workload identity) Ignored when connector_id is set. |
Read
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| delimiter | delimiter | string | No | list: groups keys into common prefixes (e.g. /). download, CSV: field separator, auto (default: detects , ; tab |) or one character (tab for TAB) | ||
| empty_as_null | empty_as_null | boolean | No | false | download, CSV/xlsx: empty cells become null instead of "" | |
| encoding | encoding | string | No | utf-8 | download, CSV: utf-8 (default; a BOM is stripped), latin-1 (iso-8859-1) or windows-1252 (cp1252); invalid bytes fail the step | |
| format | format | string | No | auto | download: auto (default: by extension or content type), raw (never parsed), json, ndjson, csv (tsv), xlsx (excel, xlsm) or parquet | |
| header | header | boolean | No | download, CSV/xlsx: false = the file has no header row (columns column_1…column_N); same as header_row 0 | ||
| header_row | header_row | number | No | min 0 | download, CSV/xlsx: 1-based row holding the column names (default 1); rows above it are skipped; 0 = no header | |
| max_bytes | max_bytes | number | No | 67108864 | min 0; max 536870912 | download, CSV/xlsx/Parquet: size of one file in bytes (default 64 MiB, at most 512 MiB; 0 = default); a larger file fails the step. Not the paging cap max_page_bytes |
| max_rows | max_rows | number | No | 100000 | min 0; max 2000000 | download, CSV/xlsx/Parquet: data rows allowed in one file (default 100000, at most 2000000; 0 = default); a larger file fails the step, never truncated |
| name | name | string | No | get: object name (put also takes it) | ||
| normalize_headers | normalize_headers | boolean | No | false | download, CSV/xlsx: column names lower-cased, accents stripped, other characters as _ ("Data Naixement" → data_naixement) | |
| raw_values | raw_values | boolean | No | false | download, xlsx: cells unformatted (date serials, full-precision numbers) instead of as displayed | |
| sheet | sheet | string | No | download, xlsx: sheet name; when empty sheet_index is used | ||
| sheet_index | sheet_index | number | No | 1 | min 1 | download, xlsx: 1-based sheet position (default 1) |
| source_field | source_field | string | No | download, CSV/xlsx/Parquet: add the object's name to every record under this field (e.g. source_file) |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Read Operation | read_operation | string | No | |||
| Write Operation | write_operation | string | No | |||
| Allow Non-Read-Only Service | allow_write_service | boolean | No | false | Required to run a service the driver does not declare read-only. Off by default so a catalogue or discovery step cannot mutate a source system by misconfiguration. | |
| Page Size | limit | number | No | Records per page for operation=read, where the driver supports it (e.g. S3 list MaxKeys). | ||
| Cursor | cursor | string | No | Continuation cursor to start the read from (a previous run's cursor output). | ||
| Read All Pages | paginate | boolean | No | false | operation=read: follow the source's continuation cursor page after page until the last page, bounded by the caps below. Output truncated is true (with truncated_reason) when a cap stopped the walk first. | |
| Max Pages | max_pages | number | No | 100 | min 1; max 10000 | Hard cap on pages per run when paginate is on. |
| Max Items | max_items | number | No | 100000 | min 1; max 1000000 | Hard cap on records per run when paginate is on. |
| Max Page Bytes | max_page_bytes | number | No | 67108864 | min 1; max 268435456 | Hard cap on the JSON-encoded size of the records kept when paginate is on. Not the object-store max_bytes, which caps the size of a downloaded file. |
| Paging Timeout (s) | paging_timeout_seconds | number | No | 120 | min 1; max 3600 | Overall time budget for the walk; when it runs out the pages read so far are returned as truncated. Keep it below the node timeout. |
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
Something unclear? Send documentation feedback by email.