Google Cloud Storage
Category: Cloud Storage
Node id: gcs-connector
Read and write objects in Google Cloud Storage buckets.
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
records | json | No | Records to send when operation=write; ignored by read-only drivers |
service_params | json | No | Parameters for operation=service; merged over the service_params config, so a pipeline can page a sweep by feeding back a cursor |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
records | json | No | Records returned by the read operation; the record shape depends on the connector driver |
count | number | No | Number of records read (operation=read) |
cursor | string | No | Continuation cursor (operation=read) |
done | boolean | No | Whether reading has finished (operation=read); with paginate, true only when the source's last page was read |
pages | number | No | Pages read (operation=read with paginate) |
bytes | number | No | JSON-encoded size of the records returned (operation=read with paginate) |
truncated | boolean | No | A cap or a source defect stopped the walk before the last page (operation=read with paginate); the records are NOT the complete set |
truncated_reason | string | No | Why the walk stopped early: max_pages, max_items, max_page_bytes, timeout, cursor_repeated or no_cursor |
cursor_skip | number | No | Records at the start of the page cursor fetches that were already returned; skip them when resuming |
written | number | No | Number of records written (operation=write) |
errors | json | No | Per-record failures (operation=write); inspect before acknowledging completion |
service_id | string | No | Invoked service identifier (operation=service) |
result | json | No | Result of operation=service; the shape depends on the service (e.g. schema_explorer returns schemas, or a page with next_cursor when limit is set) |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Connector | connector_id | connector-select | No | Use a saved connector for the connection + credentials. Leave empty to configure inline below. | ||
| Operation | operation | enum | Yes | read, write, service | read/write move records; service calls one of the driver's named services (e.g. schema_explorer, table_preview, sql_query). | |
| Service | service_id | string | No | Service to call when operation=service. Available: file_stats, list_directory, read_file. | ||
| Service Parameters | service_params | json | No | Parameters for the service, as JSON. May also be supplied on the service_params input port. |
Connection
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| billing_project | billing_project | string | No | Billing project for requester-pays buckets (defaults to project) Ignored when connector_id is set. | ||
| bucket | bucket | string | Yes | GCS bucket name Ignored when connector_id is set. | ||
| endpoint | endpoint | string | No | Custom storage endpoint (e.g. a fake-gcs emulator) Ignored when connector_id is set. | ||
| prefix | prefix | string | No | Object prefix for scoping Ignored when connector_id is set. | ||
| project | project | string | Yes | GCP project ID Ignored when connector_id is set. | ||
| service_account_json | service_account_json | secret | No | Service account key JSON (leave blank for ADC / workload identity) Ignored when connector_id is set. |
Read
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| delimiter | delimiter | string | No | list: groups keys into common prefixes (e.g. /). download, CSV: field separator, auto (default: detects , ; tab |) or one character (tab for TAB) | ||
| empty_as_null | empty_as_null | boolean | No | false | download, CSV/xlsx: empty cells become null instead of "" | |
| encoding | encoding | string | No | utf-8 | download, CSV: utf-8 (default; a BOM is stripped), latin-1 (iso-8859-1) or windows-1252 (cp1252); invalid bytes fail the step | |
| format | format | string | No | auto | download: auto (default: by extension or content type), raw (never parsed), json, ndjson, csv (tsv), xlsx (excel, xlsm) or parquet | |
| header | header | boolean | No | download, CSV/xlsx: false = the file has no header row (columns column_1…column_N); same as header_row 0 | ||
| header_row | header_row | number | No | min 0 | download, CSV/xlsx: 1-based row holding the column names (default 1); rows above it are skipped; 0 = no header | |
| max_bytes | max_bytes | number | No | 67108864 | min 0; max 536870912 | download, CSV/xlsx/Parquet: size of one file in bytes (default 64 MiB, at most 512 MiB; 0 = default); a larger file fails the step. Not the paging cap max_page_bytes |
| max_rows | max_rows | number | No | 100000 | min 0; max 2000000 | download, CSV/xlsx/Parquet: data rows allowed in one file (default 100000, at most 2000000; 0 = default); a larger file fails the step, never truncated |
| name | name | string | No | get: object name (put also takes it) | ||
| normalize_headers | normalize_headers | boolean | No | false | download, CSV/xlsx: column names lower-cased, accents stripped, other characters as _ ("Data Naixement" → data_naixement) | |
| raw_values | raw_values | boolean | No | false | download, xlsx: cells unformatted (date serials, full-precision numbers) instead of as displayed | |
| sheet | sheet | string | No | download, xlsx: sheet name; when empty sheet_index is used | ||
| sheet_index | sheet_index | number | No | 1 | min 1 | download, xlsx: 1-based sheet position (default 1) |
| source_field | source_field | string | No | download, CSV/xlsx/Parquet: add the object's name to every record under this field (e.g. source_file) |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Read Operation | read_operation | string | No | |||
| Write Operation | write_operation | string | No | |||
| Allow Non-Read-Only Service | allow_write_service | boolean | No | false | Required to run a service the driver does not declare read-only. Off by default so a catalogue or discovery step cannot mutate a source system by misconfiguration. | |
| Page Size | limit | number | No | Records per page for operation=read, where the driver supports it (e.g. S3 list MaxKeys). | ||
| Cursor | cursor | string | No | Continuation cursor to start the read from (a previous run's cursor output). | ||
| Read All Pages | paginate | boolean | No | false | operation=read: follow the source's continuation cursor page after page until the last page, bounded by the caps below. Output truncated is true (with truncated_reason) when a cap stopped the walk first. | |
| Max Pages | max_pages | number | No | 100 | min 1; max 10000 | Hard cap on pages per run when paginate is on. |
| Max Items | max_items | number | No | 100000 | min 1; max 1000000 | Hard cap on records per run when paginate is on. |
| Max Page Bytes | max_page_bytes | number | No | 67108864 | min 1; max 268435456 | Hard cap on the JSON-encoded size of the records kept when paginate is on. Not the object-store max_bytes, which caps the size of a downloaded file. |
| Paging Timeout (s) | paging_timeout_seconds | number | No | 120 | min 1; max 3600 | Overall time budget for the walk; when it runs out the pages read so far are returned as truncated. Keep it below the node timeout. |
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
¿Algo no está claro? Envíe sus comentarios sobre la documentación por correo electrónico..