Zum Hauptinhalt springen

Google Cloud Storage

Category: Cloud Storage
Node id: gcs-connector

Read and write objects in Google Cloud Storage buckets.

Input ports​

NameTypeRequiredDescription
recordsjsonNoRecords to send when operation=write; ignored by read-only drivers
service_paramsjsonNoParameters for operation=service; merged over the service_params config, so a pipeline can page a sweep by feeding back a cursor

Output ports​

NameTypeRequiredDescription
recordsjsonNoRecords returned by the read operation; the record shape depends on the connector driver
countnumberNoNumber of records read (operation=read)
cursorstringNoContinuation cursor (operation=read)
donebooleanNoWhether reading has finished (operation=read); with paginate, true only when the source's last page was read
pagesnumberNoPages read (operation=read with paginate)
bytesnumberNoJSON-encoded size of the records returned (operation=read with paginate)
truncatedbooleanNoA cap or a source defect stopped the walk before the last page (operation=read with paginate); the records are NOT the complete set
truncated_reasonstringNoWhy the walk stopped early: max_pages, max_items, max_page_bytes, timeout, cursor_repeated or no_cursor
cursor_skipnumberNoRecords at the start of the page cursor fetches that were already returned; skip them when resuming
writtennumberNoNumber of records written (operation=write)
errorsjsonNoPer-record failures (operation=write); inspect before acknowledging completion
service_idstringNoInvoked service identifier (operation=service)
resultjsonNoResult of operation=service; the shape depends on the service (e.g. schema_explorer returns schemas, or a page with next_cursor when limit is set)

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Connectorconnector_idconnector-selectNoUse a saved connector for the connection + credentials. Leave empty to configure inline below.
OperationoperationenumYesread, write, serviceread/write move records; service calls one of the driver's named services (e.g. schema_explorer, table_preview, sql_query).
Serviceservice_idstringNoService to call when operation=service. Available: file_stats, list_directory, read_file.
Service Parametersservice_paramsjsonNoParameters for the service, as JSON. May also be supplied on the service_params input port.

Connection​

SettingNameTypeRequiredDefaultAllowed valuesDescription
billing_projectbilling_projectstringNoBilling project for requester-pays buckets (defaults to project) Ignored when connector_id is set.
bucketbucketstringYesGCS bucket name Ignored when connector_id is set.
endpointendpointstringNoCustom storage endpoint (e.g. a fake-gcs emulator) Ignored when connector_id is set.
prefixprefixstringNoObject prefix for scoping Ignored when connector_id is set.
projectprojectstringYesGCP project ID Ignored when connector_id is set.
service_account_jsonservice_account_jsonsecretNoService account key JSON (leave blank for ADC / workload identity) Ignored when connector_id is set.

Read​

SettingNameTypeRequiredDefaultAllowed valuesDescription
delimiterdelimiterstringNolist: groups keys into common prefixes (e.g. /). download, CSV: field separator, auto (default: detects , ; tab |) or one character (tab for TAB)
empty_as_nullempty_as_nullbooleanNofalsedownload, CSV/xlsx: empty cells become null instead of ""
encodingencodingstringNoutf-8download, CSV: utf-8 (default; a BOM is stripped), latin-1 (iso-8859-1) or windows-1252 (cp1252); invalid bytes fail the step
formatformatstringNoautodownload: auto (default: by extension or content type), raw (never parsed), json, ndjson, csv (tsv), xlsx (excel, xlsm) or parquet
headerheaderbooleanNodownload, CSV/xlsx: false = the file has no header row (columns column_1…column_N); same as header_row 0
header_rowheader_rownumberNomin 0download, CSV/xlsx: 1-based row holding the column names (default 1); rows above it are skipped; 0 = no header
max_bytesmax_bytesnumberNo67108864min 0; max 536870912download, CSV/xlsx/Parquet: size of one file in bytes (default 64 MiB, at most 512 MiB; 0 = default); a larger file fails the step. Not the paging cap max_page_bytes
max_rowsmax_rowsnumberNo100000min 0; max 2000000download, CSV/xlsx/Parquet: data rows allowed in one file (default 100000, at most 2000000; 0 = default); a larger file fails the step, never truncated
namenamestringNoget: object name (put also takes it)
normalize_headersnormalize_headersbooleanNofalsedownload, CSV/xlsx: column names lower-cased, accents stripped, other characters as _ ("Data Naixement" → data_naixement)
raw_valuesraw_valuesbooleanNofalsedownload, xlsx: cells unformatted (date serials, full-precision numbers) instead of as displayed
sheetsheetstringNodownload, xlsx: sheet name; when empty sheet_index is used
sheet_indexsheet_indexnumberNo1min 1download, xlsx: 1-based sheet position (default 1)
source_fieldsource_fieldstringNodownload, CSV/xlsx/Parquet: add the object's name to every record under this field (e.g. source_file)

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Read Operationread_operationstringNo
Write Operationwrite_operationstringNo
Allow Non-Read-Only Serviceallow_write_servicebooleanNofalseRequired to run a service the driver does not declare read-only. Off by default so a catalogue or discovery step cannot mutate a source system by misconfiguration.
Page SizelimitnumberNoRecords per page for operation=read, where the driver supports it (e.g. S3 list MaxKeys).
CursorcursorstringNoContinuation cursor to start the read from (a previous run's cursor output).
Read All PagespaginatebooleanNofalseoperation=read: follow the source's continuation cursor page after page until the last page, bounded by the caps below. Output truncated is true (with truncated_reason) when a cap stopped the walk first.
Max Pagesmax_pagesnumberNo100min 1; max 10000Hard cap on pages per run when paginate is on.
Max Itemsmax_itemsnumberNo100000min 1; max 1000000Hard cap on records per run when paginate is on.
Max Page Bytesmax_page_bytesnumberNo67108864min 1; max 268435456Hard cap on the JSON-encoded size of the records kept when paginate is on. Not the object-store max_bytes, which caps the size of a downloaded file.
Paging Timeout (s)paging_timeout_secondsnumberNo120min 1; max 3600Overall time budget for the walk; when it runs out the pages read so far are returned as truncated. Keep it below the node timeout.

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Etwas unklar? Senden Sie Feedback zur Dokumentation per E-Mail..