Amazon S3
Category: Cloud Storage
Node id: s3-connector
Connect to an S3 bucket: list, upload, download, delete, and rename objects. Supports MinIO and LocalStack.
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
records | json | No | Records to send when operation=write; ignored by read-only drivers |
service_params | json | No | Parameters for operation=service; merged over the service_params config, so a pipeline can page a sweep by feeding back a cursor |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
records | json | No | Records returned by the read operation; the record shape depends on the connector driver |
count | number | No | Number of records read (operation=read) |
cursor | string | No | Continuation cursor (operation=read) |
done | boolean | No | Whether reading has finished (operation=read); with paginate, true only when the source's last page was read |
pages | number | No | Pages read (operation=read with paginate) |
bytes | number | No | JSON-encoded size of the records returned (operation=read with paginate) |
truncated | boolean | No | A cap or a source defect stopped the walk before the last page (operation=read with paginate); the records are NOT the complete set |
truncated_reason | string | No | Why the walk stopped early: max_pages, max_items, max_page_bytes, timeout, cursor_repeated or no_cursor |
cursor_skip | number | No | Records at the start of the page cursor fetches that were already returned; skip them when resuming |
written | number | No | Number of records written (operation=write) |
errors | json | No | Per-record failures (operation=write); inspect before acknowledging completion |
service_id | string | No | Invoked service identifier (operation=service) |
result | json | No | Result of operation=service; the shape depends on the service (e.g. schema_explorer returns schemas, or a page with next_cursor when limit is set) |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Connector | connector_id | connector-select | No | Use a saved connector for the connection + credentials. Leave empty to configure inline below. | ||
| Operation | operation | enum | Yes | read, write, service | read/write move records; service calls one of the driver's named services (e.g. schema_explorer, table_preview, sql_query). | |
| Service | service_id | string | No | Service to call when operation=service. Available: file_stats, list_directory, read_file. | ||
| Service Parameters | service_params | json | No | Parameters for the service, as JSON. May also be supplied on the service_params input port. |
Connection
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| access_key | access_key | string | No | AWS access key ID (leave blank for IAM role / instance profile) Ignored when connector_id is set. | ||
| acl | acl | enum | No | "", private, public-read, public-read-write, authenticated-read, bucket-owner-read, bucket-owner-full-control | Canned ACL applied to uploaded objects Ignored when connector_id is set. | |
| bucket | bucket | string | Yes | S3 bucket name Ignored when connector_id is set. | ||
| ca_cert | ca_cert | string | No | Custom CA bundle (PEM content or file path) the server certificate must chain to, for private/self-signed TLS. Adds trust; verification stays on Ignored when connector_id is set. | ||
| default_storage_class | default_storage_class | enum | No | "", STANDARD, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, GLACIER_IR | Storage class for uploads Ignored when connector_id is set. | |
| endpoint | endpoint | string | No | Custom endpoint URL (MinIO, LocalStack, R2) Ignored when connector_id is set. | ||
| force_path_style | force_path_style | boolean | No | false | Use path-style URLs (required for MinIO) Ignored when connector_id is set. | |
| prefix | prefix | string | No | Key prefix — scopes all operations to this path Ignored when connector_id is set. | ||
| region | region | string | Yes | AWS region (e.g. eu-west-1) Ignored when connector_id is set. | ||
| secret_key | secret_key | secret | No | AWS secret access key Ignored when connector_id is set. | ||
| session_token | session_token | secret | No | Optional session token for temporary credentials Ignored when connector_id is set. | ||
| sse | sse | enum | No | "", AES256, aws:kms | Server-side encryption for uploads Ignored when connector_id is set. | |
| sse_kms_key_id | sse_kms_key_id | string | No | KMS key ID/ARN (when sse = aws:kms) Ignored when connector_id is set. | ||
| use_accelerate | use_accelerate | boolean | No | Use S3 Transfer Acceleration endpoint Ignored when connector_id is set. |
Read
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| delimiter | delimiter | string | No | list: groups keys into common prefixes (e.g. /). download, CSV: field separator, auto (default: detects , ; tab |) or one character (tab for TAB) | ||
| empty_as_null | empty_as_null | boolean | No | false | download, CSV/xlsx: empty cells become null instead of "" | |
| encoding | encoding | string | No | utf-8 | download, CSV: utf-8 (default; a BOM is stripped), latin-1 (iso-8859-1) or windows-1252 (cp1252); invalid bytes fail the step | |
| format | format | string | No | auto | download: auto (default: by extension or content type), raw (never parsed), json, ndjson, csv (tsv), xlsx (excel, xlsm) or parquet | |
| header | header | boolean | No | download, CSV/xlsx: false = the file has no header row (columns column_1…column_N); same as header_row 0 | ||
| header_row | header_row | number | No | min 0 | download, CSV/xlsx: 1-based row holding the column names (default 1); rows above it are skipped; 0 = no header | |
| key | key | string | No | download, head: object key (the write operations also take it) | ||
| max_bytes | max_bytes | number | No | 67108864 | min 0; max 536870912 | download, CSV/xlsx/Parquet: size of one file in bytes (default 64 MiB, at most 512 MiB; 0 = default); a larger file fails the step. Not the paging cap max_page_bytes |
| max_rows | max_rows | number | No | 100000 | min 0; max 2000000 | download, CSV/xlsx/Parquet: data rows allowed in one file (default 100000, at most 2000000; 0 = default); a larger file fails the step, never truncated |
| normalize_headers | normalize_headers | boolean | No | false | download, CSV/xlsx: column names lower-cased, accents stripped, other characters as _ ("Data Naixement" → data_naixement) | |
| raw_values | raw_values | boolean | No | false | download, xlsx: cells unformatted (date serials, full-precision numbers) instead of as displayed | |
| sheet | sheet | string | No | download, xlsx: sheet name; when empty sheet_index is used | ||
| sheet_index | sheet_index | number | No | 1 | min 1 | download, xlsx: 1-based sheet position (default 1) |
| source_field | source_field | string | No | download, CSV/xlsx/Parquet: add the object's name to every record under this field (e.g. source_file) |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Read Operation | read_operation | string | No | |||
| Write Operation | write_operation | string | No | |||
| Allow Non-Read-Only Service | allow_write_service | boolean | No | false | Required to run a service the driver does not declare read-only. Off by default so a catalogue or discovery step cannot mutate a source system by misconfiguration. | |
| Page Size | limit | number | No | Records per page for operation=read, where the driver supports it (e.g. S3 list MaxKeys). | ||
| Cursor | cursor | string | No | Continuation cursor to start the read from (a previous run's cursor output). | ||
| Read All Pages | paginate | boolean | No | false | operation=read: follow the source's continuation cursor page after page until the last page, bounded by the caps below. Output truncated is true (with truncated_reason) when a cap stopped the walk first. | |
| Max Pages | max_pages | number | No | 100 | min 1; max 10000 | Hard cap on pages per run when paginate is on. |
| Max Items | max_items | number | No | 100000 | min 1; max 1000000 | Hard cap on records per run when paginate is on. |
| Max Page Bytes | max_page_bytes | number | No | 67108864 | min 1; max 268435456 | Hard cap on the JSON-encoded size of the records kept when paginate is on. Not the object-store max_bytes, which caps the size of a downloaded file. |
| Paging Timeout (s) | paging_timeout_seconds | number | No | 120 | min 1; max 3600 | Overall time budget for the walk; when it runs out the pages read so far are returned as truncated. Keep it below the node timeout. |
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
Etwas unklar? Senden Sie Feedback zur Dokumentation per E-Mail..