Amazon S3
Category: Cloud Storage
Node id: s3-connector
Connect to an S3 bucket: list, upload, download, delete, and rename objects. Supports MinIO and LocalStack.
Input ports
| Name | Type | Required | Description |
|---|---|---|---|
records | json | No | Records to send when operation=write; ignored by read-only drivers |
service_params | json | No | Parameters for operation=service; merged over the service_params config, so a pipeline can page a sweep by feeding back a cursor |
Output ports
| Name | Type | Required | Description |
|---|---|---|---|
records | json | No | Records returned by the read operation; the record shape depends on the connector driver |
count | number | No | Number of records read (operation=read) |
cursor | string | No | Continuation cursor (operation=read) |
done | boolean | No | Whether reading has finished (operation=read); with paginate, true only when the source's last page was read |
pages | number | No | Pages read (operation=read with paginate) |
bytes | number | No | JSON-encoded size of the records returned (operation=read with paginate) |
truncated | boolean | No | A cap or a source defect stopped the walk before the last page (operation=read with paginate); the records are NOT the complete set |
truncated_reason | string | No | Why the walk stopped early: max_pages, max_items, max_page_bytes, timeout, cursor_repeated or no_cursor |
cursor_skip | number | No | Records at the start of the page cursor fetches that were already returned; skip them when resuming |
written | number | No | Number of records written (operation=write) |
errors | json | No | Per-record failures (operation=write); inspect before acknowledging completion |
service_id | string | No | Invoked service identifier (operation=service) |
result | json | No | Result of operation=service; the shape depends on the service (e.g. schema_explorer returns schemas, or a page with next_cursor when limit is set) |
Settings
Basic
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Connector | connector_id | connector-select | No | Use a saved connector for the connection + credentials. Leave empty to configure inline below. | ||
| Operation | operation | enum | Yes | read, write, service | read/write move records; service calls one of the driver's named services (e.g. schema_explorer, table_preview, sql_query). | |
| Service | service_id | string | No | Service to call when operation=service. Available: file_stats, list_directory, read_file. | ||
| Service Parameters | service_params | json | No | Parameters for the service, as JSON. May also be supplied on the service_params input port. |
Connection
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| access_key | access_key | string | No | AWS access key ID (leave blank for IAM role / instance profile) Ignored when connector_id is set. | ||
| acl | acl | enum | No | "", private, public-read, public-read-write, authenticated-read, bucket-owner-read, bucket-owner-full-control | Canned ACL applied to uploaded objects Ignored when connector_id is set. | |
| bucket | bucket | string | Yes | S3 bucket name Ignored when connector_id is set. | ||
| ca_cert | ca_cert | string | No | Custom CA bundle (PEM content or file path) the server certificate must chain to, for private/self-signed TLS. Adds trust; verification stays on Ignored when connector_id is set. | ||
| default_storage_class | default_storage_class | enum | No | "", STANDARD, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, GLACIER_IR | Storage class for uploads Ignored when connector_id is set. | |
| endpoint | endpoint | string | No | Custom endpoint URL (MinIO, LocalStack, R2) Ignored when connector_id is set. | ||
| force_path_style | force_path_style | boolean | No | false | Use path-style URLs (required for MinIO) Ignored when connector_id is set. | |
| prefix | prefix | string | No | Key prefix — scopes all operations to this path Ignored when connector_id is set. | ||
| region | region | string | Yes | AWS region (e.g. eu-west-1) Ignored when connector_id is set. | ||
| secret_key | secret_key | secret | No | AWS secret access key Ignored when connector_id is set. | ||
| session_token | session_token | secret | No | Optional session token for temporary credentials Ignored when connector_id is set. | ||
| sse | sse | enum | No | "", AES256, aws:kms | Server-side encryption for uploads Ignored when connector_id is set. | |
| sse_kms_key_id | sse_kms_key_id | string | No | KMS key ID/ARN (when sse = aws:kms) Ignored when connector_id is set. | ||
| use_accelerate | use_accelerate | boolean | No | Use S3 Transfer Acceleration endpoint Ignored when connector_id is set. |
Read
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| delimiter | delimiter | string | No | list: groups keys into common prefixes (e.g. /). download, CSV: field separator, auto (default: detects , ; tab |) or one character (tab for TAB) | ||
| empty_as_null | empty_as_null | boolean | No | false | download, CSV/xlsx: empty cells become null instead of "" | |
| encoding | encoding | string | No | utf-8 | download, CSV: utf-8 (default; a BOM is stripped), latin-1 (iso-8859-1) or windows-1252 (cp1252); invalid bytes fail the step | |
| format | format | string | No | auto | download: auto (default: by extension or content type), raw (never parsed), json, ndjson, csv (tsv), xlsx (excel, xlsm) or parquet | |
| header | header | boolean | No | download, CSV/xlsx: false = the file has no header row (columns column_1…column_N); same as header_row 0 | ||
| header_row | header_row | number | No | min 0 | download, CSV/xlsx: 1-based row holding the column names (default 1); rows above it are skipped; 0 = no header | |
| key | key | string | No | download, head: object key (the write operations also take it) | ||
| max_bytes | max_bytes | number | No | 67108864 | min 0; max 536870912 | download, CSV/xlsx/Parquet: size of one file in bytes (default 64 MiB, at most 512 MiB; 0 = default); a larger file fails the step. Not the paging cap max_page_bytes |
| max_rows | max_rows | number | No | 100000 | min 0; max 2000000 | download, CSV/xlsx/Parquet: data rows allowed in one file (default 100000, at most 2000000; 0 = default); a larger file fails the step, never truncated |
| normalize_headers | normalize_headers | boolean | No | false | download, CSV/xlsx: column names lower-cased, accents stripped, other characters as _ ("Data Naixement" → data_naixement) | |
| raw_values | raw_values | boolean | No | false | download, xlsx: cells unformatted (date serials, full-precision numbers) instead of as displayed | |
| sheet | sheet | string | No | download, xlsx: sheet name; when empty sheet_index is used | ||
| sheet_index | sheet_index | number | No | 1 | min 1 | download, xlsx: 1-based sheet position (default 1) |
| source_field | source_field | string | No | download, CSV/xlsx/Parquet: add the object's name to every record under this field (e.g. source_file) |
Advanced
| Setting | Name | Type | Required | Default | Allowed values | Description |
|---|---|---|---|---|---|---|
| Read Operation | read_operation | string | No | |||
| Write Operation | write_operation | string | No | |||
| Allow Non-Read-Only Service | allow_write_service | boolean | No | false | Required to run a service the driver does not declare read-only. Off by default so a catalogue or discovery step cannot mutate a source system by misconfiguration. | |
| Page Size | limit | number | No | Records per page for operation=read, where the driver supports it (e.g. S3 list MaxKeys). | ||
| Cursor | cursor | string | No | Continuation cursor to start the read from (a previous run's cursor output). | ||
| Read All Pages | paginate | boolean | No | false | operation=read: follow the source's continuation cursor page after page until the last page, bounded by the caps below. Output truncated is true (with truncated_reason) when a cap stopped the walk first. | |
| Max Pages | max_pages | number | No | 100 | min 1; max 10000 | Hard cap on pages per run when paginate is on. |
| Max Items | max_items | number | No | 100000 | min 1; max 1000000 | Hard cap on records per run when paginate is on. |
| Max Page Bytes | max_page_bytes | number | No | 67108864 | min 1; max 268435456 | Hard cap on the JSON-encoded size of the records kept when paginate is on. Not the object-store max_bytes, which caps the size of a downloaded file. |
| Paging Timeout (s) | paging_timeout_seconds | number | No | 120 | min 1; max 3600 | Overall time budget for the walk; when it runs out the pages read so far are returned as truncated. Keep it below the node timeout. |
See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.
¿Algo no está claro? Envíe sus comentarios sobre la documentación por correo electrónico..