AWS S3 - Goldsky Docs

The files are created in Parquet format.

Pipeline configuration

{
  "sources": [],
  "transforms": [],
  "sinks": [
    {
      "description": "Type.Optional(Type.String())",
      "type": "file",
      "sourceStreamName": "Type.String()",
      "secretName": "Type.String()",
      "path": "Type.String()",
      "partitionColumns": "Type.Optional(Type.String())"
    }
  ]
}

Secrets

Create an AWS S3 secret with the following CLI command:

goldsky secret create AN_AWS_S3_SECRET --type s3 --value '{
  "accessKeyId": "Type.String()",
  "secretAccessKey": "Type.String()",
  "region": "Type.String()"
}'