secrets.sql

Access external data without adding infrastructure

Secrets — access external data without adding infrastructure

Phase: Pre-catalog | Order: 2 | Required: No

OndatraSQL runs locally, but your data doesn’t have to.

This file defines how the runtime connects to external systems — cloud storage, databases, and APIs — using DuckDB’s built-in secrets manager.

No separate credential system. No service configuration. Just SQL.

You only need this if you connect to external data sources. Local pipelines (CSV, Parquet, APIs via Starlark) do not require any secrets.

Never commit this file to version control.

Use environment variables (.env) for sensitive values, or rely on credential providers (like AWS credential chain) when available.

Common Use Cases

Cloud storage

-- Use AWS credential chain (recommended)
CREATE SECRET aws_chain (
    TYPE s3,
    PROVIDER credential_chain
);

Explicit credentials (dev/testing)

CREATE SECRET s3_secret (
    TYPE s3,
    KEY_ID '${AWS_ACCESS_KEY_ID}',
    SECRET '${AWS_SECRET_ACCESS_KEY}',
    REGION 'eu-north-1'
);

Scoped credentials

CREATE SECRET prod_bucket (
    TYPE s3,
    KEY_ID '${AWS_ACCESS_KEY_ID}',
    SECRET '${AWS_SECRET_ACCESS_KEY}',
    SCOPE 's3://prod-data/'
);

Databases

CREATE SECRET pg_secret (
    TYPE postgres,
    HOST '${PG_HOST}',
    PORT 5432,
    DATABASE 'warehouse',
    USER 'readonly',
    PASSWORD '${PG_PASSWORD}'
);

Secrets are used by DuckDB itself — not by OndatraSQL. Once defined, they apply automatically to any query that accesses external data.

Supported providers include S3, GCS, R2, Azure, PostgreSQL, and MySQL.