Cloud

CREATE ICEBERG CATALOG

The CREATE ICEBERG CATALOG statement creates a named connection to an Iceberg REST catalog. Link the Iceberg catalog to a Redpanda catalog with USING CATALOG so that queries against the linked Redpanda catalog return both live and Iceberg-committed records. Standalone querying against an Iceberg catalog is not supported. See Query Iceberg-enabled Topics for the end-to-end workflow.

The statement requires an existing storage connection that holds the object-storage credentials for the Iceberg warehouse.

In Redpanda Cloud, the storage connection and Iceberg catalog used to query Iceberg-enabled topics are configured and managed automatically when you enable Redpanda SQL on a cluster that has an Iceberg REST catalog configured. You don’t run these statements yourself for that workflow. This page documents the statement’s syntax and options so you can understand what Redpanda Cloud configures. See Query Iceberg-enabled topics.

Syntax

CREATE ICEBERG CATALOG [IF NOT EXISTS] catalog_name STORAGE storage_name
  WITH (option = 'value' [, ...]);

  • catalog_name: Name for the new Iceberg catalog.

  • IF NOT EXISTS: Optional. Prevents an error if an Iceberg catalog with the same name already exists.

  • storage_name: Name of an existing storage connection. Create it first with CREATE STORAGE.

Catalogs are created in the current schema (public by default). To create a catalog in a different schema, qualify the name as schema.catalog_name. The auto-created default_iceberg_catalog is in public.

Options

Option Type Required Description

uri

STRING

Yes

REST catalog endpoint URI.

warehouse

STRING

No

Iceberg warehouse identifier or location.

auth_type

STRING

No

Authentication type for the REST catalog. One of oauth2, basic, aws_sigv4, or gcp. If omitted, the catalog connects without authentication. Providing an auth-specific option (such as username, aws_region, or gcp_project_id) without auth_type is rejected.

gcp_project_id

STRING

Required when auth_type = 'gcp'

GCP project that BigLake API requests are billed and quota-attributed to.

oauth2_client_id

STRING

Required when auth_type = 'oauth2'

OAuth2 client ID.

oauth2_client_secret

STRING

Required when auth_type = 'oauth2'

OAuth2 client secret.

oauth2_scope

STRING

No

OAuth2 scope to request.

oauth2_token_endpoint_url

STRING

No

OAuth2 token endpoint URL. Use to override the catalog’s default token endpoint.

oauth2_token_refresh_margin_seconds

INTEGER

No

Number of seconds before token expiry to refresh. Must be between 0 and 2147483647.

username

STRING

Required when auth_type = 'basic'

Basic authentication username.

password

STRING

Required when auth_type = 'basic'

Basic authentication password.

aws_region

STRING

Required when auth_type = 'aws_sigv4'

AWS region for SigV4 request signing (for example, us-west-2).

aws_access_key_id

STRING

No

AWS access key ID for SigV4 signing. Must be set together with aws_secret_access_key. If both are omitted, the catalog uses the AWS default credential chain (environment variables, shared config, STS web identity, IMDSv2/ECS).

aws_secret_access_key

STRING

No

AWS secret access key for SigV4 signing. See aws_access_key_id for credential-chain behavior.

ssl_verify

STRING

No

'true' (default) or 'false'. Whether to verify the REST catalog’s TLS certificate.

ssl_ca_info

STRING

No

Path to a CA certificate file used to verify the REST catalog’s TLS certificate.

ssl_ca_path

STRING

No

Path to a directory containing CA certificates.

ssl_crl_file

STRING

No

Path to a certificate revocation list (CRL) file.

flat_namespaces

STRING

No

'true' or 'false'. Declares whether the backing catalog is flat (that is, it does not support nested or multi-level namespaces).

  • When 'true', Redpanda SQL skips recursive child-namespace calls for any non-root namespace.

  • If not set, Redpanda SQL auto-derives this as true when auth_type = 'aws_sigv4' and aws_service_name is glue (default) or s3tables, and false for all other auth types. Set it explicitly only if the auto-derived value does not match your backend’s actual behavior.

allowed_namespaces

STRING

No

Comma-separated list of dotted namespace paths (for example, 'analytics, logs.audit'). Scopes catalog discovery to the listed namespaces, skipping the root-level namespace walk. SELECT and REFRESH are gated on the same list; tables outside it return a "namespace not allowed" error. When unset, Redpanda SQL discovers the entire catalog tree.

Validation rules:

  • Surrounding whitespace is trimmed.

  • Dots denote namespace levels (logs.audit = namespace audit under logs).

  • Empty entries (trailing comma) are rejected at DDL time.

  • Overlapping entries (where one entry is a prefix of another) are rejected at DDL time.

  • On flat backends (flat_namespaces = 'true'), all entries must be single-segment.

Examples

Create a basic Iceberg catalog

Connect to a REST catalog without authentication. The catalog uses TLS verification by default.

CREATE ICEBERG CATALOG lakehouse_catalog STORAGE iceberg_storage
  WITH (
    uri = 'https://catalog.example.com',
    warehouse = 's3://warehouse/'
  );

Create an Iceberg catalog with OAuth2 authentication

CREATE ICEBERG CATALOG lakehouse_catalog STORAGE iceberg_storage
  WITH (
    uri = 'https://catalog.example.com',
    warehouse = 's3://lakehouse-data/',
    auth_type = 'oauth2',
    oauth2_client_id = '<client-id>',
    oauth2_client_secret = '<client-secret>',
    oauth2_scope = 'PRINCIPAL_ROLE:ALL',
    oauth2_token_endpoint_url = 'https://auth.example.com/token',
    oauth2_token_refresh_margin_seconds = 300
  );

Create an Iceberg catalog with basic authentication

CREATE ICEBERG CATALOG lakehouse_catalog STORAGE iceberg_storage
  WITH (
    uri = 'https://catalog.example.com',
    warehouse = 's3://warehouse/',
    auth_type = 'basic',
    username = '<username>',
    password = '<password>'
  );

Create an Iceberg catalog with AWS SigV4 authentication

Use for REST catalogs fronted by AWS services (such as AWS Glue).

CREATE ICEBERG CATALOG lakehouse_catalog STORAGE iceberg_storage
  WITH (
    uri = 'https://catalog.example.com',
    warehouse = 's3://warehouse/',
    auth_type = 'aws_sigv4',
    aws_region = 'us-west-2',
    aws_access_key_id = '<access-key-id>',
    aws_secret_access_key = '<secret-access-key>'
  );

To use the AWS default credential chain (for example, an EC2 instance-profile role), omit aws_access_key_id and aws_secret_access_key. They must be set together or omitted together.

Create an Iceberg catalog for AWS Glue or S3 Tables

AWS Glue and S3 Tables are flat-by-specification backends, meaning they do not support nested namespaces. Use allowed_namespaces to scope discovery to specific databases and avoid requiring the glue:GetDatabases IAM permission. The flat_namespaces option is auto-derived as true when auth_type = 'aws_sigv4' and aws_service_name is glue (default) or s3tables.

CREATE ICEBERG CATALOG glue_cat STORAGE iceberg_storage
  WITH (
    uri                = 'https://glue.us-east-1.amazonaws.com/iceberg',
    warehouse          = 's3://my-warehouse/',
    auth_type          = 'aws_sigv4',
    aws_region         = 'us-east-1',
    allowed_namespaces = 'analytics, sales'
    -- flat_namespaces is auto-derived as true for Glue/S3 Tables
  );

Create an Iceberg catalog for BigLake on GCP

Use auth_type = 'gcp' for a Google Cloud BigLake REST catalog. Redpanda SQL authenticates with Application Default Credentials (ADC), which on Google Kubernetes Engine (GKE) resolves to Workload Identity, so no static key is required. The gcp_project_id option is required. Set uri and warehouse to the values for your BigLake catalog.

CREATE ICEBERG CATALOG biglake_cat STORAGE iceberg_storage
  WITH (
    uri            = '<biglake-rest-catalog-uri>',
    warehouse      = '<warehouse>',
    auth_type      = 'gcp',
    gcp_project_id = '<gcp-project-id>'
  );

Create an Iceberg catalog scoped to specific namespaces (nested backend)

For backends that support nested namespaces (such as Polaris), use multi-segment dotted paths in allowed_namespaces.

CREATE ICEBERG CATALOG polaris_cat STORAGE iceberg_storage
  WITH (
    uri                = 'https://polaris.example.com/api/catalog',
    warehouse          = 'my_warehouse',
    auth_type          = 'oauth2',
    oauth2_client_id   = '<client-id>',
    oauth2_client_secret = '<client-secret>',
    allowed_namespaces = 'analytics, logs.audit, raw.ingestion.kafka'
  );

Create an Iceberg catalog with custom TLS settings

CREATE ICEBERG CATALOG lakehouse_catalog STORAGE iceberg_storage
  WITH (
    uri = 'https://catalog.example.com',
    warehouse = 's3://warehouse/',
    ssl_verify = 'true',
    ssl_ca_info = '/etc/ssl/certs/catalog-ca.pem'
  );
Statement Description

ALTER ICEBERG CATALOG

Modify connection properties of an existing Iceberg catalog.

DROP ICEBERG CATALOG

Remove an Iceberg catalog.

CREATE STORAGE

Create the storage connection that backs the Iceberg catalog.

CREATE REDPANDA CATALOG

Create a Redpanda catalog. Use USING CATALOG to link a Redpanda catalog to an Iceberg catalog so that queries return both live and Iceberg-committed records.