Streaming

Specify Iceberg Schema

This feature requires an enterprise license. To get a trial license key or extend your trial period, generate a new trial license key. To purchase a license, contact Redpanda Sales.

If Redpanda has enterprise features enabled and it cannot find a valid license, restrictions apply.

In Iceberg-enabled clusters, the redpanda.iceberg.mode topic property determines how Redpanda maps topic data to the Iceberg table structure. You can have the generated Iceberg table match the structure of a schema in Schema Registry, or you can use the key_value mode where Redpanda stores the record values as-is in the table.

After reading this page, you will be able to:

Configure the redpanda.iceberg.mode property when you create or update a topic
Choose the Iceberg mode that produces the table structure your data consumers need
Apply independent translation for record keys, values, and headers

Supported Iceberg modes

Redpanda supports the following modes for Iceberg topics:

key_value

Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value.

value_schema_id_prefix

Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. You must register a schema in Schema Registry and producers must write to the topic using the Schema Registry wire format.

In the Schema Registry wire format, a "magic byte" and schema ID are embedded in the message payload header. Producers to the topic must use the wire format in the serialization process so Redpanda can determine the schema used for each record, use the schema to define the Iceberg table, and store the topic values in the corresponding table columns.

value_schema_latest

Creates an Iceberg table whose structure matches the latest schema registered for the subject in Schema Registry. You must register a schema in Schema Registry.

Producers cannot use the wire format in value_schema_latest mode. Redpanda expects the serialized message as-is without the magic byte or schema ID prefix in the record value.

The value_schema_latest mode is not compatible with the rpk topic produce command which embeds the wire format header. You must use your own producer code to produce to topics in value_schema_latest mode.

The latest schema is cached periodically. The cache period is defined by the cluster property iceberg_latest_schema_cache_ttl_ms (default: 5 minutes).

disabled

Default for redpanda.iceberg.mode. Disables writing to an Iceberg table for the topic.

The following modes are compatible with producing to an Iceberg topic using Redpanda Console:

key_value
Starting in version 25.2, value_schema_latest with a JSON schema

Otherwise, records may fail to write to the Iceberg table and instead write to the dead-letter queue.

Configure Iceberg mode for a topic

You can set the Iceberg mode for a topic when you create the topic, or you can update the mode for an existing topic.

Option 1. Create a new topic and set redpanda.iceberg.mode:

rpk topic create <topic-name> --topic-config=redpanda.iceberg.mode=<iceberg-mode>

Option 2. Set redpanda.iceberg.mode for an existing topic:

rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=<iceberg-mode>

Override `value_schema_latest` default

In value_schema_latest mode, you only need to set the property value to the string value_schema_latest. This enables the default behavior of value_schema_latest mode, which determines the subject for the topic using the TopicNameStrategy. For example, if your topic is named sensor the schema is looked up in the sensor-value subject. For Protobuf data, the default behavior also deserializes records using the first message defined in the corresponding Protobuf schema stored in Schema Registry.

If you use a different strategy other than the topic name to derive the subject name, you can override the default behavior of value_schema_latest mode and explicitly set the subject name.

To override the default behavior, use the following optional syntax:

value_schema_latest:subject=<subject-name>,protobuf_name=<protobuf-message-full-name>

For both Avro and Protobuf, specify a different subject name by using the key-value pair subject=<subject-name>, for example value_schema_latest:subject=sensor-data.
For Protobuf only:
- Specify a different message definition by using a key-value pair protobuf_name=<message-full-name>. You must use the fully qualified name, which includes the package name, for example, value_schema_latest:protobuf_name=com.example.manufacturing.SensorData.
- To specify both a different subject and message definition, separate the key-value pairs with a comma, for example: value_schema_latest:subject=my_protobuf_schema,protobuf_name=com.example.manufacturing.SensorData.
If you don’t specify the fully qualified Protobuf message name, Redpanda pauses the data translation to the Iceberg table until you fix the topic misconfiguration.

Configure key, value, and header translation

For Redpanda clusters version 26.2 and later, in addition to the supported modes, redpanda.iceberg.mode also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The key_value, value_schema_id_prefix, and value_schema_latest modes are shorthand for common combinations of these sections (see Iceberg mode shorthands).

The key and headers sections change fields inside the redpanda system struct column (redpanda.key and the value field of each entry in redpanda.headers), while the value section changes the columns outside that struct. See How Iceberg modes translate to table format for the base row structure that every generated table includes.

Use the following syntax to configure one or more sections:

<section>:<option>=<value>,<option>=<value>;<section>:<option>=<value>

Valid sections are key, value, and headers.
Separate sections with ;.
Separate options within a section with ,.
Sections can appear in any order. Any section you omit uses its defaults.

Key and value section options

The key and value sections accept the same options, except for layout, which is available only in the value section.

Option Values Default Notes

Option	Values	Default	Notes
`mode`	`binary`, `schema_id_prefix`, `schema_latest`, `string`	`binary`	`string` stores the field as a UTF-8 string, replacing invalid bytes with the Unicode replacement character (`U+FFFD`). See Resulting `redpanda.key` type by `key` mode and Resulting value field type by `value` mode for the resulting column types in the generated table.
`subject`	Subject name	Empty	Requires `mode=schema_latest`. When empty, Redpanda derives the subject using the `<topic-name>-key` or `<topic-name>-value` naming convention (TopicNameStrategy).
`protobuf_name`	Fully qualified Protobuf message name	Empty	Requires `mode=schema_latest`. When empty, Redpanda uses the first message definition in the schema.
`layout`	`flat`, `nested`	`flat`	Value section only. Requires `mode=schema_id_prefix` or `mode=schema_latest`.

mode

binary, schema_id_prefix, schema_latest, string

binary

string stores the field as a UTF-8 string, replacing invalid bytes with the Unicode replacement character (U+FFFD).

See Resulting redpanda.key type by key mode and Resulting value field type by value mode for the resulting column types in the generated table.

subject

Subject name

Empty

Requires mode=schema_latest. When empty, Redpanda derives the subject using the <topic-name>-key or <topic-name>-value naming convention (TopicNameStrategy).

protobuf_name

Fully qualified Protobuf message name

Empty

Requires mode=schema_latest. When empty, Redpanda uses the first message definition in the schema.

layout

flat, nested

flat

Value section only. Requires mode=schema_id_prefix or mode=schema_latest.

Table 1. Resulting `redpanda.key` type by `key` mode
Mode	`redpanda.key` type	Description
`binary` (default)	`binary`	Raw key bytes; no decoding.
`schema_id_prefix`	`struct` (from the schema)	Decoded using the schema ID embedded in the key, in Schema Registry wire format.
`schema_latest`	`struct` (from the schema)	Decoded using the latest schema registered for the subject.
`string`	`string`	UTF-8 decoded. Invalid bytes are replaced with `U+FFFD`.

For schema_id_prefix and schema_latest, every field in the decoded key struct is optional and an absent field is stored as null, so the table can tolerate schema evolution.

Table 2. Resulting value field type by `value` mode
Mode	Value field type	Description
`binary` (default)	`binary` column named `value`	Raw value bytes (no decoding).
`schema_id_prefix`	Struct fields. Placement depends on `layout`.	Decoded using the schema ID embedded in the value, in Schema Registry wire format.
`schema_latest`	Struct fields. Placement depends on `layout`.	Decoded using the latest schema registered for the subject.
`string`	`string` column named `value`	UTF-8 decoded; invalid bytes are replaced with `U+FFFD`.

Key field examples

The key section supports the same schema-decoding modes as the value section (key:mode=schema_id_prefix and key:mode=schema_latest), but decoded key fields always land in the single redpanda.key field, and keys have no layout option.

For example, given a key schema with fields user_id (int) and region (string):

{
    "type": "record",
    "name": "OrderKey",
    "fields": [
        {
            "name": "user_id",
            "type": "int"
        },
        {
            "name": "region",
            "type": "string"
        }
    ]
}

With the default key:mode=binary, redpanda.key is a single binary field. With key:mode=schema_id_prefix or key:mode=schema_latest, redpanda.key becomes a struct whose fields match the decoded schema:

redpanda struct<
    ...,
    key: struct<
        user_id: int,
        region:  string
    >,
    ...
>

Value field examples

The layout option (value section only) controls where decoded value fields appear as columns. By default (layout=flat), Redpanda places each decoded value field as a top-level column in the generated table, alongside the redpanda system struct. If a decoded value field is named redpanda, Redpanda moves it into the redpanda system struct as a data field, to avoid colliding with the record metadata column of the same name.

Set layout=nested to nest all decoded value fields inside a single value struct column instead. A field named redpanda stays nested under the value column and is unaffected.

For example, using a value schema with fields user_id (int) and region (string), the default layout=flat promotes those fields to top-level columns:

redpanda struct<
    ...
>,
user_id: int,
region:  string

You then query the fields as top-level columns, for example SELECT user_id, region FROM orders.

With layout=nested, the same fields are wrapped inside a single value struct column instead:

redpanda struct<
    ...
>,
value: struct<
    user_id: int,
    region:  string
>

You then query the fields through the value struct, for example SELECT value.user_id, value.region FROM orders.

Headers section options

The headers section accepts a single option, value_type, which controls how header values are stored in the generated table.

value_type Header value type Description

`value_type`	Header value type	Description
`binary` (default)	`binary`	Raw header value bytes (no decoding).
`string`	`string`	UTF-8 decoded. Invalid bytes are replaced with `U+FFFD`.

binary (default)

binary

Raw header value bytes (no decoding).

string

UTF-8 decoded. Invalid bytes are replaced with U+FFFD.

Only header values are affected by value_type. Header keys are always stored as strings.

Header value examples

Consider a record with two headers: content-type with the value application/json (valid UTF-8), and trace-id with the invalid UTF-8 byte sequence 0xDEADBEEF.

With the default headers:value_type=binary, both header values are stored as raw bytes:

redpanda.headers = [
    {key: "content-type", value: b"application/json"},
    {key: "trace-id",     value: b"\xDE\xAD\xBE\xEF"}
]

With headers:value_type=string, both values are decoded as UTF-8. Valid bytes pass through unchanged, and invalid bytes are replaced with U+FFFD:

redpanda.headers = [
    {key: "content-type", value: "application/json"},
    {key: "trace-id",     value: "����"}    -- Each of the four invalid bytes replaced with U+FFFD (�)
]

Iceberg mode shorthands

The key_value, value_schema_id_prefix, and value_schema_latest modes are shorthands for common section-based configurations. Use a mode when you don’t need per-section control, and use the section-based syntax when you do.

Table 3. Shorthand equivalents for section-based configuration
Iceberg mode	Equivalent section-based configuration
`disabled`	Not applicable. `disabled` turns off Iceberg table writes for the topic entirely. No key, value, or headers section applies.
`key_value`	`key:mode=binary;value:mode=binary;headers:value_type=binary` (every section left at its default)
`value_schema_id_prefix`	`value:mode=schema_id_prefix` (the `key` and `headers` sections stay at their defaults)
`value_schema_latest[:subject=…,protobuf_name=…]`	`value:mode=schema_latest[,subject=…,protobuf_name=…]` (the `key` and `headers` sections stay at their defaults)

Validation rules

Redpanda rejects the following configurations when you create or alter a topic:

Unknown section names (anything other than key, value, or headers).
Duplicate sections, or duplicate options within a section.
Empty keys or values in an <option>=<value> pair.
subject or protobuf_name set without mode=schema_latest.
layout set in the key section.
layout=nested set without mode=schema_id_prefix or mode=schema_latest.

Option values cannot contain , or ;, and whitespace is not trimmed. Avoid extra spaces around subject names or Protobuf message names.

Configuring anything beyond key:mode=binary and headers:value_type=binary requires every broker in the cluster to be running Redpanda version 26.2 or later. Until all brokers are upgraded, Redpanda rejects these configurations with an error.

The following configurations require all brokers to be running version 26.2 or later:

key:mode= set to anything other than binary
headers:value_type=string
value:layout=nested
value:mode=string

Example configurations

To decode a schema-encoded key and store header values as strings:

rpk topic alter-config orders --set redpanda.iceberg.mode="key:mode=schema_id_prefix;headers:value_type=string"

Key: decoded using the schema ID embedded in the key, stored as a struct in redpanda.key
Value: raw bytes (default)
Headers: decoded to UTF-8 strings

To decode the key and value using the latest schema, override the key’s subject, and nest the value fields under a value column:

rpk topic alter-config orders --set redpanda.iceberg.mode="key:mode=schema_latest,subject=orders-key-v2;value:mode=schema_latest,layout=nested"

Key: decoded using the latest schema registered for the orders-key-v2 subject, stored as a struct in redpanda.key
Value: decoded using the latest schema registered for its subject, with fields nested under a value struct column
Headers: raw bytes (default)

To store the key as a plain UTF-8 string and decode headers to strings:

rpk topic create events --topic-config redpanda.iceberg.mode="key:mode=string;headers:value_type=string"

Key: stored as a UTF-8 string in redpanda.key
Value: raw bytes (default)
Headers: decoded to UTF-8 strings

To verify the current configuration:

rpk topic describe orders -c | grep redpanda.iceberg.mode

If a section-based configuration is equivalent to one of the modes described in Supported Iceberg modes, rpk topic describe displays it using that mode’s name instead of the section syntax.

Resolve schemas within a Schema Registry context

If you use Schema Registry contexts to isolate schemas (for example, by environment or tenant), set the redpanda.schema.registry.context topic property to bind the topic to that context. Redpanda then resolves the schemas referenced by records in the topic against the configured context instead of the default context (.). Depending on the topic’s Iceberg mode, Redpanda looks up either the schema ID embedded in each record (value_schema_id_prefix mode) or the latest schema for a subject (value_schema_latest mode) within that context.

Both modes rely on a schema registered in Schema Registry to determine the Iceberg table structure.

Starting in Redpanda 26.2, Schema Registry contexts are enabled by default. See Schema Registry contexts to learn about contexts and qualified subject naming before you configure this property.

Set the Schema Registry context on a new topic

rpk topic create <topic-name> --topic-config redpanda.schema.registry.context=<context-name>

Set the Schema Registry context on an existing topic

rpk topic alter-config <topic-name> --set redpanda.schema.registry.context=<context-name>

The context name must start with a period (.), for example .staging. If you don’t set this property, Redpanda resolves schemas in the default context (.).

If Redpanda cannot resolve a record’s schema within the configured context, it doesn’t translate the record and instead writes it to a dead-letter queue (DLQ) table. See Troubleshoot Iceberg Topics.

Redpanda resolves schemas using the topic’s current redpanda.schema.registry.context value, not the context that was active when a record was ingested. Changing the context on a topic that has pending, uncommitted Iceberg translation entries can cause resolution errors or unexpected DLQ routing for records still being translated.

To change the Schema Registry context on a topic that is actively translating to Iceberg:

Disable Iceberg translation for the topic.
Wait for pending translation entries to commit.
Change redpanda.schema.registry.context.
Re-enable Iceberg translation.

How Iceberg modes translate to table format

Redpanda generates an Iceberg table with the same name as the topic. In each mode, Redpanda writes to a redpanda table column that stores a single Iceberg struct per record, containing nested columns of the metadata from each record, including the record key, headers, timestamp, the partition it belongs to, and its offset.

For example, if you produce to a topic ClickEvent according to the following Avro schema:

{
    "type": "record",
    "name": "ClickEvent",
    "fields": [
        {
            "name": "user_id",
            "type": "int"
        },
        {
            "name": "event_type",
            "type": "string"
        },
        {
            "name": "ts",
            "type": "string"
        }
    ]
}

The key_value mode writes to the following table format:

CREATE TABLE ClickEvent (
    redpanda struct<
        partition:      integer,
        timestamp:      timestamptz,
        offset:         long,
        headers:        array<struct<key: string, value: binary>>,
        key:            binary,
        timestamp_type: integer
    >,
    value binary
)

Use key_value mode if you want to use the Iceberg data in its semi-structured format.

The value_schema_id_prefix and value_schema_latest modes can use the schema to translate to the following table format:

CREATE TABLE ClickEvent (
    redpanda struct<
        partition: integer,
        timestamp:      timestamptz,
        offset:         long,
        headers:        array<struct<key: string, value: binary>>,
        key:            binary,
        timestamp_type: integer
    >,
    user_id integer NOT NULL,
    event_type string,
    ts string
)

As you produce records to the topic, the data also becomes available in object storage for Iceberg-compatible clients to consume. You can use the same analytical tools to read the Iceberg topic data in a data lake as you would for a relational database.

If Redpanda fails to translate the record to the columnar format as defined by the schema, it writes the record to a dead-letter queue (DLQ) table. See Troubleshoot Iceberg Topics for more information.

By default, Redpanda stores the record key in binary format in the redpanda.key column. To decode the key using a schema, or store it as a string, configure the key section. See Configure key, value, and header translation.

Schema types translation

Redpanda supports direct translations of the following types to Iceberg value domains:

Avro
Protobuf
JSON Schema

Avro type	Iceberg type
boolean	boolean
int	int
long	long
float	float
double	double
bytes	binary
string	string
record	struct
array	list
map	map
fixed	fixed*
decimal	decimal
uuid	uuid*
date	date
time	time*
timestamp	timestamp

Avro type

Iceberg type

boolean

int

long

float

double

bytes

binary

string

record

struct

array

list

map

fixed

fixed*

decimal

uuid

uuid*

date

time

time*

timestamp

*These types are not currently supported in Unity Catalog managed Iceberg tables.

There are some cases where the Avro type does not map directly to an Iceberg type and Redpanda applies the following transformations:

Enums are translated into the Iceberg string type.
Different flavors of time (such as time-millis) and timestamp (such as timestamp-millis) types are translated to the same Iceberg time and timestamp types, respectively.
Avro unions are flattened to Iceberg structs with optional fields. For example:
- The union ["int", "long", "float"] is represented as an Iceberg struct struct<0 INT NULLABLE, 1 LONG NULLABLE, 2 FLOAT NULLABLE>.
- The union ["int", null, "float"] is represented as an Iceberg struct struct<0 INT NULLABLE, 1 FLOAT NULLABLE>.
Two-field unions that contain null are represented as a single optional field only (no struct). For example, the union ["null", "long"] is represented as long.

Some Avro types are not supported:

The Avro duration logical type is ignored.
The Avro null type is ignored and not represented in the Iceberg schema.
Recursive types are not supported.

Protobuf type	Iceberg type
bool	boolean
double	double
float	float
int32	int
sint32	int
int64	long
sint64	long
sfixed32	int
sfixed64	long
string	string
bytes	binary
map	map
message	struct

Protobuf type

Iceberg type

bool

boolean

double

float

int32

int

sint32

int

int64

long

sint64

long

sfixed32

int

sfixed64

long

string

bytes

binary

map

message

struct

There are some cases where the Protobuf type does not map directly to an Iceberg type and Redpanda applies the following transformations:

Repeated values are translated into Iceberg list types.
Enums are translated into the Iceberg string type.
uint32 and fixed32 are translated into Iceberg long types as that is the existing semantic for unsigned 32-bit values in Iceberg.
uint64 and fixed64 values are translated into their Base-10 string representation.
google.protobuf.Timestamp is translated into timestamp in Iceberg.

Recursive types are not supported.

Requirements:

Only JSON Schema Draft-07 is currently supported.
You must declare the JSON Schema dialect using the $schema keyword, for example "$schema": "http://json-schema.org/draft-07/schema#".
You must use a JSON Schema that constrains JSON documents to a strict type so Redpanda can translate to Iceberg. In most cases this means each subschema uses the type keyword, but a subschema can also use $ref if the referenced schema resolves to a strict type.

Valid JSON Schema example

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "productId": {
      "type": "integer"
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string"
      }
    }
  }
}

JSON type Iceberg type Notes

JSON type	Iceberg type	Notes
array	list	The keywords `items` and `additionalItems` must be used to constrain element types.
boolean	boolean
null		The `null` type is only supported as a nullability marker, either in a `type` array (for example, `["string", "null"]`) or in an exclusive `oneOf` nullable pattern.
number	double
integer	long
string	string	The `format` keyword can be used for custom Iceberg types. See `format` annotation translation for details.
object	struct or map	Use `properties` to define `struct` fields and constrain their types. `additionalProperties: false` is supported for closed objects. If `additionalProperties` contains a schema, it translates to an Iceberg `map<string, T>`. You cannot combine `properties` and `additionalProperties` in an object if `additionalProperties` is set to a schema.

array

list

The keywords items and additionalItems must be used to constrain element types.

boolean

null

The null type is only supported as a nullability marker, either in a type array (for example, ["string", "null"]) or in an exclusive oneOf nullable pattern.

number

double

integer

long

string

The format keyword can be used for custom Iceberg types. See format annotation translation for details.

object

struct or map

Use properties to define struct fields and constrain their types. additionalProperties: false is supported for closed objects.
If additionalProperties contains a schema, it translates to an Iceberg map<string, T>.
You cannot combine properties and additionalProperties in an object if additionalProperties is set to a schema.

Table 4. `format` annotation translation
`format` value	Iceberg type
date-time	timestamptz
date	date
time	time

The following keywords have specific behavior:

The $ref keyword is supported for internal references resolved from schema resources declared in the same document (using $id), including relative and absolute URI forms. References to external resources and references to unknown keywords are not supported. A root-level $ref schema is not supported.
The oneOf keyword is supported only for the nullable serializer pattern where exactly one branch is {"type":"null"} and the other branch is a non-null schema (T|null).
In Iceberg output, Redpanda writes all fields as nullable regardless of serializer nullability annotations.

The following are not supported for JSON Schema:

The $dynamicRef keyword
The default keyword
Conditional typing (if, then, else, dependencies keywords)
Boolean JSON Schema combinations (allOf, anyOf, and non-nullable oneOf patterns)
Dynamic object members with the patternProperties keyword
The additionalProperties keyword when set to true

Suggested labs

Search all labs

Was this helpful?

group Ask in the community

mail Share your feedback

group_add Make a contribution

What do you think of this page?

Let us know more:

Let us contact you about your feedback:

Specify Iceberg Schema

Supported Iceberg modes

key_value

value_schema_id_prefix

value_schema_latest

disabled

Configure Iceberg mode for a topic

Override value_schema_latest default

Configure key, value, and header translation

Key and value section options

Key field examples

Value field examples

Headers section options

Header value examples

Iceberg mode shorthands

Validation rules

Example configurations

Resolve schemas within a Schema Registry context

How Iceberg modes translate to table format

Schema types translation

Suggested reading

Suggested labs

Simple online edits

Contribution guide

Override `value_schema_latest` default