Streaming

rpk topic produce

Produce records to a topic. Producing records reads from STDIN, parses input according to --format, and produces records to Redpanda.

The input formatter understands a wide variety of formats. Parsing input operates on either sizes or on delimiters, both of which can be specified in the same formatting options.

Formatting

Parsing is based on percent escapes and modifiers. If using sizes to specify something, the size must come before what it is specifying. Delimiters match on an exact text basis. This command will quit with an error if any input fails to match your specified format.

Slashes can be used for common escapes:

Escape Description

\t

Tabs

\n

Newlines

\r

Carriage returns

\\

Slashes

\xNN

Hex encoded characters

Percent encoding reads into specific values of a record:

Percent encoding Description

%t

Topic

%T

Topic length

%k

Key

%K

Key length

%v

Value

%V

Value length

%h

Begin the header specification

%H

Number of headers

%p

Partition (if using the --partition flag)

%%

Percent sign

%{

Left brace

%}

Right brace

Modifiers

Text and numbers can be read in multiple formats, and the default format can be changed within brace modifiers. %v reads a value, while %v{hex} reads a value and then hex decodes it before producing. %T reads the length of a topic from the input, while %T{3} reads exactly three bytes for a topic from the input.

All modifiers go within braces following a percent-escape.

Numbers

Reading number values can have the following modifiers:

Number Description

ascii

Parse numeric digits until a non-numeric (default)

hex64

Sixteen hex characters

hex32

Eight hex characters

hex16

Four hex characters

hex8

Two hex characters

hex4

One hex character

big64

Eight byte big endian number

big32

Four byte big endian number

big16

Two byte big endian number

big8

Alias for byte

little64

Eight byte little endian number

little32

Four byte little endian number

little16

Two byte little endian number

little8

Alias for byte

byte

One byte number

<digits>

Directly specify the length as this many digits

bool

Read "true" as 1, "false" as 0

When reading number sizes, the size corresponds to the size of the encoded values, not the decoded values. %T{6}%t{hex} reads six hex bytes and decodes them into three.

Text

Reading text values can have the following modifiers:

Modifier Description

hex

Read text, then hex decode it

base64

Read text, then std-encoding base64 decode it

re

Read text matching a regular expression

json

Read text as JSON, then compact it

Headers

Headers are parsed with an internal key/value specifier format. For example, the following reads three headers that begin and end with a space and are separated by an equals sign:

%H{3}%h{ %k=%v }

Usage

rpk topic produce <topic> [flags]

Schema registry

Records can be encoded using a specified schema from your schema registry. Use the --schema-id or --schema-key-id flags to define the schema ID, and rpk will retrieve the schemas and encode the record accordingly.

Additionally, using topic in these flags allows for the use of the Topic Name Strategy. This strategy identifies a schema subject name based on the topic itself. For example:

Produce to foo, encode using the latest schema in the subject foo-value:

rpk topic produce foo --schema-id=topic

For protobuf schemas, you can specify the fully qualified name of the message you want the record to be encoded with. Use the --schema-type flag or --schema-key-type. If the schema contains only one message, specifying the message name is unnecessary. For example:

Produce to foo, using schema ID 1, message FQN Person.Name:

rpk topic produce foo --schema-id 1 --schema-type Person.Name

Tombstones

By default, records produced without a value will have an empty-string value, "". The below example produces a record with the key not_a_tombstone_record and the value "":

rpk topic produce foo -k not_a_tombstone_record
[Enter]

Tombstone records (records with a null value) can be produced by using the -Z flag and creating empty-string value records. Using the same example from above, but adding the -Z flag will produce a record with the key tombstone_record and the value null:

rpk topic produce foo -k tombstone_record -Z
[Enter]

Records produced with values of string "null" are not considered tombstones by Redpanda.

Miscellaneous

Producing requires a topic to produce to. The topic can be specified either directly as an argument, or in the input text through %t. A parsed topic takes precedence over the default passed in topic. If no topic is specified directly and no topic is parsed, this command will quit with an error.

The input format can parse partitions to produce directly to with %p. Doing so requires specifying a non-negative --partition flag. Any parsed partition takes precedence over the --partition flag; specifying the flag is the main requirement for being able to directly control which partition to produce to.

You can also specify an output format to write when a record is produced successfully. The output format follows the same formatting rules as the topic consume command. See that command’s help text for a detailed description.

Examples

This section provides examples of how to use rpk topic produce.

Format examples

A key and value, separated by a space and ending in newline

rpk topic produce my-topic -f '%k %v\n'

A four byte topic, four byte key, and four byte value

rpk topic produce -f '%T{4}%K{4}%V{4}%t%k%v'

A value to a specific partition (requires non-negative --partition flag)

rpk topic produce my-topic -p 0 -f '%p %v\n'

A big-endian uint16 key size, the text " foo ", and then that key

rpk topic produce my-topic -f '%K{big16} foo %k'

A value that can be two or three characters followed by a newline

rpk topic produce my-topic -f '%v{re#...?#}\n'

A key and a JSON value, separated by a space

rpk topic produce my-topic -f '%k %v{json}'

Flags

Value Type Description

--acks

int

Number of acknowledgments required before a produce request is considered successful (-1 = all in-sync replicas, 0 = no acknowledgment, 1 = leader only).

--allow-auto-topic-creation

bool

Auto-create non-existent topics; requires auto_create_topics_enabled on the broker.

-z, --compression

string

Compression algorithm to use when producing batches (none, gzip, snappy, lz4, zstd).

--delivery-timeout

duration

Per-record delivery timeout, if non-zero, min 1s.

-f, --format

string

Input record format.

-H, --header

stringArray

Headers in format key:value to add to each record (repeatable).

-k, --key

string

A fixed key to use for each record (parsed input keys take precedence).

--max-message-bytes

int32

If non-negative, maximum size of a record batch before compression.

-o, --output-format

string

Output to write to stdout when a record is successfully produced.

-p, --partition

int32

Partition to directly produce to, if non-negative (also allows %p parsing to set partitions).

--schema-id

string

Schema ID to encode the record value with, use topic for TopicName strategy.

--schema-key-id

string

Schema ID to encode the record key with, use topic for TopicName strategy.

--schema-key-type

string

Name of the protobuf message type to be used to encode the record key using schema registry.

--schema-type

string

Name of the protobuf message type to be used to encode the record value using schema registry.

-Z, --tombstone

bool

Produce empty values as tombstones.

Global flags

Value Type Description

--config

string

Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml.

-X, --config-opt

stringArray

Override rpk configuration settings; -X help for detail or -X list for terser detail.

--ignore-profile

bool

Ignore rpk.yaml and redpanda.yaml; use default settings.

--profile

string

rpk profile to use.

-v, --verbose

bool

Enable verbose logging.