Streaming

rpk connect streams

Run Redpanda Connect in streams mode, where multiple pipelines can be executed in a single process and can be created, updated, and removed via REST HTTP endpoints. The config field specified with the --observability/-o flag is known as the root config and should only contain observability and service-wide config fields such as http, metrics, logger, resources, and so on.

Usage

rpk connect streams [flags]

Examples

This section provides examples of how to use rpk connect streams.

Example.

rpk connect streams

Example.

rpk connect streams -o ./root_config.yaml

Example.

rpk connect streams ./path/to/stream/configs ./and/some/more

Example.

rpk connect streams -o ./root_config.yaml ./streams/*.yaml

Flags

Value Type Description

--no-api

bool

Disable the HTTP API for streams mode.

--prefix-stream-endpoints

bool

Whether HTTP endpoints registered by stream configs should be prefixed with the stream ID.

-o, --observability

string

a path to a configuration file containing general service-wide fields such as http, logger, and so on.

--log.level

string

override the configured log level, options are: off, error, warn, info, debug, trace.

-s, --set

strings

set a field (identified by a dot path) in the main configuration file, for example, "metrics.type=prometheus".

-r, --resources

strings

pull in extra resources from a file, which can be referenced the same as resources defined in the main config, supports glob patterns (requires quotes).

--chilled

bool

continue to execute a config containing linter errors.

-w, --watcher

bool

EXPERIMENTAL: watch config files for changes and automatically apply them.

--secrets

strings

Attempt to load secrets from a provided URN. If more than one entry is specified they will be attempted in order until a value is found. Environment variable lookups are specified with the URN env:, which by default is the only entry. In order to disable all secret lookups specify a single entry of none:.

--redpanda-license

string

Provide an explicit Redpanda License, which enables enterprise functionality. By default licenses found at the path /etc/redpanda/redpanda.license are applied.

--chroot

string

Chroot into the provided directory after parsing configuration. Common /etc/ files are copied to the chroot directory, and the directory is made read-only. This flag is only supported on Linux.

--chroot-passthrough

strings

Specify additional files to be copied into the chroot directory. This flag can be used multiple times. It is only supported when --chroot is used.

--disable-telemetry

bool

Disable anonymous telemetry from being emitted by this Connect instance.

--telemetry-deployment-type

string

Tags telemetry payloads with the deployment type. Expected values: self-hosted, byoc, serverless. When unset the field is omitted from the payload.

--telemetry-tenant-id

string

Tags telemetry payloads with an opaque tenant identifier (typically the Redpanda Cloud organization ID for cloud-managed deployments). When unset the field is omitted from the payload.

--rpc-plugins

strings

Plugins to load over the RPC interface. This flag should point to manifest files containing the plugin definitions. Globs are also supported.

-e, --env-file

strings

import environment variables from a dotenv file.

-t, --templates

strings

EXPERIMENTAL: import Redpanda Connect templates, supports glob patterns (requires quotes).

Global flags

Value Type Description

--config

string

Redpanda or rpk config file; default search paths are ~/.config/rpk/rpk.yaml, $PWD/redpanda.yaml, and /etc/redpanda/redpanda.yaml.

-X, --config-opt

stringArray

Override rpk configuration settings; -X help for detail or -X list for terser detail.

--ignore-profile

bool

Ignore rpk.yaml and redpanda.yaml; use default settings.

--profile

string

rpk profile to use.

-v, --verbose

bool

Enable verbose logging.

Suggested reading