Deploying with Helm Charts

This page provides instructions for deploying a Fluss cluster on Kubernetes using Helm charts. The chart creates a distributed streaming storage system with CoordinatorServer and TabletServer components.

Prerequisites

Before installing the Fluss Helm chart, ensure you have:

• Kubernetes
• Helm
• For Local Development: Minikube and Docker (see Local Development with Minikube)

note

A Fluss cluster deployment requires a running ZooKeeper ensemble. To provide flexibility in deployment and enable reuse of existing infrastructure, the Fluss Helm chart does not include a bundled ZooKeeper cluster. If you don’t already have a ZooKeeper running, the installation documentation provides instructions for deploying one using Bitnami’s Helm chart.

Supported Versions

Component	Minimum Version	Recommended Version
Kubernetes	v1.19+	v1.25+
Helm	v3.8.0+	v3.18.6+
ZooKeeper	v3.6+	v3.8+
Apache Fluss (Container Image)	1.1-SNAPSHOT	1.1-SNAPSHOT
Minikube (Local Development)	v1.25+	v1.32+
Docker (Local Development)	v20.10+	v24.0+

Installation

Running Fluss locally with Minikube

For local testing and development, you can deploy Fluss on Minikube. This is ideal for development, testing and learning purposes.

Prerequisites

• Docker container runtime
• At least 4GB RAM available for Minikube
• At least 2 CPU cores available

Start Minikube

# Start Minikube with recommended settings for Fluss
minikube start

# Verify cluster is ready
kubectl cluster-info

Configure Docker Environment (Optional)

To build images directly in Minikube you need to configure the Docker CLI to use Minikube's internal Docker daemon:

# Configure shell to use Minikube's Docker daemon
eval $(minikube docker-env)

To build custom images please refer to Custom Container Images.

Installing the chart on a cluster

This installation process is generally working for a distributed Kubernetes cluster or a Minikube setup.

Step 1: Deploy ZooKeeper (Optional if ZooKeeper is existing)

To start Zookeeper use Bitnami’s chart or your own deployment. If you have an existing Zookeeper cluster, you can skip this step. Example with Bitnami’s chart:

# Add Bitnami repository
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# Deploy ZooKeeper
helm install zk bitnami/zookeeper

Step 2: Deploy Fluss

Install from Helm repo

helm repo add fluss https://downloads.apache.org/fluss/helm-chart
helm repo update
helm install helm-repo/fluss

Install from Local Chart

helm install fluss ./helm

Install with Custom Values

You can customize the installation by providing your own values.yaml file or setting individual parameters via the --set flag. Using a custom values file:

helm install fluss ./helm -f my-values.yaml

Or for example to change the ZooKeeper address via the --set flag:

helm install fluss ./helm \
  --set configurationOverrides.zookeeper.address=<my-zk-cluster>:2181

Cleanup

# Uninstall Fluss
helm uninstall fluss

# Uninstall ZooKeeper
helm uninstall zk

# Delete PVCs belonging to this release. Scope by instance: the
# app.kubernetes.io/name label is shared by every Fluss release in the
# namespace, and carries nameOverride when one is set.
kubectl delete pvc -l app.kubernetes.io/instance=fluss

# Stop Minikube
minikube stop

# Delete Minikube cluster
minikube delete

Architecture Overview

The Fluss Helm chart deploys the following Kubernetes resources:

Core Components

• CoordinatorServer: 1x StatefulSet with Headless Service for cluster coordination
• TabletServer: 3x StatefulSet with Headless Service for data storage and processing
• ConfigMap: Configuration management for server.yaml settings
• Services: Headless services providing stable pod DNS names, plus optional dedicated headless services when metrics are enabled

Step 3: Verify Installation

# Check pod status
kubectl get pods -l app.kubernetes.io/name=fluss

# Check services
kubectl get svc -l app.kubernetes.io/name=fluss

# View logs
kubectl logs -l app.kubernetes.io/component=coordinator
kubectl logs -l app.kubernetes.io/component=tablet

Configuration Parameters

The following table lists the configurable parameters of the Fluss chart, and their default values.

Global Parameters

Parameter	Description	Default
nameOverride	Override the name of the chart	""
fullnameOverride	Override the full name of the resources	""
uniqueResourceNames	Give every resource a name unique to the release, and scope the ZooKeeper root path to the namespace and release, so that several Fluss clusters can share a namespace. See Running several Fluss clusters in one namespace.	false

Image Parameters

Parameter	Description	Default
image.registry	Container image registry	""
image.repository	Container image repository	fluss
image.tag	Container image tag	1.1-SNAPSHOT
image.pullPolicy	Container image pull policy	IfNotPresent
image.pullSecrets	Container image pull secrets	[]

Application Configuration

Parameter	Description	Default
clusterDomain	Kubernetes cluster domain used in chart-owned CLIENT advertised FQDNs	cluster.local
listeners.internal.port	Internal communication port	9123
listeners.client.port	Client port (in-cluster advertised address)	9124
listeners.external.enabled	Bind an extra EXTERNAL listener so out-of-cluster clients can be advertised separately from in-cluster CLIENT	false
listeners.external.port	Bind port for the extra listener (must differ from internal, client, and metrics ports)	9125
listeners.external.advertisedHost	Shared advertised host for EXTERNAL (hostname/IP or ${NODE_IP}; required when enabled unless overridden per component)	""
listeners.external.advertisedPort	Shared advertised port for EXTERNAL (defaults to listeners.external.port)	""
coordinator.listeners.external.advertisedHost	Coordinator EXTERNAL advertised host (overrides the shared host)	""
coordinator.listeners.external.advertisedPort	Coordinator EXTERNAL advertised port	""
tablet.listeners.external.advertisedHost	Shared EXTERNAL host for every tablet (overrides the shared host)	""
tablet.listeners.external.advertisedHosts	Per-ordinal EXTERNAL hosts; length must equal tablet.numberOfReplicas	[]
tablet.listeners.external.advertisedPort	Shared EXTERNAL port for every tablet	""
tablet.listeners.external.advertisedPorts	Per-ordinal EXTERNAL ports; length must equal tablet.numberOfReplicas	[]

Security Configuration

Parameter	Description	Default
security.client.sasl.mechanism	Client listener SASL mechanism ("", plain)	""
security.internal.sasl.mechanism	Internal listener SASL mechanism ("", plain)	""
security.external.sasl.mechanism	EXTERNAL listener SASL mechanism ("", plain). Empty is PLAINTEXT (warns when EXTERNAL is enabled)	""
security.client.sasl.plain.users	Client listener username and password pairs for PLAIN	[]
security.external.sasl.plain.users	EXTERNAL listener username and password pairs for PLAIN	[]
security.internal.sasl.plain.username	Internal listener PLAIN username	""
security.internal.sasl.plain.password	Internal listener PLAIN password	""
security.internal.sasl.plain.existingSecret	Reference to a pre-existing Secret for internal SASL credentials	{}

Only plain mechanism is supported for now. An empty string disables SASL and maps to the PLAINTEXT protocol.

If the internal SASL username or password is left empty, the chart automatically generates credentials based on the Helm release name:

• Username is set to the "fluss-internal-user-<release-name>"
• Password is set to the SHA-256 hash of "fluss-internal-password-<release-name>"

It is recommended to set these explicitly in production.

ZooKeeper SASL Parameters

Parameter	Description	Default
security.zookeeper.sasl.mechanism	ZooKeeper SASL mechanism ("", plain)	""
security.zookeeper.sasl.plain.username	ZooKeeper SASL username	""
security.zookeeper.sasl.plain.password	ZooKeeper SASL password	""
security.zookeeper.sasl.plain.loginModuleClass	JAAS login module class for ZooKeeper	org.apache.fluss.shaded.zookeeper3.org.apache.zookeeper.server.auth.DigestLoginModule
security.zookeeper.sasl.plain.existingSecret	Reference to a pre-existing Secret for ZooKeeper SASL credentials	{}

Sourcing SASL Credentials from a Pre-existing Secret

To keep SASL passwords out of values.yaml and the Helm release storage, reference a Secret managed separately — e.g., via External Secrets Operator, Sealed Secrets, or a CI pipeline.

For internal and ZooKeeper listeners, set existingSecret on the listener:

security:
  internal:
    sasl:
      mechanism: plain
      plain:
        existingSecret:
          name: fluss-internal-sasl   # required
          usernameKey: username       # optional, defaults to "username"
          passwordKey: password       # optional, defaults to "password"
  zookeeper:
    sasl:
      mechanism: plain
      plain:
        existingSecret:
          name: fluss-zk-sasl

Client users follow the same shape as internal/ZooKeeper listeners: each entry is either a literal {username, password} pair or an existingSecret reference that sources both fields from a Secret.

security:
  client:
    sasl:
      mechanism: plain
      plain:
        users:
          - username: alice
            password: alice-literal-password   # literal — visible in values.yaml
          - existingSecret:                    # or resolved at pod startup
              name: fluss-client-sasl-bob
              usernameKey: username            # optional, defaults to "username"
              passwordKey: password            # optional, defaults to "password"

Whenever JAAS is required, the chart renders a ConfigMap (<release>-fluss-sasl-jaas-config) containing a jaas.conf template with ${FLUSS_JAAS_…} placeholders — no credentials. An init container mounts that template, runs envsubst with credentials supplied via env vars (either literal value: entries from values.yaml or valueFrom.secretKeyRef to a pre-existing Secret), and writes the resolved jaas.conf to an in-memory emptyDir that the main Fluss container reads.

• Literal and Secret-sourced credentials can be mixed across listeners.
• When every credential comes from a Secret, no plaintext password lives in the Helm release.
• The init container reuses the main Fluss image (already present on the node), keeping zero extra image dependencies.

Example: External Secrets Operator

If you use External Secrets Operator to sync credentials from an upstream secret manager (AWS Secrets Manager, Vault, GCP Secret Manager, etc.), the flow is: upstream → ExternalSecret CR → a Kubernetes Secret → the chart.

For internal listener credentials stored at prod/fluss/internal in AWS Secrets Manager with fields username and password:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: fluss-internal-sasl
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: fluss-internal-sasl
  data:
    - secretKey: username
      remoteRef:
        key: prod/fluss/internal
        property: username
    - secretKey: password
      remoteRef:
        key: prod/fluss/internal
        property: password

Then in values.yaml:

security:
  internal:
    sasl:
      mechanism: plain
      plain:
        existingSecret:
          name: fluss-internal-sasl

For the multi-user client listener, provision one Secret per user with username and password keys:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: fluss-client-sasl-alice
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: fluss-client-sasl-alice
  data:
    - secretKey: username
      remoteRef:
        key: prod/fluss/clients/alice
        property: username
    - secretKey: password
      remoteRef:
        key: prod/fluss/clients/alice
        property: password
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: fluss-client-sasl-bob
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: fluss-client-sasl-bob
  data:
    - secretKey: username
      remoteRef:
        key: prod/fluss/clients/bob
        property: username
    - secretKey: password
      remoteRef:
        key: prod/fluss/clients/bob
        property: password

security:
  client:
    sasl:
      mechanism: plain
      plain:
        users:
          - existingSecret: { name: fluss-client-sasl-alice }
          - existingSecret: { name: fluss-client-sasl-bob }

The same pattern works with Sealed Secrets, HashiCorp Vault Agent Injector (producing a native Secret), or any other controller that lands credentials in a Secret — the chart only cares about the final Secret, not how it got there.

Metrics Parameters

Parameter	Description	Default
metrics.reporters	Comma-separated reporter selector; use "" to disable metrics	""
metrics.jmx.port	JMX reporter port range	9250
metrics.prometheus.port	Prometheus reporter port	9249
metrics.prometheus.service.portName	Named port exposed on metrics services	metrics
metrics.prometheus.service.labels	Additional labels added to metrics services	{}
metrics.prometheus.service.annotations	Optional annotations added to metrics services	{}

Fluss Configuration Overrides

Parameter	Description	Default
configurationOverrides.default.bucket.number	Default number of buckets for tables	3
configurationOverrides.default.replication.factor	Default replication factor	3
configurationOverrides.zookeeper.path.root	ZooKeeper root path for Fluss. Becomes /fluss-<namespace>/<release> when uniqueResourceNames is enabled	/fluss
configurationOverrides.zookeeper.address	ZooKeeper ensemble address	zk-zookeeper.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}:2181
configurationOverrides.remote.data.dir	Remote data directory for snapshots	/tmp/fluss/remote-data
configurationOverrides.data.dir	Local data directory	/tmp/fluss/data
configurationOverrides.internal.listener.name	Internal listener name	INTERNAL

Secrets in Configuration Overrides

Parameter	Description	Default
secrets.basePath	Base directory for secret mounts	/etc/fluss/secrets
secrets.mounts	Secrets mounted as files under <basePath>/<name>, entries of {name, secretName}	[]
secrets.env	Secret values injected as env vars via secretKeyRef, entries of {name, secretName, key}	[]

Any configurationOverrides value can reference a secret through a config provider marker instead of a literal, so the rendered server.yaml ConfigMap never contains secret material. The chart generates the matching config.providers settings, restricted to the declared mounts and env names.

kubectl create secret generic fluss-paimon-creds \
  --from-literal=access-key=... --from-literal=secret-key=...

secrets:
  mounts:
    - name: paimon-creds
      secretName: fluss-paimon-creds

configurationOverrides:
  datalake.paimon.s3.access-key: ${directory:/etc/fluss/secrets/paimon-creds:access-key}
  datalake.paimon.s3.secret-key: ${directory:/etc/fluss/secrets/paimon-creds:secret-key}

Alternatively, a value can be injected as an environment variable:

secrets:
  env:
    - name: PAIMON_S3_ACCESS_KEY
      secretName: fluss-paimon-creds
      key: access-key

configurationOverrides:
  datalake.paimon.s3.access-key: ${env:PAIMON_S3_ACCESS_KEY}

To rotate a secret, update the Kubernetes Secret and restart the pods (kubectl rollout restart statefulset -l app.kubernetes.io/name=fluss); markers are resolved at server startup.

This works with any Secret producer — for example, an External Secrets Operator ExternalSecret syncing from AWS Secrets Manager or Vault into the referenced Secret. To restart pods automatically when the Secret changes, a controller such as Reloader can watch it (annotate the StatefulSets with secret.reloader.stakater.com/reload: "<secret-name>"), making rotation fully hands-off.

Tablet Server Parameters

Parameter	Description	Default
tablet.numberOfReplicas	Number of TabletServer replicas to deploy	3

Coordinator Server Parameters

Parameter	Description	Default
coordinator.numberOfReplicas	Number of CoordinatorServer replicas to deploy. One replica is the elected leader and the others run as warm standbys. See Coordinator High Availability	1

Scheduling Parameters

Parameter	Description	Default
tablet.affinity	Affinity rules for TabletServer pods	{}
tablet.nodeSelector	Node selector for TabletServer pods	{}
tablet.tolerations	Tolerations for TabletServer pods	[]
tablet.topologySpreadConstraints	Topology spread constraints for TabletServer pods	[]
coordinator.affinity	Affinity rules for CoordinatorServer pods	{}
coordinator.nodeSelector	Node selector for CoordinatorServer pods	{}
coordinator.tolerations	Tolerations for CoordinatorServer pods	[]
coordinator.topologySpreadConstraints	Topology spread constraints for CoordinatorServer pods	[]

Storage Parameters

Parameter	Description	Default
coordinator.storage.enabled	Enable persistent volume claims for CoordinatorServer	false
coordinator.storage.size	Coordinator persistent volume size	1Gi
coordinator.storage.storageClass	Coordinator storage class name	nil (uses default)
tablet.storage.enabled	Enable persistent volume claims for TabletServer	false
tablet.storage.size	Tablet persistent volume size	1Gi
tablet.storage.storageClass	Tablet storage class name	nil (uses default)

Resource Parameters

Parameter	Description	Default
resources.coordinatorServer.requests.cpu	CPU requests for coordinator	Not set
resources.coordinatorServer.requests.memory	Memory requests for coordinator	Not set
resources.coordinatorServer.limits.cpu	CPU limits for coordinator	Not set
resources.coordinatorServer.limits.memory	Memory limits for coordinator	Not set
resources.tabletServer.requests.cpu	CPU requests for tablet servers	Not set
resources.tabletServer.requests.memory	Memory requests for tablet servers	Not set
resources.tabletServer.limits.cpu	CPU limits for tablet servers	Not set
resources.tabletServer.limits.memory	Memory limits for tablet servers	Not set

Pod Extension Parameters

Parameter	Description	Default
coordinator.annotations	Annotations to add to the CoordinatorServer StatefulSet	{}
coordinator.service.annotations	Annotations to add to the CoordinatorServer headless Service	{}
coordinator.service.publishNotReadyAddresses	Publish DNS records for CoordinatorServer pods before they are Ready, so a restarting server stays resolvable by its advertised hostname	true
coordinator.extraVolumes	Extra volumes to add to the CoordinatorServer pod spec	[]
coordinator.extraVolumeMounts	Extra volume mounts to add to the coordinator container	[]
coordinator.initContainers	Init containers to run before the coordinator container starts	[]
coordinator.extraEnv	Additional environment variables for the coordinator container	[]
coordinator.envFrom	Additional envFrom sources (e.g., Secrets, ConfigMaps) for the coordinator container	[]
coordinator.podAnnotations	Annotations to add to CoordinatorServer pods	{}
coordinator.podLabels	Additional labels to add to CoordinatorServer pods	{}
coordinator.podDisruptionBudget.enabled	Enable PodDisruptionBudget for CoordinatorServer	false
coordinator.podDisruptionBudget.minAvailable	Minimum available coordinator pods during disruption	Not set
coordinator.podDisruptionBudget.maxUnavailable	Maximum unavailable coordinator pods during disruption	Not set
tablet.annotations	Annotations to add to the TabletServer StatefulSet	{}
tablet.service.annotations	Annotations to add to the TabletServer headless Service	{}
tablet.service.publishNotReadyAddresses	Publish DNS records for TabletServer pods before they are Ready, so a restarting server stays resolvable by its advertised hostname	true
tablet.extraVolumes	Extra volumes to add to TabletServer pod specs	[]
tablet.extraVolumeMounts	Extra volume mounts to add to the tablet container	[]
tablet.initContainers	Init containers to run before the tablet container starts	[]
tablet.extraEnv	Additional environment variables for the tablet container	[]
tablet.envFrom	Additional envFrom sources (e.g., Secrets, ConfigMaps) for the tablet container	[]
tablet.podAnnotations	Annotations to add to TabletServer pods	{}
tablet.podLabels	Additional labels to add to TabletServer pods	{}
tablet.podDisruptionBudget.enabled	Enable PodDisruptionBudget for TabletServer	false
tablet.podDisruptionBudget.minAvailable	Minimum available tablet server pods during disruption	Not set
tablet.podDisruptionBudget.maxUnavailable	Maximum unavailable tablet server pods during disruption	Not set

Workload and service annotations are where controllers such as Reloader (restart on Secret change) or Argo CD (sync-waves) hook in:

coordinator:
  annotations:
    secret.reloader.stakater.com/reload: fluss-internal-sasl
tablet:
  annotations:
    secret.reloader.stakater.com/reload: fluss-internal-sasl

Advanced Configuration

Injecting Environment Variables from External Secrets

You can inject environment variables from Kubernetes Secrets or ConfigMaps using envFrom. This is useful when combined with the External Secrets Operator or similar tools that provision Secrets from external stores (AWS Secrets Manager, HashiCorp Vault, etc.).

tablet:
  envFrom:
    - secretRef:
        name: aws-credentials
coordinator:
  envFrom:
    - secretRef:
        name: aws-credentials

You can also set individual environment variables using extraEnv:

tablet:
  extraEnv:
    - name: AWS_REGION
      value: us-east-1
    - name: MY_SECRET
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: password

Custom ZooKeeper Configuration

For external ZooKeeper clusters:

configurationOverrides:
  zookeeper.address: "zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181"
  zookeeper.path.root: "/my-fluss-cluster"

Running several Fluss clusters in one namespace

By default the chart gives each resource a fixed name (coordinator-server, tablet-server, fluss-conf-file, and the matching headless Services — the metrics Services already carry the release name) and puts every cluster on the ZooKeeper root path /fluss. Two releases in the same namespace would therefore collide on both.

Set uniqueResourceNames to give every resource a name unique to the release and move the ZooKeeper root path to /fluss-<namespace>/<release>:

helm install orders ./helm --set uniqueResourceNames=true
helm install payments ./helm --set uniqueResourceNames=true

This gives you orders-fluss-coordinator-server alongside payments-fluss-coordinator-server, on ZooKeeper roots /fluss-<namespace>/orders and /fluss-<namespace>/payments. The root carries the namespace because release names are only unique within one, so two orders releases in different namespaces can share a ZooKeeper ensemble. It is useful when namespace creation is restricted and one team needs more than one cluster.

Two things still need attention:

• configurationOverrides.remote.data.dir holds kv snapshots and tiered log segments. Whenever it points at a real remote filesystem, such as s3://bucket/fluss or an HDFS path, both releases write to the same place, so give each one its own path. The chart default, /tmp/fluss/remote-data, is a path inside each pod's own container filesystem and is not shared.
• The release name becomes part of every generated name through fluss.fullname, which is the release name plus -fluss, or the release name on its own when it already contains fluss. That prefix has to stay within 33 characters — a StatefulSet name cannot exceed 52, because each pod labels itself with the StatefulSet name plus a 10 character revision hash — so a release name without fluss in it is limited to 27. Set fullnameOverride for anything longer. The chart fails the render with an explicit message when the prefix is too long. That check covers the resources named after fluss.fullname; the metrics Services are named from the release name alone and are not covered by it.

To enable the option on a cluster that already exists, read Enabling unique resource names on an existing release first.

Network Configuration

By default the chart supports in-cluster clients only.

Each server binds INTERNAL (inter-broker) and CLIENT on the pod IP, and advertises CLIENT as the StatefulSet pod FQDN:

CLIENT://<pod>.tablet-server-hs.<ns>.svc.<clusterDomain>:9124

clusterDomain defaults to cluster.local. Set it if your cluster uses a different DNS domain. The headless service name in that FQDN is tablet-server-hs (or coordinator-server-hs) unless uniqueResourceNames prefixes it.

• Internal port (9123): server-to-server communication
• Client port (9124): in-cluster client connections (chart-owned advertised FQDN)
• External port (9125): optional extra listener for out-of-cluster clients (disabled by default)

bind.listeners and advertised.listeners are always written by the StatefulSet startup command. Do not set them in configurationOverrides. If an existing deployment still sets either key there, upgrade will fail with VALUES VALIDATION until those overrides are moved to listeners.*.

Out-of-cluster clients

A client bootstraps against the coordinator, then metadata redirects produce/fetch/lookup to each tablet's advertised address. In-cluster CLIENT FQDNs do not resolve outside the cluster, so exposing only the coordinator (a port-forward or a shared LoadBalancer) lets bootstrap succeed and writes fail.

Enable a second EXTERNAL listener and bootstrap against that advertised address, not the in-cluster CLIENT DNS. Keep the default CLIENT advertised FQDN for in-cluster workloads (Flink, Spark, other pods).

Each tablet must advertise a unique host or port. A single hostname for every replica cannot work: the client connects directly to the bucket leader. Uniqueness is configured with per-ordinal arrays (advertisedHosts / advertisedPorts), not shell formulas.

EXTERNAL requires coordinator.numberOfReplicas: 1. A standby coordinator rejects metadata, and per-ordinal coordinator addresses are not supported yet, so several coordinators would all advertise one EXTERNAL address. The chart fails the render in that case.

The chart does not create NodePort or LoadBalancer Services per pod. You still have to make the advertised addresses resolve and route to the right pod.

Per-pod DNS (recommended):

listeners:
  external:
    enabled: true
    port: 9125
coordinator:
  listeners:
    external:
      advertisedHost: coordinator-server-0.fluss.example.com
tablet:
  listeners:
    external:
      advertisedHosts:
        - tablet-server-0.fluss.example.com
        - tablet-server-1.fluss.example.com
        - tablet-server-2.fluss.example.com

Host IP + unique port (for example when you create per-pod NodePorts yourself). ${NODE_IP} is the only runtime token accepted in advertisedHost:

listeners:
  external:
    enabled: true
    port: 9125
    advertisedHost: "${NODE_IP}"
tablet:
  listeners:
    external:
      advertisedPorts: [30001, 30002, 30003]

Coordinator and tablet can override host/port independently when they must not share a value. Example for kubectl port-forward from a laptop (unique localhost ports; coordinator-server-0 and tablet-server-0 both have ordinal 0):

listeners:
  external:
    enabled: true
    port: 9125
coordinator:
  listeners:
    external:
      advertisedHost: "127.0.0.1"
      advertisedPort: "9125"
tablet:
  listeners:
    external:
      advertisedHost: "127.0.0.1"
      advertisedPorts: [9126, 9127, 9128]

Every pod listens on listeners.external.port (9125). The ports in advertisedPorts are on your machine, not on the pod:

kubectl port-forward pod/coordinator-server-0 9125:9125
kubectl port-forward pod/tablet-server-0 9126:9125
kubectl port-forward pod/tablet-server-1 9127:9125
kubectl port-forward pod/tablet-server-2 9128:9125

Bootstrap the client at 127.0.0.1:9125.

EXTERNAL security is independent of CLIENT. Default is PLAINTEXT, which prints a VALUES WARNING when EXTERNAL is enabled. Set security.external.sasl.mechanism to plain and provide security.external.sasl.plain.users. To share credentials between CLIENT and EXTERNAL, point both listeners' user entries at the same existingSecret:

security:
  client:
    sasl:
      mechanism: plain
      plain:
        users:
          - existingSecret: { name: fluss-sasl-alice }
  external:
    sasl:
      mechanism: plain
      plain:
        users:
          - existingSecret: { name: fluss-sasl-alice }

Setting security.external.sasl.* without listeners.external.enabled: true has no effect and also prints a VALUES WARNING. SASL/PLAIN without TLS sends credentials in cleartext; keep EXTERNAL on a trusted network.

Custom bind ports:

listeners:
  internal:
    port: 9123
  client:
    port: 9124

Enabling Secure Connection

With the helm deployment, you can specify authentication mechanisms when connecting to the Fluss cluster.

The following table shows the supported mechanisms and security they provide:

Mechanism	Method	Authentication	TLS Encryption
""	PLAINTEXT	No	No
plain	SASL	Yes	No

By default, the PLAINTEXT protocol is used.

You can set the SASL authentication by enabling plain mechanism.

security:
  client:
    sasl:
      mechanism: plain
      plain:
        users:
          - username: client-user
            password: client-password
  internal:
    sasl:
      mechanism: plain
      plain:
        username: internal-user
        password: internal-password

Enabling ZooKeeper SASL Authentication

You can enable ZooKeeper ensemble SASL authentication, with the following values in the Fluss Helm chart:

security:
  zookeeper:
    sasl:
      mechanism: plain
      plain:
        username: fluss-zk-user
        password: fluss-zk-password

The security.zookeeper.sasl.plain.username and security.zookeeper.sasl.plain.password fields are required when security.zookeeper.sasl.mechanism is set to plain.

ZooKeeper SASL can be enabled independently or together with the listeners SASL authentication.

Metrics and Monitoring

When metrics.reporters is set, the chart adds the following server.yaml configuration entries:

• metrics.reporters: comma-separated reporter names from metrics.reporters
• metrics.reporter.<name>.port: port value from metrics.<name>.port

These values are managed by the chart and cannot be set via configurationOverrides. All other metrics reporter options (refer to the Fluss configuration) should be specified via configurationOverrides.

Prometheus Annotation Based Scraping

The example values below show how to add annotations to the metrics services so that a Prometheus server can discover and scrape them automatically based on the annotations:

metrics:
  reporters: prometheus
  prometheus:
    port: 9249
    service:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9249"

Prometheus ServiceMonitor Based Scraping

Similarly, if using the Prometheus Operator, use the values below to add labels to the metrics services:

metrics:
  reporters: prometheus
  prometheus:
    port: 9249
    service:
      portName: metrics
      labels:
        monitoring: enabled

Then create a ServiceMonitor that selects them matching the labels:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: fluss-metrics
spec:
  selector:
    matchLabels:
      monitoring: enabled
  endpoints:
    # Matches `metrics.prometheus.service.portName`
    - port: metrics

Storage Configuration

Configure different storage volumes for coordinator or tablet pods:

coordinator:
  storage:
    enabled: true
    size: 5Gi
    storageClass: fast-ssd

tablet:
  storage:
    enabled: true
    size: 20Gi
    storageClass: fast-ssd

Configure remote storage:

configurationOverrides:
  data.dir: "/data/fluss"
  remote.data.dir: "s3://my-bucket/fluss-data"

Pod Scheduling

By default, Kubernetes may schedule all tablet server pods on the same node. Even with replication factor 3, a single node failure could take out all replicas simultaneously, causing data loss for segments not yet tiered to remote storage.

Use pod anti-affinity to spread tablet server pods across availability zones and nodes:

tablet:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            topologyKey: topology.kubernetes.io/zone
            labelSelector:
              matchLabels:
                app.kubernetes.io/instance: <release-name>
                app.kubernetes.io/component: tablet
        - weight: 50
          podAffinityTerm:
            topologyKey: kubernetes.io/hostname
            labelSelector:
              matchLabels:
                app.kubernetes.io/instance: <release-name>
                app.kubernetes.io/component: tablet

Replace <release-name> with your Helm release name (the value passed to helm install) so the selector scopes to pods of this release only. This matters when multiple Fluss releases share the cluster — otherwise anti-affinity would count pods across releases.

This configuration prioritizes zone-level spreading (weight 100) while also avoiding co-location on the same node (weight 50). For stricter guarantees, use requiredDuringSchedulingIgnoredDuringExecution instead — but note that pods will stay pending if no suitable node is available.

Alternatively, use topologySpreadConstraints for even distribution across failure domains:

tablet:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app.kubernetes.io/instance: <release-name>
          app.kubernetes.io/component: tablet
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app.kubernetes.io/instance: <release-name>
          app.kubernetes.io/component: tablet

You can also pin pods to specific nodes using nodeSelector or allow scheduling on tainted nodes with tolerations:

tablet:
  nodeSelector:
    workload: fluss
  tolerations:
    - key: dedicated
      operator: Equal
      value: fluss
      effect: NoSchedule

The same scheduling fields are available for coordinator servers under coordinator.affinity, coordinator.nodeSelector, coordinator.tolerations, and coordinator.topologySpreadConstraints.

Loading Filesystem Plugins via Init Containers

Fluss discovers filesystem plugins at startup by scanning subdirectories under $FLUSS_HOME/plugins/.
To load a plugin that is not bundled in the base image, you can use an init container to download the plugin jar into a shared emptyDir volume before the main container starts.

The example below loads the Azure filesystem plugin (fluss-fs-azure) so that Fluss can read and write remote data to Azure Blob Storage (the example is for version 0.9, adapt to your necessities):

_fsAzurePlugin: &fsAzurePlugin
  extraVolumes:
    - name: azure-plugin
      emptyDir: {}
  extraVolumeMounts:
    - name: azure-plugin
      mountPath: /opt/fluss/plugins/azure
      subPath: azure
  initContainers:
    - name: download-fs-azure
      image: alpine:3.20
      command:
        - sh
        - -c
        - |
          wget -O /plugins/azure/fluss-fs-azure-1.1-SNAPSHOT.jar \
            https://repo1.maven.org/maven2/org/apache/fluss/fluss-fs-azure/1.1-SNAPSHOT/fluss-fs-azure-1.1-SNAPSHOT.jar
      volumeMounts:
        - name: azure-plugin
          mountPath: /plugins

coordinator:
  <<: *fsAzurePlugin

tablet:
  <<: *fsAzurePlugin

Upgrading

Upgrade the Chart

# Upgrade to a newer chart version
helm upgrade fluss ./helm

# Upgrade with new configuration
helm upgrade fluss ./helm -f values-new.yaml

Listener configuration overrides

bind.listeners and advertised.listeners in configurationOverrides fail the render. The chart writes both from listeners.*. Move any existing overrides there before upgrading, or the upgrade stops with VALUES VALIDATION.

Enabling unique resource names on an existing release

Turning on uniqueResourceNames renames the StatefulSets, and a StatefulSet cannot be renamed in place. Helm creates the new one and deletes the old one, so the pods are replaced rather than rolled and the cluster restarts.

The headless Services are renamed too. Anything that reaches the cluster through coordinator-server-hs, such as bootstrap.servers in clients and Flink jobs, needs the new name afterwards: the coordinator StatefulSet name from the render below, plus -hs.

Pin the ZooKeeper root path the release uses today, so the new pods inherit the existing cluster metadata. Read it from the running configuration:

kubectl get configmap fluss-conf-file -n <namespace> \
  -o jsonpath='{.data.server\.yaml}' | grep zookeeper.path.root

uniqueResourceNames: true
configurationOverrides:
  zookeeper.path.root: /fluss   # or the root the release already uses

Without the pin they start against an empty /fluss-<namespace>/<release> and the old cluster's metadata is left behind.

The pin is permanent: this release keeps its current root for good. That is safe to leave alongside others, because the scoped roots sit under /fluss-<namespace>, a sibling of /fluss rather than a child. Fluss uses the root as a ZooKeeper namespace, so a release nested under this one would sit among its data.

The claims created from volumeClaimTemplates are named data-<statefulset>-<ordinal>, so renaming the StatefulSets renames every claim. Left alone, the new pods start on fresh empty volumes and the old claims are orphaned.

Read the new StatefulSet names off a render rather than assuming a prefix: fluss.fullname collapses to the release name on its own when that already contains fluss, and fullnameOverride replaces it outright.

helm template <release> ./helm -n <namespace> --set uniqueResourceNames=true -f <your-values> \
  | awk '/^kind: StatefulSet/{f=1} f&&/^  name:/{print $2; f=0}'

A release called prod prints prod-fluss-coordinator-server and prod-fluss-tablet-server, so its claims are data-prod-fluss-tablet-server-0 and so on. A release called fluss prints fluss-tablet-server, and its claim is data-fluss-tablet-server-0 — not data-fluss-fluss-tablet-server-0.

The volumes can be carried over. A StatefulSet creates a claim only when one of that name is missing, so the job is to make sure each new claim binds to the volume its predecessor used. You do that from the volume: a PersistentVolume whose claimRef names a claim is reserved for it, even before that claim exists. Reserve each volume for the claim the new StatefulSet is going to create, and the rest happens on its own.

warning

The cluster is down for the whole procedure, not just for the upgrade at the end. Any helm upgrade that enables this option replaces the pods rather than rolling them, but here the outage also covers the manual steps in between, so plan a maintenance window. If a GitOps controller syncs this release, suspend it first, or it will recreate the StatefulSets you delete in step 2.

Test this on a non-production cluster first. Binding behaviour varies by storage class: reserving a volume under WaitForFirstConsumer — the default on EKS and GKE — takes a different path through the scheduler than immediate binding does.

Do not change storage.size in this upgrade. Each new claim has to fit the volume you reserve for it, and one that asks for more will not bind at all: the pod never starts and the data is never picked up.

Back up or snapshot the volumes before you start. Pass -n <namespace> to every command below: the old names are the same in every namespace, so a command run against the wrong one acts on another cluster. Then:

1. Write down which volume backs which claim. You need this mapping in step 4, and it is gone once the claims are deleted.

   kubectl get pvc -n <namespace> -l app.kubernetes.io/instance=<release> \
     -o custom-columns=CLAIM:.metadata.name,VOLUME:.spec.volumeName

   Select on instance, not on app.kubernetes.io/name: the name label changes with nameOverride, and every release in the namespace shares it.

   CLAIM                      VOLUME
   data-coordinator-server-0  pvc-8f3a1c02-...
   data-tablet-server-0       pvc-b71e4d55-...
   data-tablet-server-1       pvc-2c9f0ab8-...
   data-tablet-server-2       pvc-5d84e719-...

   Then make each volume survive its claim being deleted:

   kubectl patch pv <volume> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

2. Delete the old StatefulSets, then wait for their pods to go. Nothing recreates them: a StatefulSet is a top-level object, and Helm only acts on the release when you run a command against it. The release is simply without them until step 5.

   kubectl delete statefulset -n <namespace> coordinator-server tablet-server
   kubectl wait --for=delete pod -n <namespace> -l app.kubernetes.io/instance=<release> --timeout=5m

   Do not skip the wait: a claim cannot be deleted while a pod still refers to it, and one left stuck in Terminating fails the upgrade at step 5.

3. Delete every old claim listed in step 1. Each volume becomes Released.

   kubectl delete pvc -n <namespace> data-coordinator-server-0 \
     data-tablet-server-0 data-tablet-server-1 data-tablet-server-2

4. Point each volume at the claim the new StatefulSet will create, using the names from the render above and keeping the ordinal the same. The volume that backed data-tablet-server-1 is reserved for the new tablet claim ending -1, and so on for every claim including the coordinator's. Ordinals matter because a tablet server takes its id from its pod ordinal: cross two of them over and a server boots on another server's data, with nothing to warn you.

   The claim need not exist yet. What matters is that the reference carries no uid — a stale one leaves the volume Released rather than reserved. So replace the whole claimRef, or null the uid explicitly; setting only name keeps it.

   kubectl patch pv <volume> --type json -p '[{
     "op": "replace",
     "path": "/spec/claimRef",
     "value": {
       "namespace": "<namespace>",
       "name": "data-<new-statefulset>-<ordinal>"
     }
   }]'

   Check the result before going on. Every volume should read Available with this namespace and its new claim name against it:

   kubectl get pv -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,NAMESPACE:.spec.claimRef.namespace,CLAIM:.spec.claimRef.name

5. Run the upgrade, repeating every -f and --set the release already had alongside the settings above:

   helm upgrade <release> ./helm -n <namespace> -f <your-values> -f <the-settings-above>

   Do not use --reuse-values: it replaces the new chart's defaults with the old release's values. Pass every value explicitly, or use --reset-then-reuse-values (Helm 3.14 or later). The new StatefulSets create their claims, each binds to the volume reserved for it, and the pods come up on the original disks.

If it goes wrong, the way back is the way in. The volumes are on Retain, so patch each claimRef to the old claim name instead and upgrade again without uniqueResourceNames.

Restore each volume's original reclaim policy afterwards if it was not Retain.

This needs no upkeep afterwards. The claims are created and labelled by the StatefulSet controller, exactly as they would have been on a fresh install, so you are not left owning extra objects. Later upgrades do not revisit them: volumeClaimTemplates is immutable once a StatefulSet exists, the controller only creates claims that are missing, and nothing deletes them — the claims are not part of the Helm release, and the StatefulSet's claim retention policy defaults to keeping them on both deletion and scale-down. Each claimRef becomes an ordinary binding once its claim appears and stays that way.

With storage.enabled: false, the default, the pods use emptyDir and there is nothing to preserve: run the upgrade and let the pods come back empty.

Delete the old StatefulSets first even then, and wait for their pods. Helm creates the new resources before deleting the old ones, so the two generations overlap and collide on their ZooKeeper identities. The server that arrives second retries for 60 seconds and then crash-loops until a later attempt wins. An ungraceful exit holds the old identity until its session expires, which zookeeper.client.session-timeout also defaults to 60 seconds.

Rolling Updates

The StatefulSets support rolling updates. When you update the configuration, pods will be restarted one by one to maintain availability.

Cluster Health Readiness Probe

The TabletServer readiness probe performs a two-step check to ensure safe rolling upgrades:

1. Local TCP port check — confirms the TabletServer process is up and listening.
2. Cluster health check — calls the Coordinator's Cluster Health API to confirm cluster status is GREEN (all replicas in-sync, all leaders active) across the cluster.

Only when both steps pass is the pod marked as Ready, allowing the StatefulSet controller to proceed to the next pod. If the Coordinator does not support the Cluster Health API (e.g., during a mixed-version upgrade from an older version), the probe immediately falls back to TCP-only port readiness.

You can tune the probe parameters in your values:

tablet:
  readinessProbe:
    # Timeout in ms for the RPC call to Coordinator (default: 5000)
    rpcTimeoutMs: 5000
    # Standard Kubernetes probe parameters (tuned for recovery checks)
    failureThreshold: 200
    timeoutSeconds: 10
    initialDelaySeconds: 15
    periodSeconds: 5

Parameter	Default	Description
rpcTimeoutMs	5000	Timeout in milliseconds for the RPC call to the Coordinator.
failureThreshold	360	Max consecutive probe failures before marking the pod as unready. With periodSeconds=5, this allows up to ~30 minutes for recovery.
periodSeconds	5	How often the probe runs.

note

The Coordinator should be upgraded after all TabletServers are fully upgraded and recovered.

Coordinator High Availability

Set coordinator.numberOfReplicas above 1 to run the CoordinatorServer with high availability. One replica is elected leader. The others run as warm standbys and take over through leader election when the leader fails. The default is 1, so high availability is opt-in.

coordinator:
  numberOfReplicas: 3
  podDisruptionBudget:
    enabled: true
    maxUnavailable: 1

Coordinator Readiness Probe

A standby coordinator binds the same client port as the leader, so a TCP check cannot tell a healthy standby from a hung one. The coordinator readiness probe instead sends a cluster health request to the local server, and the answer tells the probe the server's role:

• The leader answers. It is Ready with any cluster health status, so a coordinator rollout does not depend on TabletServer health. The TabletServer probes already hold a rolling upgrade until the cluster has recovered.
• A standby rejects the request, which shows that it handles requests. The probe then asks the other coordinator pods, through their headless Service DNS names, for the cluster description. The standby becomes Ready once the leader lists the pod's advertised address among the standby coordinators. The leader lists a coordinator while its ZooKeeper registration exists. A rolling update therefore stops at a restarted pod when no leader answers, or when the leader does not see the pod. The probe asks all other coordinators at once, so a hung coordinator cannot use up the probe timeout before the leader is asked. After an unclean exit, the old registration stays until its ZooKeeper session expires (zookeeper.client.session-timeout, 60 seconds by default). A pod that restarts within that time can become Ready on the old registration.

The role check runs only while a pod starts. Once it passes, or once the healthCheckTimeoutSeconds budget (20 minutes by default) runs out, the probe switches to a TCP-only check for the rest of the pod's life. A later ZooKeeper outage therefore cannot mark every coordinator NotReady at the same time. The probe parameters are the same as the TabletServer ones, under coordinator.readinessProbe.*.

When the client listener enforces SASL, both readiness probes need a credential. The chart fails to render if either one has none. Set healthCheckAuth on both coordinator.readinessProbe and tablet.readinessProbe, or keep the credential out of the StatefulSet spec by reading it from a Secret into the READINESS_HEALTH_CHECK_AUTH env var, which the probe reads:

secrets:
  env:
    - name: READINESS_HEALTH_CHECK_AUTH
      secretName: fluss-probe-auth
      key: health-check-auth  # e.g. client.security.protocol:SASL;client.sasl.mechanism:PLAIN;...

An entry in secrets.env reaches both components. A READINESS_HEALTH_CHECK_AUTH entry in coordinator.extraEnv or tablet.extraEnv covers that component only. The chart cannot see variables loaded through envFrom, so they do not satisfy this check.

The probe script ships inside the Fluss image, so this chart version needs an image whose readiness-check.sh supports the coordinator role (Fluss 1.1 or later). A standby probe that finds no leader that lists it, while another coordinator runs a version without the cluster description API, falls back to the TCP-only check at once. An older image ships a readiness-check.sh that ignores the role and runs the TabletServer check against the coordinator. That check expects a TabletServer at the probed address and fails on any coordinator, leader or standby. Every coordinator pod then stays NotReady until the healthCheckTimeoutSeconds budget expires and the probe falls back to the TCP-only check. The rollout completes, but each coordinator pod waits out the full budget, one after another. Images without readiness-check.sh never become Ready under this chart. Upgrade the chart and the image together, or lower healthCheckTimeoutSeconds for the transition rollout. An older chart with a newer image keeps its TCP-only coordinator probe and is not affected.

Running More Than One Replica

• Client bootstrap: list every coordinator replica in bootstrap.servers, for example coordinator-server-0.coordinator-server-hs.<ns>:9124,coordinator-server-1.coordinator-server-hs.<ns>:9124,coordinator-server-2.coordinator-server-hs.<ns>:9124. A standby answers client requests with NotCoordinatorLeaderException and clients retry the other bootstrap addresses, so any live replica in the list keeps bootstrap working during a failover.
• Checking the replicas: Admin.describeCluster() returns the leader as getCoordinatorServer() and the other registered replicas as getCoordinatorStandbys() (see Cluster Inspection). A listed standby is registered with ZooKeeper. Whether it is also serving requests is what its readiness probe reports.
• PodDisruptionBudget: enable coordinator.podDisruptionBudget with maxUnavailable: 1 so voluntary disruptions (node drains) never take down more than one coordinator at a time.
• Rolling updates: the StatefulSet updates one pod at a time, from the highest ordinal down. When the leader restarts, a standby takes over through leader election. The restarted pod rejoins as a standby and becomes Ready once the new leader lists it.
• Spreading: use coordinator.affinity or coordinator.topologySpreadConstraints to keep replicas on separate nodes or zones. Replicas on the same node share its failure domain.

Custom Container Images

Building Custom Images

To build and use custom Fluss images:

1. Build the project with Maven:

mvn clean package -DskipTests

2. Build the Docker image:

# Copy build artifacts
cp -r build-target/* docker/fluss/build-target

# Build image
cd docker
docker build -t my-registry/fluss:custom-tag .

3. Use in Helm values:

image:
  registry: my-registry
  repository: fluss
  tag: custom-tag
  pullPolicy: Always

Monitoring and Observability

Health Checks

The chart includes liveness and readiness probes. Liveness uses a TCP socket check:

livenessProbe:
  tcpSocket:
    port: 9124
  initialDelaySeconds: 10
  periodSeconds: 3
  failureThreshold: 100

Readiness depends on the server role. TabletServers check cluster health before reporting Ready, so a rolling upgrade waits for recovery (see Cluster Health Readiness Probe). CoordinatorServers check whether they are the leader or a standby that the leader lists (see Coordinator Readiness Probe).

Logs

Access logs from different components:

# Coordinator logs
kubectl logs -l app.kubernetes.io/component=coordinator -f

# Tablet server logs
kubectl logs -l app.kubernetes.io/component=tablet -f

# Specific pod logs
kubectl logs coordinator-server-0 -f
kubectl logs tablet-server-0 -f

Troubleshooting

Common Issues

Pod Startup Issues

Symptoms: Pods stuck in Pending or CrashLoopBackOff state

Solutions:

# Check pod events
kubectl describe pod <pod-name>

# Check resource availability
kubectl describe nodes

# Verify ZooKeeper connectivity
kubectl exec -it <fluss-pod> -- nc -zv <zookeeper-host> 2181

Image Pull Errors

Symptoms: ImagePullBackOff or ErrImagePull

Solutions:

• Verify image repository and tag exist
• Check pull secrets configuration
• Ensure network connectivity to registry

Connection Issues

Symptoms: Clients cannot connect to Fluss cluster

Solutions:

# Check service endpoints
kubectl get endpoints

# Test network connectivity
kubectl exec -it <client-pod> -- nc -zv <fluss-service> 9124

# Verify DNS resolution
kubectl exec -it <client-pod> -- nslookup <fluss-service>

Debug Commands

# Get all resources
kubectl get all -l app.kubernetes.io/name=fluss

# Check configuration
kubectl get configmap fluss-conf-file -o yaml


# Get detailed pod information
kubectl get pods -o wide -l app.kubernetes.io/name=fluss