Updating Configs

Overview

Fluss allows you to update cluster or table configurations dynamically without requiring a cluster restart or table recreation. This section demonstrates how to modify and apply such configurations.

Updating Cluster Configs

From Fluss version 0.8 onwards, some of the server configs can be updated without restarting the server.

Currently, the supported dynamically updatable server configurations include:

• datalake.enabled: Control whether the cluster is ready to create and manage lakehouse tables. When this option is explicitly configured to true, datalake.format must also be configured.
• datalake.format: Specify the lakehouse format, e.g., paimon, iceberg. When enabling lakehouse storage explicitly, use it together with datalake.enabled = true.
• Options with prefix datalake.${datalake.format}
• log.retention.roll-active-segment.enabled: Control whether a non-empty active segment can be rolled after the effective local cleanup TTL expires and the high watermark reaches the log end offset. See TTL.
• kv.rocksdb.shared-rate-limiter.bytes-per-sec: Control RocksDB flush and compaction write rate shared across all RocksDB instances on the TabletServer. The rate limiter is always enabled. Set to a lower value (e.g., 100MB) to limit the rate, or a very high value to effectively disable rate limiting.
• security.sasl.plain.credentials: Add, change, or remove the users that can authenticate with SASL/PLAIN, see Authentication.
• server.historical-partition.thread-pool.max-size: Change the maximum number of threads used for historical partition operations.
• netty.server.max-queued-historical-requests: Change the maximum number of in-flight historical partition operations, including lookups and writes, before throttling new operations.

You can update the configuration of a cluster with Java client or Flink SQL.

Java Client

// Enable lakehouse storage with Paimon format
admin.alterClusterConfigs(
        Arrays.asList(
                new AlterConfig(DATALAKE_ENABLED.key(), "true", AlterConfigOpType.SET),
                new AlterConfig(DATALAKE_FORMAT.key(), "paimon", AlterConfigOpType.SET)));

// Disable lakehouse storage
admin.alterClusterConfigs(
        Collections.singletonList(
                new AlterConfig(DATALAKE_ENABLED.key(), "false", AlterConfigOpType.SET)));

// Set RocksDB shared rate limiter to 200MB/sec
admin.alterClusterConfigs(
        Collections.singletonList(
                new AlterConfig(KV_SHARED_RATE_LIMITER_BYTES_PER_SEC.key(), "200MB", AlterConfigOpType.SET)));

The AlterConfig class contains three properties:

• key: The configuration key to be modified (e.g., datalake.format)
• value: The configuration value to be set (e.g., paimon)
• opType: The operation type, either AlterConfigOpType.SET or AlterConfigOpType.DELETE

Flink SQL

-- Use the Fluss catalog (replace 'fluss_catalog' with your catalog name if different)
USE fluss_catalog;

-- Enable lakehouse storage with Paimon format
CALL sys.set_cluster_configs(
  config_pairs => 'datalake.enabled', 'true', 'datalake.format', 'paimon'
);

-- Disable lakehouse storage
CALL sys.set_cluster_configs(
  config_pairs => 'datalake.enabled', 'false'
);

-- Set RocksDB shared rate limiter to 200MB/sec
CALL sys.set_cluster_configs(
  config_pairs => 'kv.rocksdb.shared-rate-limiter.bytes-per-sec', '200MB'
);

-- Allow expired active log segments to be rolled
CALL sys.set_cluster_configs(
  config_pairs => 'log.retention.roll-active-segment.enabled', 'true'
);

See Procedures for detailed documentation on get_cluster_configs, set_cluster_configs, append_cluster_configs, subtract_cluster_configs, and reset_cluster_configs procedures.

Updating Table Configs

The connector options on a table including Storage Options can be updated dynamically by ALTER TABLE ... SET statement. See the example below:

-- Enable lakehouse storage for the given table
ALTER TABLE my_table SET ('table.datalake.enabled' = 'true');