Amazon S3

Amazon Simple Storage Service (Amazon S3) is cloud object storage with industry-leading scalability, data availability, security, and performance.

Dependencies

Apache Fluss publishes the S3 filesystem plugin to Maven Central:

Artifact	Jar
Fluss S3 filesystem plugin	fluss-fs-s3-1.0-SNAPSHOT.jar

Maven coordinates:

<dependency>
  <groupId>org.apache.fluss</groupId>
  <artifactId>fluss-fs-s3</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency>

Verify downloaded JARs against the KEYS file using the verification instructions.

Configurations setup

To enabled S3 as remote storage, there are some required configurations that must be added to Fluss' server.yaml:

# The dir that used to be as the remote storage of Fluss
remote.data.dir: s3://<your-bucket>/path/to/remote/storage
# access key
s3.access-key: <your-access-key>
# secret key
s3.secret-key: <your-secret-key>
# region
s3.region: <your-s3-region>

S3-Compatible Storage (RustFS, MinIO, etc.)

For S3-compatible storage services such as RustFS or MinIO, you need to configure a custom endpoint and enable path-style access:

remote.data.dir: s3://<your-bucket>/path/to/remote/storage
s3.endpoint: http://<your-s3-compatible-endpoint>:9000
s3.access-key: <your-access-key>
s3.secret-key: <your-secret-key>
s3.region: us-east-1
s3.path-style-access: true

AssumeRole STS Configuration

Some S3-compatible services (such as RustFS) require the use of AssumeRole instead of GetSessionToken to obtain temporary security credentials. This is necessary for features like KV snapshots that rely on delegation tokens.

To enable AssumeRole, add the following configurations alongside the base S3 settings above:

remote.data.dir: s3://<your-bucket>/path/to/remote/storage
s3.endpoint: http://<your-s3-compatible-endpoint>:9000
s3.access-key: <your-access-key>
s3.secret-key: <your-secret-key>
s3.region: us-east-1
s3.path-style-access: true
s3.assumed.role.arn: <your-role-arn>
s3.assumed.role.sts.endpoint: http://<your-s3-compatible-endpoint>:9000

Configuration	Description
s3.assumed.role.arn	The ARN of the IAM role to assume. When set, Fluss uses AssumeRole instead of GetSessionToken to obtain temporary credentials. The s3.access-key and s3.secret-key are still required — they authenticate the AssumeRole call itself.
s3.assumed.role.sts.endpoint	Custom STS endpoint URL. Required for S3-compatible services that host their own STS API. When not set, the default AWS STS endpoint is used.

note

Without s3.assumed.role.arn, Fluss falls back to GetSessionToken (the default AWS behavior). This is fully backward compatible — existing AWS users do not need to change their configuration.

Default AWS Credential Chain (IRSA, Instance Profiles)

When running Fluss on Kubernetes with IAM Roles for Service Accounts (IRSA) or on EC2 with instance profiles, you can omit s3.access-key and s3.secret-key. The server will authenticate using the default AWS credential chain.

note

IRSA and instance profiles return session credentials, so they must use this default chain — not the explicit Custom AWS Credentials Provider mode, which requires long-term credentials.

In this mode, s3.assumed.role.arn is required — the server uses AssumeRole to generate temporary credentials for clients (Flink/Spark connectors) that read tiered data from S3.

remote.data.dir: s3://<your-bucket>/path/to/remote/storage
s3.region: <your-s3-region>
s3.assumed.role.arn: <your-delegation-role-arn>

The server's IAM role (e.g., the IRSA service account role) must have:

• Read/write permissions on the S3 bucket (for the server's own data access)
• sts:AssumeRole permission on the role specified in s3.assumed.role.arn

note

The server authenticates using its own credentials (static keys, IRSA, instance profile, or environment variables) for S3 data access. For delegation, the server calls AssumeRole with the configured s3.assumed.role.arn, so clients receive credentials for that role — which can have different permissions (e.g., read-only). Note that the server uses the same identity for both its own S3 access and the STS AssumeRole call. Further decoupling is planned.

Custom AWS Credentials Provider

For deployments that manage long-term credentials through a standard AWS SDK or Hadoop S3A credentials provider (for example, a rotating profile), set s3.aws.credentials.provider to the provider class. Fluss then treats it as the authoritative server-side credential source, so rotated credentials are picked up without restarting the servers.

remote.data.dir: s3://<your-bucket>/path/to/remote/storage
s3.region: <your-s3-region>
s3.aws.credentials.provider: com.amazonaws.auth.profile.ProfileCredentialsProvider

Configuration	Description
s3.aws.credentials.provider	Fully qualified class name of an AWS SDK / Hadoop S3A credentials provider. It is used both for the server's own S3 access and for resolving the credentials used to mint temporary client credentials via GetSessionToken. Takes precedence over s3.access-key/s3.secret-key when both are configured.

note

This mode only supports providers that return long-term access-key credentials, and it cannot be combined with s3.assumed.role.arn. Providers that return session credentials — including IRSA and EC2 instance profiles — must use the default AWS credential chain instead.