Skip to main content
Version: Next

Polaris

Introduction​

Apache Polaris is an open-source, fully-featured catalog for Apache Iceberg. It implements Iceberg's REST Catalog interface, making Iceberg tables discoverable and queryable by any Iceberg-compatible engine, with role-based access control and credential vending built in.

This guide explains how to configure Fluss to use Polaris as its Iceberg catalog. For general Iceberg integration details (table mapping, data types, limitations), see Iceberg.

How It Works​

When Fluss is configured with Polaris as its Iceberg REST catalog:

  1. Fluss creates and manages Iceberg table metadata through Polaris's REST API
  2. The tiering service writes data to object storage and commits snapshots via Polaris
  3. Any Iceberg-compatible engine (Flink, Spark, Trino, StarRocks, etc.) can discover and query the tiered tables through Polaris

Prerequisites​

Running Polaris Instance​

You need a running Polaris instance with a catalog and a principal (a client_id / client_secret pair) that can access it. The fastest way to get started is the Polaris Quickstart, which starts Polaris and automatically creates a quickstart_catalog plus a quickstart_user principal, printing the principal's credentials in the container logs.

To create a catalog manually, first obtain an access token with your root credentials:

export TOKEN=$(curl -s http://<polaris-host>:8181/api/catalog/v1/oauth/tokens \
    -d 'grant_type=client_credentials' \
    -d 'client_id=<root-client-id>' \
    -d 'client_secret=<root-client-secret>' \
    -d 'scope=PRINCIPAL_ROLE:ALL' | jq -r '.access_token')

NOTE: These commands target Polaris's default realm. If your deployment uses a custom realm, add a -H "Polaris-Realm: <your-realm>" header to both the token request above and the catalog request below — otherwise they resolve against the default realm and the requests may be silently misrouted.

Then create a catalog backed by your object storage:

curl -X POST http://<polaris-host>:8181/api/management/v1/catalogs \
    -H "Authorization: Bearer ${TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{
      "catalog": {
        "name": "my_catalog",
        "type": "INTERNAL",
        "properties": { "default-base-location": "s3://my-bucket/iceberg" },
        "storageConfigInfo": {
          "storageType": "S3",
          "allowedLocations": ["s3://my-bucket/iceberg"],
          "roleArn": "<your-role-arn>"
        }
      }
    }'

NOTE: Adjust the storageConfigInfo to match your storage backend. Polaris supports S3, Azure, and GCS. You also need a principal with a role granting TABLE_WRITE_DATA on the catalog — see the Polaris documentation for catalog, principal, and access-control setup.

Vended credentials vs. static keys​

Polaris hands storage credentials to Fluss in one of two ways, depending on your object store:

  • Vended credentials (AWS S3 with STS) — the path used throughout this guide. The catalog's storageConfigInfo must include a roleArn, and Polaris must be allowed to AssumeRole on it; Polaris then vends temporary, scoped credentials per request (enabled by the X-Iceberg-Access-Delegation: vended-credentials header in the Fluss config below). Without a roleArn, table operations fail with Failed to get subscoped credentials: roleArn must not be null.
  • Static keys (MinIO, NooBaa, or other S3-compatible stores without STS) — there is no role to assume. Mark the catalog's storage config with "stsUnavailable": true, and provide static s3.access-key-id, s3.secret-access-key, and client.region to Fluss instead of the vended-credentials header. (Polaris also has a SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION feature flag, but it is test/dev-only — prefer stsUnavailable on the storage config.)

Configure Fluss with Polaris​

Cluster Configuration​

Add the following to your server.yaml:

datalake.format: iceberg
datalake.iceberg.type: rest
datalake.iceberg.io-impl: org.apache.iceberg.aws.s3.S3FileIO
datalake.iceberg.uri: http://<polaris-host>:8181/api/catalog
datalake.iceberg.warehouse: <catalog-name>
datalake.iceberg.credential: <client-id>:<client-secret>
datalake.iceberg.scope: PRINCIPAL_ROLE:ALL
datalake.iceberg.oauth2-server-uri: http://<polaris-host>:8181/api/catalog/v1/oauth/tokens
datalake.iceberg.header.X-Iceberg-Access-Delegation: vended-credentials

Fluss strips the datalake.iceberg. prefix and passes the remaining properties to the Iceberg REST catalog client. The io-impl property selects Iceberg's S3FileIO for warehouse data access; set it explicitly, because the default ResolvingFileIO requires Hadoop classes that the Fluss servers' Iceberg plugin does not bundle, and the catalog otherwise fails to load. The credential (client_id:client_secret), scope, and oauth2-server-uri properties configure OAuth2 client-credentials authentication against Polaris. Setting oauth2-server-uri explicitly is recommended: the Iceberg client can otherwise fall back to <uri>/v1/oauth/tokens, but that implicit fallback is deprecated and logs a warning. You can add any additional Iceberg REST catalog properties using the same prefix.

With credential vending enabled (X-Iceberg-Access-Delegation: vended-credentials), Polaris returns temporary, scoped storage credentials for each table request, so Fluss does not need static object-storage credentials. For stores without STS (e.g. MinIO), drop this header, set "stsUnavailable": true on the catalog's storage config, and supply static s3.access-key-id, s3.secret-access-key, and client.region to Fluss instead, as described in Vended credentials vs. static keys above.

Start Tiering Service​

Follow the Iceberg tiering service setup to prepare the required JARs and start the tiering service. Use the REST catalog parameters when launching the Flink tiering job:

${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-1.0-SNAPSHOT.jar \
    --fluss.bootstrap.servers <coordinator-host>:9123 \
    --datalake.format iceberg \
    --datalake.iceberg.type rest \
    --datalake.iceberg.io-impl org.apache.iceberg.aws.s3.S3FileIO \
    --datalake.iceberg.uri http://<polaris-host>:8181/api/catalog \
    --datalake.iceberg.warehouse <catalog-name> \
    --datalake.iceberg.credential <client-id>:<client-secret> \
    --datalake.iceberg.scope PRINCIPAL_ROLE:ALL \
    --datalake.iceberg.oauth2-server-uri http://<polaris-host>:8181/api/catalog/v1/oauth/tokens \
    --datalake.iceberg.header.X-Iceberg-Access-Delegation vended-credentials

Usage Example​

Create a Datalake-Enabled Table​

Flink SQL
USE CATALOG fluss_catalog;

CREATE TABLE orders (
    `order_id` BIGINT,
    `customer_id` INT NOT NULL,
    `total_price` DECIMAL(15, 2),
    `order_date` DATE,
    `status` STRING,
    PRIMARY KEY (`order_id`) NOT ENFORCED
) WITH (
    'table.datalake.enabled' = 'true',
    'table.datalake.freshness' = '30s'
);

Once the tiering service is running, Fluss automatically creates the corresponding Iceberg table in Polaris and begins tiering data.

Query Data​

Flink SQL
SET 'execution.runtime-mode' = 'batch';

-- Union read: combines fresh data in Fluss with historical data in Iceberg
SELECT COUNT(*) FROM orders;

For details on union reads, streaming reads, and reading with other engines, see Iceberg - Read Tables.

Further Reading​