🌊 We are excited to announce the official release of Apache Fluss 1.0!

AI agents and applications need data that keeps up with the world around them: current user profiles, recent interactions, changing business state, and fresh features for inference. Training pipelines also need access to historical data and newly arriving events. Fluss 1.0 strengthens the data layer connecting these workloads, from straightforward HTTP access for agents and native Python integration to continuous processing with Flink and training data preparation with Spark.

This is the first major release since Fluss graduated from the Apache Incubator to become a Top-Level Project. The community took a longer development cycle to prepare for this milestone. The release brings together over a thousand commits and 450,000 new lines of code, roughly three times the amount added in previous releases. These numbers reflect the breadth of feature development, refinement, and validation, as well as the Fluss community's continued collaboration and active development.

A production case study in columnar streaming, cold-data isolation, and lakehouse integration Presented at Flink Forward Asia 2026.

Rednote (Xiaohongshu) is a lifestyle community platform centered on content discovery and sharing. During the FIFA World Cup, it becomes a hub for live coverage, pre-match analysis, trending posts, and fan discussions.

At that scale, hundreds of millions of users can hit live streaming, search, and feeds at once. Content must refresh in moments while advertising, search, and recommendation services stay stable—powered by a real-time indexing pipeline that continuously ingests data and updates indexes.

That pipeline is critical to Rednote's content distribution and user experience, but its Kafka-based wide-table architecture was reaching cost and stability limits. To scale without compromising real-time performance, Rednote introduced Fluss into its core production path and began progressively migrating index data from Kafka to Fluss.

As officially announced by the Apache Software Foundation, we are thrilled to share that Apache Fluss has graduated from the Apache Incubator to become a Top Level Project (TLP).

The project's graduation proposal received unanimous approval from the Apache Incubator Project Management Committee (IPMC) and was subsequently approved by the ASF Board of Directors. This milestone not only marks a new stage in Fluss's journey, but also further advances the convergence of streaming storage, the real-time Lakehouse, and AI data infrastructure, opening a new chapter for real-time data infrastructure.

Part 1 and Part 2 built up everything you need to know about how tiering behaves: the mental model, the dials, the queue dynamics, the scale-out story. This part is about what to do with all of that. What breaks at runtime, and which of those failures self-heal versus need operator action. The design mistakes that look fine on day one but come back to bite you on day two. And the operator's daily view: which five numbers tell you whether tiering is healthy on a Tuesday afternoon, where each one comes from, and why two of them can only come from your Flink-side dashboards.

Tiering Service Deep Dive, 3-parts:

• Part 1 - The Mental Model: how one tiering round actually works, from timer fire to lake commit.
• Part 2 - Tuning: per-table dials, multi-table dynamics, and scaling out.
• Part 3 - In Production: failure modes, design pitfalls, and the dashboard that tells you everything is fine.

Part 1 built the mental model. What tiering is, who does what, how the round runs end-to-end.

This part adds the dials. Buckets and splits determine how a round parallelizes. Log and PK tables behave so differently on round one that the difference deserves its own treatment. The freshness setting, the one knob most users actually touch, does two different jobs that share the same value. Once a single job is handling many tables, queue position starts to dominate effective freshness more than any per-table setting. And once that happens, you have a deployment-shape decision: stay with one job, or scale out. By the end, you'll know which levers matter most and how to use them.

Tiering Service Deep Dive, 3-parts:

• Part 1 - The Mental Model: how one tiering round actually works, from timer fire to lake commit.
• Part 2 - Tuning: per-table dials, multi-table dynamics, and scaling out.
• Part 3 - In Production: failure modes, design pitfalls, and monitoring.

If you're new to Fluss, the lake-tiering story is one of those topics where every explanation seems to assume you already know how it works. This three-part walkthrough aims to bring some clarity to the confusing parts of the system, and to help you understand how it works in practice.

Part 1 builds the mental model from scratch and by the end of it you'll be able to describe, step by step, what happens between the moment a tiering timer fires and the moment a lake snapshot is committed.

Part 2 and Part 3 take that mental model and add the dials (parallelism, table kinds, freshness, multi-table behavior, scale-out) and then put it into a real production deployment (failures, pitfalls, monitoring).

Tiering Service Deep Dive, 3-parts:

• Part 1 - The Mental Model: how one tiering round actually works, from timer fire to lake commit.
• Part 2 - Tuning: per-table dials, multi-table dynamics, and scaling out.
• Part 3 - In Production: failure modes, design pitfalls, and monitoring.

Apache Fluss stores data in three places: local disk on the tablet server, remote object storage like S3, and the lakehouse. Which place holds which data at any given moment, and what is responsible for moving it between them, is the foundation everything else rests on. Your capacity plan depends on it. Your latency targets depend on it. Your disaster-recovery story depends on it. So does your ability to predict, in advance, that a particular configuration change is going to fill up local disk a week later.

TL;DR:

Apache Kafka's "column pruning" is actually pseudo-pruning. All fields still cross the network, and clients discard unwanted ones after the fact. Apache Fluss redesigns the storage format, server-side read path, and write-side batching strategy from the ground up with Arrow IPC columnar storage, zero-copy server-side pruning, and client-side pre-shuffle batching. The result: pruning 90% of columns yields a 10x read throughput improvement, with performance scaling linearly with the pruning ratio.

Every autumn in China, social media floods with posts about "The First Cup of Milk Tea in Autumn." With a tap on their phone, consumers expect their order delivered within 30 minutes. That effortless experience is no accident: it is the result of Taobao Instant Commerce making thousands of data-driven decisions every second.

Taobao Instant Commerce has scaled from a single-category food delivery service into a high-frequency platform spanning fresh produce, consumer electronics (3C), and beauty products. It operates under two very different modes: steady high-frequency daily transactions, and explosive traffic surges during promotional events where order volumes can multiply within minutes. Both demand the same thing: real-time responsiveness across hundreds of millions of SKUs.

Real-time is not a nice-to-have here; it is the lifeline for three critical functions:

• Operations: Refresh conversion rates and funnels within 30 seconds.
• Algorithms: Order prediction models must iterate at minute-level granularity.
• Quality Assurance: Canary release anomalies must be detected within seconds and trigger instant alerts.

The existing pipeline (built on Kafka, Flink, Paimon, and StarRocks) handled this at one scale.

Note: In Alibaba's internal infrastructure, TT (TimeTunnel) is the internal equivalent of Apache Kafka — a high-throughput distributed message queue. Throughout this post, "Kafka" refers to TT in the Taobao Instant Commerce context. But as the business grew, three fundamental bottlenecks emerged: unbounded state growth from stream joins, mounting complexity in building multi-stream denormalized tables, and excessive resource consumption from lakehouse synchronization. Together they formed an impossible triangle: no matter how the team tuned the system, latency, consistency, and cost could not all be optimized at once.

Fluss broke this impasse. By replacing the fragmented stream-batch architecture with a unified storage layer, its features (Delta Join, Partial Update, Streaming-Lakehouse Unification, Column Pruning, and Auto-Increment Columns) systematically eliminated all three bottlenecks and fundamentally reshaped how Taobao Instant Commerce handles real-time decision-making at scale.

UV (Unique Visitors) measures the count of distinct users who visited a page or triggered an event within a given time window — unlike PV (Page Views), which counts every request regardless of who made it. For any product or platform, accurate real-time UV statistics across dimensions like channel, city, date, and hour are a core analytical requirement. The full combination of four dimensions means 16 grouping methods; when the dimension count increases to seven, the number of possible groupings reaches 128.

How can multi-dimensional deduplication be both accurate and flexible while maintaining real-time performance? Behind this challenge lie two very different computing paradigms: direct deduplication of raw data, or set operations based on bitmaps.