- Published on
In this post, we explore how to use Glassflow's built-in deduplication fewature to prevent duplicate log events from polluting your ClickHouse database. With a simple update to the Fluent Bit Lua script, we simulate duplicate log entries by reusing the same `counter` value. Galssflow's streaming deduplication logic ensures only the first occurence is written to ClickHouse, based on a configurable time window and deduplication key. This results in cleaner logs, more accurate analysis, and reduced storage usage - all without post-processing or complex query logic.