Kafka

In this post, we explore how to use Glassflow's built-in deduplication fewature to prevent duplicate log events from polluting your ClickHouse database. With a simple update to the Fluent Bit Lua script, we simulate duplicate log entries by reusing the same `counter` value. Galssflow's streaming deduplication logic ensures only the first occurence is written to ClickHouse, based on a configurable time window and deduplication key. This results in cleaner logs, more accurate analysis, and reduced storage usage - all without post-processing or complex query logic.

This post walks through a complete local setup of Glassflow's Clickhouse ETL using Docker Compose. It showcases how to build a real-time log pipeline that generates synthetic logs with Fluent Bit, forwards them using OpenTelemetry Collector to Kafka, and finally ingests into ClickHouse using Glassflow's no-code UI. With clear configuration examples and a working schema, this guide is perfect for anyone looking to prototype or explore observability pipelines using ClickHouse and Kafka.