Kafka

Learn how to set up ClickHouse for observability workloads using Docker. This post walks through installation, key configs, and best practices for log ingestion.

In this post, we explore how to enrich streaming logs in real time using Glassflow's **join feature**. By joining a primary log stream with a secondary `users` topic on the `user_id` field, we demonstrate how to dynamically add contextual metadata like `username` and `region` to logs before inserting them into ClickHouse.

In this post, we explore how to use Glassflow's built-in deduplication fewature to prevent duplicate log events from polluting your ClickHouse database. With a simple update to the Fluent Bit Lua script, we simulate duplicate log entries by reusing the same `counter` value. Galssflow's streaming deduplication logic ensures only the first occurence is written to ClickHouse, based on a configurable time window and deduplication key. This results in cleaner logs, more accurate analysis, and reduced storage usage - all without post-processing or complex query logic.

This post walks through a complete local setup of Glassflow's Clickhouse ETL using Docker Compose. It showcases how to build a real-time log pipeline that generates synthetic logs with Fluent Bit, forwards them using OpenTelemetry Collector to Kafka, and finally ingests into ClickHouse using Glassflow's no-code UI. With clear configuration examples and a working schema, this guide is perfect for anyone looking to prototype or explore observability pipelines using ClickHouse and Kafka.