TransactIQ: Building Real-Time Transaction Intelligence with Go, Kafka, and ClickHouse

Indonesia processes over Rp 3,000 trillion in digital transactions a year, yet most SMEs and mid-size fintechs still learn about suspicious activity from a report generated the next day. Real-time fraud monitoring has traditionally been the privilege of companies with big-tech infrastructure budgets. TransactIQ is my answer to a simple question: what would it take to give that capability to everyone else?

The Goals

I set four hard requirements before writing any code:

Ingest must be fast: accepting a transaction should take single-digit milliseconds, no matter what analysis happens later.

Anomalies must be explainable: a flagged transaction should say why it was flagged, not just emit a black-box score.

Analytics must be conversational: an ops person should be able to ask "transaksi gagal per kota minggu ini" in plain Bahasa Indonesia and get an answer.

The whole thing must run with one command: docker compose up, reproducible anywhere.

Architecture: Separate the Hot Path from Everything Else

The core design principle is that accepting a transaction and analyzing it are different jobs with different latency budgets. The API should never wait for analysis.

Client → FastAPI gateway → Kafka (transactions topic)
                              ├─→ Go consumer → ClickHouse   (analytics)
                              └─→ Python risk engine → PostgreSQL (anomaly cases)
                                        └─→ Kafka (anomaly-alerts) → WebSocket → operators

The FastAPI gateway authenticates (JWT, role-based: merchant / risk analyst / admin), validates, and publishes to Kafka, all in under 5ms end to end. Everything downstream consumes the stream in parallel at its own pace, and Kafka's durability means a crashed consumer replays exactly where it left off.

Why Go for the Consumer

The ClickHouse writer is the throughput bottleneck of the system, so it got its own dedicated service in Go 1.22. Goroutines make concurrent partition consumption trivial, and the service accumulates rows and batch-inserts into ClickHouse, which is the single most important optimization, since ClickHouse loves large inserts and hates row-by-row writes. The Go consumer exposes Prometheus metrics, so consumer lag and insert latency are always visible in Grafana.

Could Python have done this? At demo scale, sure. But the goal was an architecture whose ceiling is the message broker, not the consumer.

Two Databases, Deliberately

• ClickHouse owns analytics: summaries, hourly/daily timeseries, merchant rankings via materialized views. Aggregations over millions of rows return in milliseconds.
• PostgreSQL owns truth: users, merchants, and anomaly cases that require ACID updates as analysts investigate and resolve them.

Trying to make one database do both jobs is how systems end up slow at both. The OLAP/OLTP split adds a second database but removes an entire category of performance tuning.

The Risk Engine: Boring on Purpose

The risk engine is rule-based, and that is a feature, not a limitation. Three configurable, layered rules:

Rule	Condition	Score
High value	amount ≥ Rp 10,000,000	+40
Suspicious category	gambling / crypto / adult	+30
High velocity	> 10 transactions in 5 minutes	+30

Score ≥ 70 flags the transaction into anomaly_cases and publishes to the anomaly-alerts topic, which a WebSocket broadcaster pushes to connected operators instantly. Every flag is fully explainable: the case record shows exactly which rules fired. For fraud workflows where an analyst must justify blocking someone's money, explainability beats a model you can't interrogate.

The velocity rule is powered by Redis sliding windows, a sorted-set pattern that answers "how many transactions did this account make in the last 5 minutes" in constant time, without ever touching a database.

Natural-Language Analytics with a Safety Cage

The NL2SQL layer translates Bahasa Indonesia questions into ClickHouse SQL using Groq's hosted Llama 3.3-70b (fast, and it genuinely understands Indonesian). LLM-generated SQL is obviously dangerous, so it runs inside a cage: a SELECT-only guard rejects any statement that isn't a read, and results are cached in Redis for five minutes so repeated dashboard questions don't burn LLM calls.

This one feature changed how the dashboard feels: instead of building a filter UI for every conceivable question, the ops user just asks.

Observability and the Dashboard

Prometheus scrapes every service (the FastAPI gateway via instrumentator middleware, the Go consumer natively) and Grafana dashboards track ingest rate, consumer lag, risk-engine throughput, and alert volume. The user-facing dashboard is Next.js with SWR for polling reads and Recharts for timeseries, deliberately thin, because the intelligence lives in the backend.

Lessons Learned

Latency budgets are architecture. Deciding "ingest gets 5ms, analytics gets a stream" dictated every component choice that followed.

Batching is the cheapest performance win in streaming systems. The Go consumer's batch inserts did more for throughput than any other optimization.

Explainable beats clever for fraud. Rules with scores that a human can read out loud earn trust in a way opaque models don't; start there, add ML later if the data demands it.

LLMs belong behind guards. NL2SQL is a superpower with a SELECT-only cage around it, and a liability without one.

TransactIQ ended up being the project where all the distributed-systems theory (brokers, consumer groups, OLAP/OLTP separation, sliding windows) stopped being theory and became a running docker compose stack I can demo from a cold start in one command.