TL;DR: The POC from Part 1 used a go func() and hoped for the best. The production version has a Redis-backed event queue with sharded workers, distributed locking for multi-replica safety, an idempotency layer (because Slack will retry), and integration tests against real Redis containers. This is what it took to make it reliable enough to run at scale.

The go func() Betrayal

The original handler acknowledged Slack's webhook with 200 OK, then spawned a goroutine:

go func() {
    h.metrics.RecordNewThread(ctx, event.Channel, event.Ts)
}()

Simple. Elegant. Wrong.

Three things broke in production within the first week:

1. Process restart during processing — the goroutine dies, the event is lost, Slack never retries (it already got 200 OK).
2. Replica contention — with two pods, both process the same event, doubling every counter.
3. Out-of-order processing — a reaction_removed arrives before its reaction_added because the goroutine scheduler interleaved them differently than the Event API delivered them.

The fix was a durable event queue, but the implementation taught me more about Redis than I expected.

Architecture: What Changed

Every event goes through a Redis list as a durable buffer. The HTTP handler acknowledges Slack immediately, then enqueues. Background workers dequeue and process. If the process dies mid-processing, the event stays on the queue. If another replica picks it up, the idempotency key prevents double-counting.

The Sharding Problem

A single Redis list with multiple workers is a race: two workers both BRPop the same list and process sibling events concurrently. A reply and its parent message, or a reaction_added / reaction_removed pair, lose ordering guarantees.

The solution was routing keys + sharded queues:

// routingKey identifies the "thread of conversation" an event belongs to
func routingKey(event *SlackEventData) string {
    switch event.Type {
    case "message":
        if event.ThreadTS != "" {
            return channel + ":" + event.ThreadTS  // reply → thread
        }
        return channel + ":" + event.TS             // top-level → itself
    case "reaction_added", "reaction_removed":
        return channel + ":" + event.Item.TS        // reaction → target message
    }
    return channel
}

A FNV-32a hash of the routing key maps to one of N shards (swh:queue:events:0 through swh:queue:events:N-1). Every event for the same thread lands on the same shard, consumed by the same worker, in order.

Distributed Locking: The Shard Lock

With multiple replicas (EKS, two+ pods), two workers from different pods can pop from the same shard simultaneously — defeating the ordering guarantee. Each shard can only have one consumer cluster-wide at a time.

Enter the shard lock: a Redis key (swh:lock:shard:0) with a TTL-based lease. The worker acquires it via SETNX, then periodically renews it via a Lua script that atomically checks ownership before extending TTL:

-- renewShardLockScript: only renew if still the owner
if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("PEXPIRE", KEYS[1], ARGV[2])
else
    return 0
end

The conditional renewal is critical — without it, a worker that lost its lock (e.g. network partition) would blindly extend the lock held by another replica, breaking the mutual-exclusion guarantee.

Lock Lifecycle

The 5s renew interval with a 15s TTL gives three missed renewals before the lease expires — enough to tolerate a GC pause without losing ownership.

Idempotency: Slack Retries Are the Least of Your Worries

The original article covered Slack's 3-second retry. But there's a subtler case: what happens when the worker crashes after dequeueing but before storing to Redis?

The event is gone from the queue (BRPop removed it) but never processed. Slack doesn't retry because the HTTP handler already returned 200 OK.

The fix is two-layered:

Layer 1: Event ID deduplication. Before enqueueing, the handler tries SETNX swh:processed:{event_id} with a 1-hour TTL. If the key already exists, the event was already accepted (by this pod or another) — return 200 OK without enqueueing again.

func (h *Handler) markProcessed(ctx context.Context, eventID string) (duplicate bool, err error) {
    if eventID == "" {
        return false, nil  // no event_id = can't dedupe
    }
    key := constants.ProcessedEventKeyPrefix + eventID
    ok, err := h.redisClient.SetNX(ctx, key, 1, constants.IdempotencyTTL).Result()
    if err != nil {
        return false, err
    }
    return !ok, nil  // false = first time, true = already seen
}

Layer 2: Event backlog. The queue itself (Redis list) provides durability against pod restarts during processing. An event stays in the queue until a worker successfully processes it. The only gap is the DEAD-letter scenario — event dequeued, worker crashes mid-dispatch. That's a known limitation we accept (rare enough that manual replay from Slack's API is acceptable).

The Metrics Security Wake-Up Call

The /metrics endpoint exposes Prometheus counters with labels like channel, user_id, thread_id, and reaction. Those labels come straight from Slack payloads — and they leak sensitive information.

In the POC, metrics were wide open on the same port as the webhook handler. In production, /metrics is gated by an IP allowlist:

metricsGroup := r.Group("/")
metricsGroup.Use(middleware.IPAllowlist(cfg.MetricsAllowedCIDRs))
middleware.MountMetrics(metricsGroup)

The allowlist defaults to RFC1918 private ranges + loopback — safe for an EKS pod whose Prometheus collector scrapes via the VPC. But there's a trap: this only works if SetTrustedProxies is configured correctly.

Without trusted proxy configuration, Gin blindly trusts X-Forwarded-For headers. Any client on the internet can forge their source IP to match the allowlist and access /metrics. The fix is explicit:

trustedProxies := cfg.TrustedProxyCIDRs
if len(trustedProxies) == 0 {
    trustedProxies = middleware.DefaultInternalCIDRs
}
r.SetTrustedProxies(trustedProxies)

This tells Gin: only respect X-Forwarded-For from known proxies (ALBs, service mesh sidecars). Everything else gets c.ClientIP() from the actual TCP connection.

The Testing Epiphany

The POC had zero tests. The production code has ~5,000 lines of tests. The biggest lesson was which kind of test catches which kind of bug.

What I Learned About Test Levels

The most valuable tests were the integration tests that spin up a real Redis 7-alpine container per test via testcontainers-go:

type suite struct {
    t      *testing.T
    srv    *httptest.Server
    redis  *redis.Client
    rc     *testredis.Instance
    appURL string
}

func newSuite(t *testing.T) *suite {
    ti := testredis.Start(t)  // real container via testcontainers
    rc := redis.NewClient(&redis.Options{Addr: ti.Addr})
    srv := httptest.NewServer(newRouter(ctx, cfg, rc, rl))
    // ...
}

The comment in the codebase explains the decision:

miniredis is a faithful reimplementation for simple cases, but it doesn't implement BRPop with the same blocking semantics as real Redis. One worker test was consistently passing against miniredis and failing in staging. After eight hours of debugging, the root cause was a subtle difference in how miniredis handles concurrent blocking pops from the same list key. Real containers or nothing.

Kill Your Container Mid-Test

The test framework supports a Kill() method that terminates the Redis container mid-test, allowing tests for behavior during Redis outages:

func (i *Instance) Kill(t *testing.T) {
    i.mu.Lock()
    defer i.mu.Unlock()
    if err := i.container.Terminate(context.Background()); err != nil {
        t.Fatalf("testredis: kill container: %v", err)
    }
}

Tests that exercise the worker's reconnection loop with a live-and-then-dead Redis caught the "infinite retry spam" bug — when Redis goes away, the worker's BRPop error loop logs once per second forever. The fix was a backoff: sleepOrDone(ctx, time.Second) on error before retrying.

Graceful Shutdown: The Final Mile

The original handler had no shutdown logic. SIGTERM would kill the process mid-webhook, mid-goroutine, mid-Redis-write. The production version uses Go's signal.NotifyContext with proper context propagation:

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()

// workers stop when ctx is done
h.StartWorkers(ctx, cfg.WorkerCount)

// server shutdown with 15s timeout
<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
srv.Shutdown(shutdownCtx)

The workers check ctx.Done() at the top of every loop iteration. On shutdown, they finish their current event (up to redisOpTimeout), then exit. No half-processed events, no abandoned locks (the lock TTL handles orphaned locks if a node hard-crashes).

Config Structure: From Hardcoded to 12-Factor

The POC had constants scattered across files. The maintenance commit centralizes everything:

type Config struct {
    Port                int
    RedisURL            string
    RedisTLS            bool
    RedisCluster        bool
    RedisClusterAddrs   []string
    RedisOperationTimeout time.Duration
    ReadTimeout, WriteTimeout, IdleTimeout time.Duration
    WorkerCount         int
    MetricsAllowedCIDRs []string
    TrustedProxyCIDRs   []string
    SlackSigningSecret  string
    GinMode             string
    // ...
}

Every value comes from an environment variable with sensible defaults. Production mode (Gin release) requires SLACK_SIGNING_SECRET — the app refuses to start without it:

func (c *Config) IsProduction() bool {
    return c.GinMode == "release"
}

func (c *Config) Validate() error {
    if c.IsProduction() && c.SlackSigningSecret == "" {
        return errors.New("slack signing secret is required in production mode")
    }
    // ...
}

This caught a near-miss in staging: someone forgot to set the signing secret in the Helm values, and the app refused to start rather than silently accepting unauthenticated webhooks.

What's Still Missing

The queue gives durability against process restarts, but there's still a dead-letter gap: if a worker dequeues an event and crashes before fully processing it, that event is lost. The fix would be a Redis-backed scheduled retry queue with a visibility timeout (like SQS), but that's a future iteration.

Redis Cluster routing is also untested in CI — each integration test gets a single-node container. The shard-key hashing and lock-key prefixing might behave differently under CLUSTER with actual shard slots. We validated this manually in staging, but it deserves automated coverage.

Wrapping Up

The journey from go func() to sharded workers with distributed locks was driven by concrete production failures, not premature optimization:

• Event loss → durable Redis queues
• Duplicate metrics → event ID deduplication + idempotency keys
• Out-of-order reactions → routing keys + sharded queues
• Replica contention → distributed shard locks with Lua-based lease management
• Security leaks → IP allowlist + trusted proxy configuration
• Silent startup failures → config validation with production-mode guards
• Redis outage chaos → graceful error handling with backoff
• Undefined shutdown behavior → signal handling + context propagation

Each layer adds complexity but eliminates a real failure mode. The test suite grew 10:1 vs. production code, but every test justifies its line count by catching a bug that would otherwise need an incident post-mortem to discover.

The full codebase is at internal/handler/ in the repository — about 800 lines of production code and 3,000 lines of tests. The most important file is worker.go (290 lines): it contains the queue, the locks, the routing, and the graceful-shutdown logic. Everything else is plumbing around it.