System Design Q&A

Why is consistent hashing considered fine for caches but a poor fit for durable data stores (Cassandra, DynamoDB, ScyllaDB all use it anyway)?

Consensus was that caching tolerates rebalancing-on-failure because losing a cache entry is cheap; durable stores generally avoid rebalancing on transient failures because it's resource-intensive, and virtual nodes at real scale mean a lot of metadata. DynamoDB itself, despite citing the original Dynamo paper, actually uses range-based partitioning rather than classic consistent hashing.

In a 5-node Redlock-style distributed lock, if a client holds the majority (3/5) and 2 of those nodes go down, can a second client acquire the lock on the remaining 2?

No — 2 out of 5 is not a majority, so the second client's acquisition attempt should fail outright; the quorum threshold doesn't shrink with node failures.

Should Redis-based distributed locks (Redlock) be replaced with ZooKeeper or etcd?

Redlock is generally fine for cron dedup, background jobs, and best-effort mutual exclusion; for financial correctness, distributed transactions, or critical state machines, a consensus-backed store (ZooKeeper, etcd, Consul) is the safer choice.

How does a heartbeat/gossip system using the phi-accrual failure detector actually transport its pings — TCP or UDP?

UDP — heartbeats are inherently connectionless, and phi-accrual just consumes heartbeat timing as an input signal to compute a continuous suspicion level rather than a binary up/down.

A FAANG interview asked to "design a distributed linked list" — how do you even approach a data-structure-shaped system design question?

The community's take: treat it less literally — most "distributed X data structure" interview prompts really want a persistent-data-structure discussion backed by a consensus layer (raft/ZK), not an actual linked list with pointers across machines.

Assigning 20,000 vouchers to the first 1M concurrent requesters using Postgres `SELECT ... FOR UPDATE SKIP LOCKED` works at small scale but starts silently under-assigning vouchers as concurrency rises — why?

The thread didn't reach a fully confirmed root cause, but suspicion centered on lock contention/row visibility under heavy concurrent SKIP LOCKED scans rather than an application bug; recommended debugging by testing at intermediate scale (100s of concurrent requests) before jumping to full load.

Booking a conference room by checking for overlapping time-interval rows — how do you apply pessimistic locking when there are no pre-existing rows to lock?

Two paths surfaced: pre-populate fixed time-slot rows (e.g. 15/30-min buckets) so there's always something to lock, or create a "draft" booking row immediately and lock on the interval-overlap check within a `REPEATABLE READ` transaction, treating overlapping bookings as a strictly-serialized queue and non-overlapping ones as freely concurrent.

Two concurrent INSERTs can both pass a "does this row exist" check and create duplicate rows even with `SELECT FOR UPDATE`, since there's nothing to lock before the row exists — what's the fix?

Use `INSERT ... SELECT ... WHERE NOT EXISTS` as an atomic guard, or rely on a unique constraint on the natural key and let the DB reject the duplicate.

Postgres `REPEATABLE READ` vs `SERIALIZABLE` for implementing optimistic locking — is a version-column WHERE-clause check better than just raising the isolation level?

Postgres's `REPEATABLE READ` already implements snapshot isolation; raising to `SERIALIZABLE` (serializable snapshot isolation) buys stronger anomaly protection at the cost of more transaction aborts/retries under contention — the choice comes down to whether the app can tolerate retry storms.

A "hot row" problem: many queue workers doing `SELECT ... FOR UPDATE` + `UPDATE` on the same one or two aggregate-counter rows starts throwing MySQL "communication link failure" errors under load.

Classic hot-row contention from serializing all writers on a single row; the accepted mitigation direction is sharding the counter (e.g. suffix with a random partitioning key and sum across shards on read) rather than trying to tune the lock away.

MySQL's default `REPEATABLE READ` isolation doesn't fully prevent phantom reads — what's the fix without dropping to `SERIALIZABLE` and losing concurrency?

No clean universal answer emerged beyond the standard tradeoff: accept phantom reads for the use case, or scope `SERIALIZABLE` narrowly to the specific transaction that can't tolerate them rather than the whole app.

Designing a system to filter ~15 user attributes across sharded Aadhar-scale data, where a query might only specify a subset of attributes — how do you avoid searching every shard?

You have to give up cross-shard completeness: fire the query against one shard at a time and stop once you have enough results, accepting no global ordering. If global ordering is required, the only option is filtering on every shard, pulling all filtered results to one machine, then sorting centrally — expensive but sometimes necessary.

Using a monotonically-increasing timestamp as a DynamoDB/HBase sort key seems natural for "give me all records between X and Y," but someone flagged it as a hotspot pattern — why?

Tying a sort key directly to timestamp means one hash key absorbs unbounded write pressure as new data continuously appends to the same partition/region — DynamoDB's on-demand auto-split takes minutes to react, so a bursty writer can throttle itself out. The standard fix is a GSI (or HBase-style secondary index) that decouples the write-optimized base table schema from the timestamp-range read pattern, accepting the extra storage/cost of denormalization.

Postgres table has grown to billions of rows and query latency is climbing — is periodically deleting old rows the right long-term answer?

Periodic deletes themselves cost CPU/IOPS and can trigger incidents at scale; a better pattern is periodic (e.g. monthly) sharding/partitioning so entire old shards/partitions can simply be dropped without touching live traffic.

How would you scale a single-writer Postgres/Aurora setup across new geographic regions when Aurora only supports one writer instance globally?

Two directions floated: run fully independent regional databases with cross-region replication to keep them in sync (if global consistency is required), or stand up region-specific data centers and route users to their nearest one by geography — the right call depends on whether cross-region consistency is actually a hard requirement.

Zero-downtime approach for dropping a MySQL column from a huge table?

Prefer not deleting the column at all unless it's large; if removal is unavoidable, `ALGORITHM=INSTANT` (MySQL 8+) combined with a phased rollout ensuring nothing still reads the column is the recommended path.

Debouncing (single-flighting) a cache-miss request works on one server — how does it work once you have multiple servers that could all independently miss the same key at once?

Either centralize the debounce state (shared lock/marker) across servers, or simply relax the constraint — it's often acceptable to leak one origin request per server rather than guaranteeing exactly one across the whole fleet.

Debouncing a Redis-backed cache read introduces a consistency tradeoff — what exactly are you trading away?

You're already eventually-consistent the moment you introduce a cache; debouncing additionally trades a small amount of staleness for a large reduction in redundant origin calls during a spike (protecting against thundering herd), not a fundamentally different consistency guarantee.

How do you prevent an intentionally-crafted public API request from bloating a Redis-backed session cache (DoS via cache-key flooding)?

Validate the incoming key against the upstream source of truth before writing to cache where feasible; if that would overload the upstream, apply column/entry compression and short TTLs to bound the blast radius rather than eliminating it entirely.

Best place to store notification/email HTML templates — Postgres column, blob storage (S3), or NoSQL?

Store only the minimal structural "bare bones" needed to reconstruct the template (fields, not the full rendered HTML) in Postgres, and materialize/render on demand at the backend — full embedded HTML bloats storage (Postgres TOASTs large fields to disk) for content that's read far less often than it's stored.

Redis in cluster mode doesn't support client-side tracking, forcing a pull-based TTL refresh instead of push invalidation via a change-event bus — is that intentional?

Best guess from the community (including a direct response from the course instructor): Redis prioritizes simplicity as a core design philosophy over supporting every advanced feature in every mode; DiceDB was cited as a project explicitly trying to close this gap with built-in reactivity.

One generic Kafka topic carries every change to a shared "contract" record, but each downstream microservice only cares about 5–10% of the messages — how do you avoid every service filtering nearly everything it consumes?

Options ranged from a shared filtering library embedded in every publisher (push filtering upstream) to a dedicated intermediate filter/fan-out layer that re-publishes only the relevant subset per downstream topic — the latter trades an extra hop for keeping publishers dumb and downstream consumers lean.

A hashtag-generation service risks a single Kafka partition getting overwhelmed by one viral hashtag, starving other hashtags sharing that partition — how do you fix it without losing per-hashtag batching efficiency?

Suffix a random number (e.g. 1–100) onto the hot key purely for partitioning purposes, spreading a single logical hashtag's events across many physical partitions — the same write-sharding trick used for DynamoDB hot partitions.

Setting up disaster recovery for Kafka using MirrorMaker 2 (MM2) — topics, messages and active consumer offsets replicate fine, but inactive consumers and some offset translations don't sync. Any alternative?

No fully satisfying answer emerged in-thread; it was flagged as a known MM2 limitation rather than a misconfiguration, with the active-passive DR use case being particularly exposed to it.

Why is Kafka generally avoided for financial/payment transactions given its at-least-once delivery model?

The risk isn't Kafka specifically but duplicate delivery in general — the fix is exactly-once *processing* semantics achieved by enforcing an idempotency key (e.g. `payment_id`) as a unique/primary-key constraint at the write layer, so a duplicate delivery simply fails its insert rather than double-processing.

How do you handle out-of-order message consumption when reading from a multi-partition Kafka topic?

Kafka only guarantees ordering within a single partition, not across partitions — if cross-message ordering matters, route related messages to the same partition via a consistent key; consumer-side re-ordering (e.g. merging per-partition sorted queues) is possible but throttles throughput toward single-threaded.

In a Slack-style chat system, if client A connects to edge server A1 and client B connects to edge server B2, how does a message from A actually reach B's server?

Each edge server subscribes to a Redis pub/sub channel per user/channel it's handling as part of the connection handshake; a message is persisted, then published to the relevant channel, and Redis fans it out to whichever edge server(s) currently hold a live subscription for the recipient.

A per-pod Redis subscriber thread (used to forward pub/sub messages to the pod holding the relevant WebSocket connection) silently dies after a few hours — how do you make it self-healing?

Recommended modeling it as a supervised actor: a "chat supervisor" that spawns per-chat-room actors and monitors/restarts a dead subscriber automatically, rather than relying on a bare long-running coroutine with no watchdog.

Does subscribing a million individual per-user DM channels on Redis pub/sub actually scale, or does it fall over with that many channels?

Pattern-based subscriptions in Redis can handle very large channel counts without issue in principle; each edge server maintains only a handful of connections to Redis itself (not one per subscribed channel), so the bottleneck is elsewhere in the system, not channel count.

If a user is marked offline after a 30-second missed-heartbeat window, how does their "back online" status propagate promptly to friends already viewing their (stale) offline status?

Real-time propagation requires push (inform all watchers the instant the heartbeat resumes) rather than polling; polling every N seconds is simpler but introduces up to N seconds of staleness by design — the choice is purely an SLA/complexity tradeoff, not a technical constraint.

Is a persistent HTTP/1.1 keep-alive connection functionally equivalent to a WebSocket for real-time communication?

No — keep-alive still requires a new request/response per message (server can't push unprompted), while WebSocket is a genuinely bidirectional, full-duplex channel; keep-alive just avoids TCP/TLS handshake overhead between separate requests.

Should an SSE endpoint live behind the same API gateway/microservice as everything else, or get its own dedicated service?

No strong consensus reached, but the discussion leaned toward isolating long-lived connection types (SSE/WebSocket) into their own service so they can be scaled and rate-limited independently from short-lived request/response traffic.

Two microservices each own a table, and you need to join across them for a UI that paginates 5,000+ rows without hammering the owning service per-row — what's the pattern?

Three named options: API composition (in-memory join across live calls to each owning service — simplest, weakest at scale), CQRS with a dedicated read-replica/materialized view kept in sync via events, or GraphQL as a composition layer over the same underlying services. Choice depends entirely on data volume and consistency requirements — no single "always correct" answer.

For a social-network "who liked what" feature spanning User, Item, and Like microservices — which service owns which query?

Keep queries close to the data they primarily read: the Like service resolves "liked items for a user," then calls out to Item/User services as needed for enrichment — avoid centralizing cross-cutting queries in a service that doesn't own the core data.

CDC (via Debezium-style binlog tailing) vs. an ORM-level "on save" callback publishing to Kafka — which is the right way to fan out a news-feed-style update?

A real, unresolved debate in the thread: CDC advocates argued for letting the database's own change log do the work rather than leaking rows into application code; the counter-argument was that CDC ties you to a specific DB's binlog format, adds a whole separate operational surface (compatibility, SRE ownership), and for use cases that can tolerate occasional missed/duplicate events (a news feed, not a ledger), a simple async ORM-callback-to-queue publish with periodic backfill is dramatically simpler and was reported working at production scale.

What's the standard way to handle the "noisy neighbour" problem in a multi-tenant system where one client's traffic degrades service for everyone else?

Two complementary mechanisms: rate limiting to cap any single tenant's resource draw, and "swimlanes" — reserved/isolated capacity for tenants who need guaranteed throughput, usually offered as a paid tier since it costs more to provision. Real systems tend to combine both rather than picking one.

Should you use 2PC/distributed transactions or SAGA/event-driven for coordinating an e-commerce order service with an inventory service?

General guidance leaned toward SAGA/async for this specific case: inventory is typically "blocked" (reserved, not deducted) at order time, with the actual deduction and reconciliation happening asynchronously afterward — full 2PC synchronous transactions across services are usually avoided due to their blocking, tightly-coupled failure modes.

How do you detect and route around "hot nodes" in a reactive load balancer without a single centralized Redis tracking connection counts becoming its own bottleneck at ~1M TPS?

Pointed toward Twitter's deterministic aperture algorithm (client-side load balancing that delegates load-awareness to clients rather than a centralized store) as the reference architecture, plus a broader principle: production load balancers (NGINX, Envoy, HAProxy) avoid writing to any external database at all, favoring in-memory, serializable state machines per upstream connection instead.

Designing a distributed rate limiter that works correctly across multiple regions — what's the hard part?

No silver bullet surfaced; the core tension is always the same — a truly global, strongly-consistent counter serializes every request through one place (defeating the point of being distributed), while a per-region approximate counter risks over-admitting slightly past the intended global limit. Redis sorted sets were discussed as one sliding-window implementation technique, alongside token-bucket variants.

Is there a CPU-utilization-based backpressure mechanism (beyond JVM-specific ones) for protecting nodes in a distributed system from CPU-bound overload?

No off-the-shelf generic answer emerged; JVM-level backpressure tools were the closest known examples, with the broader pattern being to cap concurrent in-flight work per node rather than relying purely on external health checks.

Is it a good idea to run production databases inside Kubernetes pods?

Strong pushback from someone who'd lost data doing exactly this: databases are tuned for proximity to native hardware, and the pod abstraction layer undermines that — expect to burn significant extra tuning time getting a DB to behave correctly under k8s, and many production setups still run databases on bare EC2/bare-metal rather than in pods.

What's the actual difference between rolling deployments and canary deployments — they sound similar?

Rolling deployment replaces the existing fleet in place, one instance at a time, with no way to route specific user segments to the new version; canary deployment stands up a small parallel fleet on the new version and explicitly controls what fraction of traffic (or which users) reach it, making rollback as simple as stopping the traffic split.

Are AWS Lambda functions a good fit for a distributed task scheduler when some tasks run long (Lambda caps out at 15 minutes)?

Not for long-running tasks — Lambda's execution cap rules it out; AWS ECS/Fargate was recommended for long-running work, and dkron was repeatedly recommended as a purpose-built distributed scheduler alternative to hand-rolling one on serverless infra.

Early-stage startup, ~10K expected users in 6 months — worth dockerizing the app, MongoDB, Redis and Elasticsearch now, or premature?

10K users is small enough that scalability isn't the real driver; the community leaned toward "do it for the learning experience" rather than a hard operational necessity at that scale, since cost/time weren't cited as constraints in this case.

Designing an eBay-style live bidding system — should bids go through Kafka (eventually consistent, but resilient) or straight to the DB (consistent, but couples the write path to bidding throughput)? And WebSocket or SSE for pushing live bid updates to viewers?

Bids are recommended to land directly in Redis rather than routing through Kafka first — Kafka can still sit downstream for durability/analytics, but the hot path favors low latency over async durability. For live updates, WebSocket was favored over SSE specifically because SSE runs into the browser's ~6-connection-per-domain HTTP limit, which becomes painful once a page needs several concurrent SSE streams.

Designing a Google-News-style aggregator: how does content actually get from publishers into the feed, and how is it personalized?

Modeled as two halves: an ingestion pipeline (CDC + Kafka fan-out as publishers push new articles, landing in a database first) feeding a separate recommendation/ranking system that curates per-user "reader space" views — explicitly distinct from a social-graph newsfeed since there's no follow-graph driving distribution, closer in shape to a content-recommendation problem like TikTok/YouTube Shorts.

"Distribute 6 million free items in 10 minutes via a button-click, guaranteeing no one gets more than one and no more than 6M total are promised" — how do you avoid over-promising under massive concurrent load?

Proposed approach: pre-populate 6M sharded rows representing available inventory, and use `SKIP LOCKED`-style claiming so concurrent requests each grab a distinct row and move on rather than queueing behind a single contended counter.

Designing malware-scanning-as-a-service (VirusTotal-style) — any pointers on where to start?

Flagged as a genuinely under-documented space compared to more common interview prompts; no concrete architecture reference was surfaced in-thread, just acknowledgment that it's a legitimately hard, less-rehearsed system design question.

Designing WhatsApp/Zoom-style video conferencing — where's the actual complexity versus one-way live streaming (which has plenty of public reference material)?

The instructor's own take: there's genuinely little public reference material for real conferencing (as opposed to streaming) architectures — the recommended path was to study HLS, Akamai's config, and AWS MediaConvert, then reason from WebRTC signaling primitives (TURN/STUN, SFU projects like Janus/Kurento) rather than expect to find a ready-made blog post.

In a system design interview, the interviewer kept diving deep into the first component discussed instead of letting the candidate cover breadth first — how do you keep control of the pace?

Explicitly narrate the plan out loud ("let's cover the high-level shape first, then I'll go deep wherever you want") and hold that boundary — most of the derailment comes from ambiguous communication inviting more clarifying questions, not from the interviewer being unreasonable. Practicing through real mock interviews was repeatedly cited as the actual fix, more than any framework.

Best resources for finding recently-asked system design interview questions at specific companies?

Interview-experience sections on Glassdoor/AmbitionBox and GeeksforGeeks were the most commonly cited sources; no single canonical "question bank" was agreed on.

Struggling with system design because of feeling like every topic requires understanding several prerequisite topics first ("half-knowledge" from quick-bytes-style content is actively hurting in interviews) — DDIA cover-to-cover first, or dive into course material directly?

The most detailed community response argued against trying to fully understand everything before starting: identify specifically where you get stuck in real interviews or practice, trace that gap "up a level of abstraction" until you hit a concept you are comfortable with, then work back down — treating perfect prerequisite knowledge as a trap rather than a genuine requirement.