Spring Kafka/Messaging

Asynchronous communication where producers send messages and consumers process them.

Decouples services, improves scalability, and enables asynchronous workflows.

A distributed event streaming platform for high-throughput, durable message processing.

Spring project providing convenient Kafka producer/consumer integration.

Named stream/category where Kafka records are published.

Ordered sub-log of a topic enabling parallelism and scalability.

Position of a record within a partition log.

Client that publishes records to Kafka topics.

Client that reads records from Kafka topics.

Set of consumers sharing work of topic partitions.

Each partition is consumed by at most one consumer in a group at a time.

Kafka server node storing and serving topic partitions.

Number of copies of each partition across brokers.

Improves fault tolerance and availability.

Replica handling reads/writes for a partition.

Replicas syncing from leader for redundancy.

Replicas sufficiently caught up with leader.

Key, value, headers, timestamp, topic, partition, offset metadata.

Control partitioning and preserve per-key ordering.

Order is guaranteed within a partition, not across partitions.

Main helper for sending messages in Spring applications.

Typically with @KafkaListener.

Annotation marking method to receive Kafka messages from topics.

Config object controlling listener container behavior.

Converts byte data from Kafka into Java objects.

Converts Java objects to bytes for Kafka transmission.

JSON, Avro, Protobuf, plain strings, binary custom formats.

Initial broker addresses for Kafka client connection discovery.

Broker acknowledgment level for write durability behavior.

0, 1, all (or -1 equivalent).

Leader waits for all in-sync replicas before acknowledging.

Messages may be lost, but never redelivered.

Messages are not lost (under expected conditions) but may be duplicated.

Processing/output appears once logically, requiring transactional/idempotent patterns.

No, it needs careful end-to-end design.

Periodic automatic offset commits by client.

Offsets may commit before processing truly succeeds.

Application explicitly commits offsets after successful processing.

Better control over processing and failure semantics.

Redistribution of partitions among consumers in a group.

Consumer join/leave, crashes, subscription changes, metadata changes.

Pause processing and increase latency/duplicate risk.

Topic where failed/unprocessable messages are routed.

Preserve problematic events for analysis/replay without blocking pipeline.

Failed messages retried via delayed/retry topics before DLT.

Consumer logic safe against duplicate message processing.

At-least-once delivery means duplicates can happen.

Record consistently failing processing due to bad data/schema/business rule.

Route to DLT with diagnostics instead of infinite retries.

Metadata key-value pairs attached to records.

Trace IDs, schema version, tenant, source system identifiers.

Difference between produced and consumed offsets.

Indicates backlog growth and processing health.

How long records are kept (time/size policy).

Retention mode keeping latest record per key.

State/event-upsert streams where latest key state matters.

Assuming no duplicates and writing non-idempotent consumers.

Track throughput, lag, error rates, retry/DLT counts.

TLS, authentication, ACL authorization, secret management.

Design for retries, duplicates, and failure from day one.

Feature ensuring producer retries don’t create duplicates on broker log per session.

Set producer idempotence configuration and compatible reliability settings.

Producer that groups writes (and optionally offsets) into atomic transactions.

Consumers read only committed transactional records.

Consumers may read uncommitted/aborted transactional records.

Producer identity enabling transactional guarantees across restarts/sessions.

Write domain change + outbox row in DB transaction, publish asynchronously to Kafka.

Avoid dual-write inconsistency between DB and Kafka send.

DB commit succeeds but publish fails, or publish succeeds but DB rollback.

CDC streams DB changes; polling periodically reads outbox table.

DB and Kafka transactions are distinct unless explicitly coordinated pattern used.

Attempts multi-resource coordination; often replaced by outbox for simplicity/resilience.

Number of concurrent consumer threads/containers.

Effective parallelism limited by partition count per group.

Extra threads idle for that topic/group combination.

Max records returned in one poll call.

Balance throughput, processing time, and rebalance timeout risk.

Max allowed delay between polls before consumer considered failed.

Consumer removed from group causing rebalance/duplicates.

Failure detection timeout for consumer heartbeats.

Frequency of consumer liveness heartbeats to group coordinator.

Temporarily stop/resume partition fetching for backpressure/control.

Downstream outage, overload, or controlled throttling scenarios.

Centralized listener error handling strategy in modern Spring Kafka.

Configurable error handler supporting retries, backoff, and recovery.

Delay policy between retry attempts (fixed/exponential).

Reduces pressure on failing dependencies over time.

Publishes failed records to DLT after retries exhausted.

Original topic/partition/offset, exception type, stack summary, timestamp, trace ID.

Move offset position to reprocess or skip records intentionally.

Defines when offsets are committed (record, batch, manual, etc.).

Commit per record vs per batch of polled records.

Listener receives list of records for batch processing.

Higher throughput but more complex error granularity.

Drop duplicates/unwanted records before listener logic.

Spring hook for filtering records pre-processing.

Supports backward/forward-compatible deserialization and migration.

Central service storing and validating message schemas.

Strong contracts and evolution controls for event payloads.

New schema can read old data.

Old consumers can read new producer data under defined rules.

Both backward and forward compatibility constraints satisfied.

Compacted-topic record with key and null value indicating delete semantics.

Deserializer/business logic must distinguish tombstones from malformed data.

Stable business key balancing ordering needs and partition distribution.

Uneven key distribution overloads specific partition/consumer.

Better key strategy, more partitions, key salting (carefully), workload redesign.

More partitions improve parallelism but ordering is per key/partition only.

Pattern using reply topics/correlation IDs for pseudo-synchronous workflows.

Low-latency synchronous requirements often better served by HTTP/gRPC.

Kafka Streams has built-in EOS features; plain consumers need explicit patterns.

Route failures through staged retry topics with increasing delays.

Main consumer continues processing newer messages while failed ones retry asynchronously.

Consumer thread retries immediately, potentially reducing throughput.

Infinite retries with no DLT and no observability.

Bounded retries + backoff + DLT + replay tooling.

Reprocess historical or DLT events safely with idempotent logic.

Incidents/data fixes often require controlled reprocessing.

Avoid unsafe polymorphic deserialization that enables gadget attacks.

Trusted packages, explicit type mapping, schema validation, minimal polymorphism.

Authorization rules controlling topic/group operations.

Limit blast radius of compromised credentials/services.

Team can explain delivery semantics and offset strategy per consumer.

Embedded/integration tests with real broker containers + failure scenarios.

They frequently reveal duplication/order and timeout bugs.

Validate producer/consumer schema and semantic expectations continuously.

Spikes indicate upstream regressions or schema/data drift.

Scale consumers based on lag/throughput metrics.

Need partition limits, processing cost, and downstream capacity context.

Optimize for correctness first, then throughput with measured tuning.

Requires coordinated idempotency/transactions across producers, brokers, consumers, and sinks.

External DB/API side effects still need idempotent handling.

Persist processed event IDs/keys to prevent duplicate side effects.

Extra storage/lookups vs correctness under retries/replays.

Relay throughput, lock contention, and ordering guarantees need careful design.

Polling simpler but adds query load/latency; CDC near-real-time but operationally heavier.

Use explicit version fields and compatibility rules per event type.

Breaking field changes without staged consumer migration.

Producers coordinate changes based on downstream consumer compatibility needs.

Standard metadata wrapper (id, type, version, timestamp, trace) around payload.

Consistent observability, routing, governance across topics.

Tracking origin and transformations of data across services/topics.

DLT + triage + fix + controlled replay + postmortem.

Reprocessing should produce predictable results with versioned logic.

Retry paths can alter apparent processing order across partitions/topics.

Keyed partitions + careful retry routing + ordered consumer execution policies.

Retries and downstream slowness consume consumer capacity.

Bound retries, circuit breakers, adaptive pause, retry budgets.

Control poll/processing rate and downstream concurrency limits.

Rebalance strategy reducing stop-the-world partition movement impact.

Reduces disruptive rebalances during brief restarts.

Replica placement across failure domains for resilience.

Can improve availability but risks data loss.

Defines durability threshold for successful writes with stronger ack settings.

False sense of durability if ISR requirement too low.

Compress batches (snappy/lz4/zstd/gzip) to reduce bandwidth/storage.

CPU overhead vs network/storage efficiency.

Wait briefly to batch more records, improving throughput with slight latency cost.

Producer batch memory target influencing throughput/latency profile.

Affects ordering guarantees during retries for some configurations.

TLS+mTLS, SCRAM/OAuth auth, ACL governance, secret rotation, audit logs.

Protect sensitive payload fields end-to-end beyond transport encryption.

Schema classification, minimization, retention controls, deletion workflows.

Immutable logs complicate deletion; use compaction/tombstones and data minimization strategies.

Replication/failover across clusters for DR and geo requirements.

Conflict resolution, ordering, duplication, and failover coordination challenges.

Replicate topics across clusters for disaster recovery/regional distribution.

Targets for publish latency, consumer lag, durability, and availability.

Lag, poll duration, processing latency, retry counts, DLT volume, rebalance frequency.

Send latency, error rate, record retry rate, batch utilization, buffer exhaustion.

Propagate trace context in headers and create spans for produce/consume/process.

Identify scope, isolate failing consumers, control retries, drain/replay safely.

Planned drills for broker outages, lag spikes, schema breakages, replay operations.

Treating messaging as fire-and-forget without contracts, observability, and replay design.

Clear ownership of schemas, semantics, retries, and recovery workflows.

Assume duplicates, delays, and partial failures are normal conditions.

Tune producers/consumers empirically with production-like workloads.

Secure brokers, clients, payloads, and operational access paths end-to-end.

Automate monitoring, DLT handling, replay tooling, and runbooks.

Spring Kafka excellence means predictable correctness and operability at scale.