Zipkin & OpenTelemetry & Micrometer Tracing

Distributed tracing is the technique of tracking a request as it moves through multiple services, components, and systems. It helps answer: "Where did this request spend time, and what service caused the delay?"

In systems with many services, a single request may cross dozens of components. Without tracing, debugging latency and failures becomes extremely difficult.

A trace is the complete end-to-end story of a request or workflow across distributed components. It is usually represented as a tree of spans.

A span is a single unit of work inside a trace, such as an HTTP call, database query, or method invocation. Each span has metadata like start time, duration, tags, and logs.

A trace is the whole journey. A span is one step or operation inside that journey.

Zipkin is an open-source distributed tracing system. It collects and visualizes traces to help analyze latency and dependencies.

OpenTelemetry is an open standard for observability signals, including traces, metrics, and logs. It aims to provide a vendor-neutral API and SDK ecosystem.

Micrometer Tracing is a small tracing facade used in Spring-based applications. It integrates with OpenTelemetry or Brave and supports common tracing instrumentation patterns.

OpenTelemetry provides instrumentation and data generation. Zipkin is often the backend/backend collector and visualizer for traces.

Instrumentation is the code added to applications to emit telemetry: traces, metrics, and logs. Without instrumentation, tools cannot observe the application.

Telemetry is operational data collected from a system, such as traces, metrics, and logs. It is used to monitor, debug, and analyze software behavior.

A trace ID uniquely identifies one end-to-end request across multiple services. All spans belonging to the same request share the same trace ID.

A span ID uniquely identifies one unit of work inside a trace. It distinguishes one step from another.

A parent span is the span that started a child span. This creates hierarchical relationships and helps reconstruct the call tree.

A child span is a span created from a parent span, usually representing nested work. Example: a controller request creates a span, which then creates downstream database and HTTP spans.

Propagation is the process of carrying trace context across process or service boundaries. It allows a trace to continue when a request moves to another service.

Without propagation, each service would create independent traces. That would make end-to-end request analysis impossible.

Trace context is the set of metadata used to continue a trace, such as trace ID, span ID, and trace flags. It is passed in headers like traceparent and tracestate.

A service graph shows how services communicate with each other. It is often derived from distributed traces and network metadata.

Latency is the time spent before work completes. In tracing, latency is often visualized as span duration across services.

A downstream call is a request from one service to another service. This is a common source of trace spans.

An upstream call is a request that a service receives from a client or another service. It often becomes the root of a trace.

A root span is the top-level span for a request or workflow. It usually represents the start of a trace.

A client span represents an outgoing request from a service to another service or dependency. Example: a REST call to another API.

A server span represents the receiving side of a request. Example: a web server handling an HTTP request.

They show the full request lifecycle and help locate where delays happen. A slow response might be due to the client waiting or the server processing slowly.

An async boundary is a point where execution continues later, often in a different thread or queue. Tracing across async boundaries is a common challenge.

Because context can be lost when tasks continue later without proper propagation. This is one of the most common tracing pitfalls.

Correlation is the ability to connect events or logs to the same request or transaction. Tracing data helps correlate logs, metrics, and spans.

They automatically instrument common libraries and frameworks. Examples include HTTP clients, server frameworks, SQL drivers, message brokers, and gRPC.

The API is the programming interface used by developers to create spans and metrics. It is framework-agnostic and vendor-neutral.

The SDK implements the API and handles sampling, processors, exporters, and resource configuration. It is responsible for actual generation and publication of telemetry.

An exporter sends telemetry to a backend such as Zipkin, OTLP, Jaeger, Prometheus, or vendor systems. It is the bridge between app instrumentation and observability backends.

OTLP stands for OpenTelemetry Protocol. It is the standard way to send telemetry data to collectors and backends.

A collector accepts telemetry from applications and forwards it to one or many backends. It can also transform, filter, or batch data.

Micrometer is a metrics facade used widely in the Spring ecosystem. It provides a vendor-neutral API for exposing metrics to backends like Prometheus and Atlas.

Micrometer Tracing builds on Micrometer’s philosophy of a simple API for instrumentation. It provides integration for tracing with providers such as OpenTelemetry or Brave.

Brave is a Java tracing library that can be used for instrumentation and propagation. Micrometer Tracing can integrate with Brave or OpenTelemetry.

Micrometer Tracing is a higher-level tracing abstraction for Spring applications. OpenTelemetry is a broader, vendor-neutral observability standard with a more general ecosystem.

OpenTelemetry is increasingly the standard across multiple languages and platforms. It is a more future-proof choice in polyglot systems.

A span attribute is a key/value pair attached to a span. Examples include customer ID, region, HTTP method, or database name.

A span event is an annotation-like timestamped event inside a span. It helps capture meaningful moments during a request.

A log record can be attached to a span to provide contextual debugging details. It is often used alongside a span to capture state changes.

A sampling policy decides which traces are recorded and which are dropped. This is necessary to control cost and volume in production systems.

Head-based sampling decides at the start of a trace whether to record it. This is common in tracing systems and works well for system-wide sampling.

Tail-based sampling decides after the whole trace is known whether to keep it. This is often better for capturing rare high-value traces.

Tracing at full volume can become extremely expensive. Sampling reduces overhead while preserving representative diagnostics.

A trace waterfall is a visual timeline of spans in a request. It shows nested and overlapping durations in a readable form.

A service dependency map shows which services call which other services. It is often derived from span relationships and service names.

A request path is the path or route a request follows through the system. In tracing, it is often represented in span names and tags.

HTTP instrumentation records inbound and outbound HTTP requests. This is one of the most common ways to observe service-to-service traffic.

Database instrumentation records queries and spans around database calls. It helps identify slow SQL or query bottlenecks.

This records producer and consumer work around queues or topics. It helps correlate asynchronous messaging flows across services.

Baggage carries application-specific context across process boundaries. It is distinct from trace metadata and can carry business or security context.

It is the propagation of key/value pairs along the trace path. Examples include tenant ID, user ID, or request origin.

No. Trace context identifies the trace, while baggage carries user or operational context. Both can coexist in propagation headers.

A span kind tells whether a span is client, server, producer, consumer, internal, or unspecified. This helps tell semantics of the operation.

A server span represents work performed as the server side of a received request. It is a semantic category used for standard instrumentation.

A producer span represents work that sends a message to a queue or topic.

A consumer span represents work that processes a message from a queue or broker.

It is built around spans and trace context with a parent-child model. It is designed to be generic across platforms and languages.

A tracer is an object used to start spans and create telemetry. It is the primary API entry point for instrumentation code.

A meter is the API object used to create metrics instruments. Metrics and tracing are separate but often used together.

Metrics show aggregate patterns and trends. Traces explain individual request behavior and latency components.

A trace-based dashboard shows key traces or service slices by latency, error rate, or workload. These are often used during incident response.

A span name should be concise and semantically meaningful. For example: GET /api/orders/42 or orders-service.findOrder.

Good naming makes trace visualization and debugging easier. It reduces confusion and speeds root-cause analysis.

In many ecosystems, they are effectively similar. OpenTelemetry typically uses attributes; Zipkin historically used tags.

A trace query is a request to search traces by parameters like service name, duration, or span tags. This is how users find relevant traces in a backend UI.

A service name identifies the service generating the span. It is often a logical application name like orders-service.

A resource describes the entity producing telemetry, like service name, version, deployment environment, and pod metadata. It helps distinguish spans from different services.

They make traces understandable in large systems. Without them, you may see many generic spans without context.

For HTTP, the trace ID is usually transmitted as part of trace context propagation headers. This allows the downstream service to continue the same trace.

Span linking allows explicit relationships between spans without a hierarchical parent-child relationship. This is useful for asynchronous or event-driven systems.

A trace tree is the structure of parent-child spans in a single trace. It resembles a call graph for one request.

A root cause is the primary component or dependency responsible for a failure or latency. Tracing helps isolate that cause.

Because thread-local context often holds the active span. If it is lost across async boundaries, instrumented calls can no longer correlate properly.

MDC stands forMapped Diagnostic Context. It is common in Java logging and often used to attach trace IDs to logs.

Logs attach correlation IDs so operators can find the same request across services. This helps connect logs to traces.

Log correlation is the linking of log entries to a trace or request using IDs like trace ID and span ID. This makes incident analysis more effective.

A microservice is a small independently deployable service with a focused responsibility. Tracing is especially important in microservice architectures.

Because requests move across service boundaries and networks. Failure and latency can be caused by downstream dependencies rather than local logic.

A saga is a sequence of distributed operations, often with compensation logic. Tracing helps understand where each step failed or slowed down.

Timeouts are safety measures to prevent requests from hanging forever. Tracing helps identify whether slow dependencies are due to timeouts or overloaded services.

A retry storm happens when many services retry failed dependencies at once. This creates amplification and can worsen outages.

By showing which service generated repeated downstream calls and which dependency was failing. It turns a distributed failure into an actionable pattern.

They carry the trace information required to continue tracing across process boundaries. This is the fundamental mechanism behind distributed tracing.

Semantic conventions define standard names for spans, attributes, and operations. Examples include HTTP, database, messaging, FaaS, and RPC conventions.

They enable interoperability across different instrumentations and backends. Without them, traces are harder to compare or analyze.

Vendor-neutral means the instrumentation and API are not tied to a single vendor backend. This allows you to move between systems without rewriting your app.

It reduces lock-in and simplifies multi-language and multi-tooling observability. This is valuable in heterogeneous environments.

A backend is the storage and visualization platform for telemetry, like Zipkin, Jaeger, or Grafana Tempo. It receives exported spans and helps query them.

Zipkin’s UI shows traces, dependencies, and dependencies graph. It is commonly used to inspect latency and service relationships.

A dependency graph shows service-to-service connections and communication frequency. It can reveal hotspots or unusual traffic patterns.

A trace query searches by span name, service name, duration, or time window. It helps find relevant traces for debugging.

Zipkin is a backend and visualization tool. OpenTelemetry is the instrumentation and standardization layer.

It helps instrument Spring MVC, WebFlux, REST clients, and asynchronous call stacks. It can integrate with OpenTelemetry or Brave and expose spans to backends.

Actuator exposes operational endpoints and health/metrics data. It integrates with observability tools and often provides tracing metadata.

Spring Cloud Sleuth was an earlier Spring tracing solution. Micrometer Tracing is the modern evolution of this approach.

Spring Boot 3 adopted Micrometer and OpenTelemetry-friendly instrumentation patterns. This reflects the industry move toward standard observability APIs.

It carries the trace context required to continue a distributed trace. A tracing library usually creates this automatically for HTTP propagation.

It carries vendor-specific information and additional metadata related to the trace. It is often used for implementation-specific context.

A service mesh is an infrastructure layer that connects and governs service-to-service communication. It often provides mTLS, retries, and observability at the network layer.

A service mesh may generate or enrich telemetry, including traces and logs. This can help see network-level behavior in addition to application-level spans.

eBPF tracing captures kernel and network-level behavior with low overhead. It complements application instrumentation and can surface OS-level performance issues.

Distributed debugging involves tracing a request through multiple services to find where it failed or slowed down. This is a key use case for observability systems.

An SLO is a service level objective: a target for service performance or reliability. Tracing helps measure whether SLOs are being met.

It provides the underlying latency, failure, and dependency data needed to evaluate SLO compliance. It can show which service is causing end-user impact.

An SLA is a contractual agreement around service availability or performance. Tracing helps validate whether actual behavior meets that agreement.

A dependency bottleneck is a slow or overloaded downstream service causing delays. Tracing helps spot the exact dependency responsible.

A queue backlog is a large number of waiting messages or jobs. It often causes long latencies and trace spans that show long queue times.

Application latency is the time spent in business logic or framework code. Network latency is the time spent communicating across boundaries.

Because a slow request may be due to application code, network slowness, or a downstream service. Tracing helps break these apart.

A latency budget is the acceptable time spent in each component or dependency. It is often used in design and SLO planning.

An annotation is a user-defined marker in a trace, often added for important milestones. It helps explain request flow and status transitions.

Time is critical because every span carries start and end times. This allows ordering and duration analysis across components.

Clock skew is the difference between system clocks across machines. It can distort trace timestamps when services are not time-synchronized.

Because distributed traces rely on timestamps to order events across services. This is critical for accurate analysis.

NTP is the Network Time Protocol used to synchronize clocks across hosts. It helps reduce skew in distributed systems.

Because machines may drift or be temporarily unsynchronized. This can make timeline ordering appear inconsistent.

End-to-end latency is the total time from request start to completion across the whole system. It is the most user-visible metric.

High-cardinality tags are tags with many unique values, such as account IDs or session IDs. These can make telemetry storage expensive and noisy.

They can create huge amounts of telemetry and reduce backend performance. This is especially relevant for production tracing.

It is a condition where too many unique tag values produce an unmanageable number of traces or span records. This can degrade observability systems.

Span limits ensure instrumentation does not generate excessive telemetry. They protect the backend from excessive storage and processing costs.

A retention policy determines how long trace data is kept in storage. This balances history needs with cost.

It includes sampling, selective export, batching, and compression. These reduce cost while preserving the important observations.

A collector pipeline is the flow of telemetry through a collector: receive, process, enrich, export. This is a key part of modern observability architecture.

It often uses a binary protocol with gRPC for efficient transfer of traces and metrics. This is common in production systems.

It uses standard HTTP payloads to send telemetry to receivers. This is convenient for integrations and debugging.

A span processor handles spans before export, such as sampling or enrichment. It sits between instrumentation and exporter.

A span exporter sends spans to a backend or collector. It is the final step in the telemetry pipeline.

A resource detector attaches environment metadata to telemetry. Examples include host, service name, and cloud metadata.

Attach environment, version, and deployment identifiers as resource attributes. This helps compare traces before and after changes.

Release correlation is the mapping of traces to app versions and deploys. This is useful during incident investigations and rollback analysis.

Because a bad release may cause new latency or errors. Observability becomes much more actionable when traces include version metadata.

A hot path is a frequently executed or performance-critical code path. Tracing helps identify hot paths that contribute disproportionate latency.

Request fan-out is when one request triggers multiple downstream calls. It creates a more complex trace tree and often more latency.

Because it explains how a single request can cause a cascade of dependent calls. This is common in microservice systems.

A slow dependency path is a series of spans in a trace that collectively consume enormous time. It often reveals a network or DB bottleneck.

A queueing delay is the wait time before a job or request is processed. It is often visible as long spans before actual work begins.

Time-to-first-byte measures how quickly the system starts sending response data. It is a useful latency signal in HTTP-based systems.

CPU bound workloads are limited by processor time. I/O bound workloads are limited by database, network, or storage latency. Tracing helps distinguish these patterns.

Metrics are usually aggregated and cheaper to retain. Traces are much more detailed and often require shorter retention.

Aggregation summarizes many events into a metric. Tracing preserves per-request detail and causal relationships.

It is the tracing of a request or workflow that spans multiple services or components. This is the core concept behind distributed tracing.

Distributed tracing focuses on technical execution flow. Business process tracing focuses on workflows and domain operations rather than infrastructure.

A workflow trace captures a user/business operation across many technical steps. It can represent a more domain-level story than a pure technical trace.

Because developers in different languages follow the same conceptual model. This leads to better cross-team understanding and easier troubleshooting.

It is when telemetry lacks enough context, strong naming, or proper instrumentation. Low-quality traces are difficult to diagnose even though the system is technically generating them.

Standards reduce inconsistencies and make observability easier to reason about at scale. They are especially important in organizations with many teams.

A dead-letter queue stores messages that fail processing repeatedly. Tracing these flows helps identify failure patterns and root causes.

It connects producer and consumer spans and provides visibility into message flow. This allows diagnosing bottlenecks and failures in asynchronous systems.

A sidecar proxy is a process running next to a service, often in a service mesh. It can inject or collect telemetry for network traffic and request routing.

Span flattening refers to transforming a hierarchical trace into a simpler or denormalized form. This is often done for UI or analytics use cases.

It is the propagation of trace context between different processes or services. This is the core of distributed tracing.

Strong consistency here means that all events are ordered and records are not silently lost. This is often a backend concern for trace storage.

Telemetry may be available with some delay or duplication. This is acceptable in many observability systems but can affect debugging.

A trace DAG is a directed graph of spans showing dependencies and causal ordering. This is especially useful when there are fan-outs and concurrency.

Fan-out means one request triggers many downstream calls. Fan-in means many calls converge to a single dependency or service. Both patterns produce complex traces.

It is the linking of a trace with aggregated metrics like latency and error rate. This gives contextual insight into the behavior of a request.

Because it connects user experience, service behavior, infrastructure load, and deep technical bottlenecks. This reduces time to detect and resolve incidents.

Observability maturity describes how well a team can diagnose and understand system behavior. It often correlates with instrumentation quality, dashboards, alerting, and ownership.

Monitoring answers "Is the system healthy?" Observability answers "Why is it unhealthy and what is happening?" Tracing is a key part of the latter.

It means instrumenting systems in a way that makes behavior visible and queryable. This is a central idea behind OpenTelemetry and tracing.

A trace-based alert can fire when latency or error patterns exceed thresholds in a certain request path. This is more precise than broad infrastructure alerts.

It is the rule used to define which span becomes the root for a trace. This is often based on inbound request entry points or trigger events.

It occurs when spans or tags explode in unique combinations and overwhelm storage. This is a major challenge in high-traffic systems.

A best-effort trace is a trace that may be sampled or partially captured when resources are constrained. This is often acceptable in large systems.

More instrumentation gives more context but also increases cost and complexity. Teams must balance observability value against overhead.

Because latency or errors often matter only when tied to a user journey or business action. This context is often propagated through baggage or request metadata.

It is the combination of traces and failure/error markers. This helps correlate slow or broken requests with underlying exceptions.

Request normalization is the process of mapping different request variants to a common operation name or path. This improves trace grouping and queries.

They show the causal chain of service interactions and enable focus on the slowest or failing span. This reduces guesswork during incident response.

Kubernetes introduces many moving parts: pods, services, ingress, and sidecars. Tracing helps map requests across those moving parts and dependency boundaries.

OpenTelemetry uses W3C trace context and a standard propagation format. This is now widely adopted across languages and systems.

W3C trace context is a standards-based propagation mechanism for distributed tracing. It defines headers that carry trace identifiers and parent information.

It allows interoperable tracing across vendors, libraries, and platforms. This is one of the most important changes in observability history.

Because Spring apps often run in distributed systems and need standard observability. OpenTelemetry gives them a vendor-neutral, modern instrumentation story.

They are not competing in the same dimension:

• OpenTelemetry is the instrumentation and standards layer
• Zipkin is a tracing backend and visualization tool
• Micrometer Tracing is a Spring-friendly abstraction over tracing instrumentation

Together, they provide the observability foundation for modern distributed systems.