OpenTelemetry Implementation Guide: Unified Observability for Microservices

How to instrument, collect, and export telemetry data across distributed services using the vendor-neutral standard that's replacing proprietary agents.

By Subeesh Sivanandan, Founder & CEO, Stonetusker Systems · Updated October 2026 · Est. reading time: 14 minutes

Why OpenTelemetry, and Why Now

Most microservices architectures accumulate observability tooling the same way they accumulate technical debt: one vendor agent for APM, a different library for custom metrics, a sidecar for distributed tracing, and a logging framework that predates all of them. Each tool has its own SDK, its own configuration surface, and its own opinion about what data to collect. When you multiply that by 30, 50, or 200 services, the instrumentation layer becomes its own operational burden.

OpenTelemetry changes the economics. Instead of N vendor-specific integrations per service, you get one instrumentation API that produces traces, metrics, and logs in a standardized format. The data goes to a Collector you run, and from there to whatever backend you've chosen. Swap the backend and your application code doesn't change. That's the promise, and for teams that implement it carefully, it actually holds.

The CNCF adoption numbers back this up. OTel is the second-most-active CNCF project by contributor count, behind only Kubernetes. The tracing and metrics specifications hit GA in 2023. The logging specification stabilized in 2025. Major cloud providers (AWS, GCP, Azure) now ship OTel-native SDKs, and every major observability vendor accepts OTLP natively. Waiting for the project to "mature" is no longer a defensible position.

But adoption without planning creates its own category of mess. We've seen teams deploy OTel auto-instrumentation across 80 microservices in a week and then spend three months untangling the resulting data quality problems. The instrument-everything-immediately approach fails for the same reason any big-bang migration fails: you don't discover the edge cases until they're everywhere.

Core Concepts: Signals, SDKs, and the Collector

OpenTelemetry organizes telemetry into three signal types: traces, metrics, and logs. Each has its own specification, its own SDK surface, and its own data model. They aren't interchangeable, and treating them as if they are leads to architectural mistakes that surface months later.

Traces

A trace represents the full journey of a single request through your system. It's composed of spans, each representing a unit of work (an HTTP handler, a database query, a message publish). Spans carry attributes (key-value pairs), events (timestamped annotations), and links (connections to other traces). The trace propagates context across service boundaries through headers, so the database call in Service C can be connected to the API call in Service A that triggered it.

Metrics

OTel metrics are aggregated measurements: counters, histograms, gauges. They differ from Prometheus-style metrics in a few ways that matter operationally. OTel uses a "push" model by default, sending metric data to the Collector, which then exports to your backend. If you're already running Prometheus, the OTel SDK can expose a Prometheus-compatible scrape endpoint, so you don't have to rearchitect your metrics pipeline on day one.

Logs

The OTel logging bridge connects your existing logging library (Log4j, slog, Python's logging, Winston) to the OTel pipeline. It doesn't replace your logger. It enriches each log record with trace context (trace ID, span ID) so you can correlate a log line with the exact trace and span that produced it. This is where the real power of unified observability shows up: you go from "this error log happened at 14:32" to "this error log happened during span X of trace Y, which started when user Z hit the checkout endpoint."

The SDK and the Collector

The OTel SDK lives in your application process. It provides the API for creating spans and recording metrics, and it manages the export of that data. You can export directly to a backend from the SDK, but you almost never should. Instead, export to an OTel Collector.

The Collector is a standalone binary that receives, processes, and exports telemetry. It's where you do filtering, sampling, attribute enrichment, and protocol translation. Running the Collector as a separate process (or as a Kubernetes DaemonSet or sidecar) gives you a configuration surface that you can change without redeploying your application. That operational flexibility is the single biggest reason to adopt OTel, even if you don't plan to change backends.

Auto-Instrumentation: What You Get for Free

OTel provides auto-instrumentation libraries for Java, Python, .NET, Node.js, Go (partially), and several other runtimes. These libraries hook into common frameworks and libraries at startup, generating spans for inbound and outbound HTTP calls, database queries, message queue operations, and gRPC calls without you writing a single line of instrumentation code.

For Java, it's a single JAR agent attached at JVM startup. For Python, it's a wrapper command. For Node.js, it's an SDK initialization at the top of your entry file. The setup takes minutes per service.

What auto-instrumentation captures well:

• Inbound HTTP request spans with method, route, status code, and latency
• Outbound HTTP client calls with full URL and response status
• Database client calls with query type and (optionally) sanitized query text
• gRPC calls with service and method names
• Message queue publish and consume operations

What it doesn't capture: anything specific to your business domain. Auto-instrumentation has no idea what a "checkout" is, what "order processing" means, or which customer triggered a particular request. It sees HTTP handlers and database queries. That distinction matters more than most teams realize during initial setup.

A service with only auto-instrumentation will show you that a request took 340ms and that 280ms of that was a PostgreSQL query. It won't show you that the query was fetching pricing data for 47 line items in a wholesale order for Customer #8823, which is context you need for actual debugging.

Manual Instrumentation: Adding Business Context

Manual instrumentation is where you create custom spans and attach domain-specific attributes. This is the work that turns generic trace data into something an on-call engineer can use at 3 AM.

The pattern in most languages looks roughly like this:

tracer = trace.get_tracer("order-service")

with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", order.id)
    span.set_attribute("order.item_count", len(order.items))
    span.set_attribute("customer.tier", customer.tier)

    # Business logic here
    result = calculate_pricing(order)
    span.set_attribute("order.total_cents", result.total)
    span.add_event("pricing_calculated")

Three rules for manual instrumentation that save time later:

First, instrument at business boundaries, not implementation details. Create spans for "validate order," "calculate pricing," and "reserve inventory," not for internal utility functions. You want the trace to read like a business process, not a stack trace.

Second, attach attributes that help you filter and group. Customer tier, order type, feature flag states, tenant ID in multi-tenant systems. These become the dimensions you'll use in queries and alerts. An attribute you skip now is a query you won't be able to run six months from now.

Third, use span events for significant moments within a span, not separate child spans. "Cache miss" is an event. "Retry attempt #3" is an event. Creating a child span for each of these bloats your trace data without adding useful latency decomposition.

Designing Your Collector Pipeline

The OTel Collector has three pipeline stages: receivers, processors, and exporters. A receiver accepts data in a specific format (OTLP, Jaeger, Prometheus, Zipkin). A processor transforms or filters data in transit. An exporter sends data to a backend. You compose these into pipelines defined in a YAML config file.

A minimal production Collector config looks deceptively simple:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 8192
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  otlp:
    endpoint: tempo.monitoring:4317
  prometheusremotewrite:
    endpoint: http://mimir.monitoring/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

The memory_limiter processor is non-negotiable in production. Without it, a spike in telemetry volume can OOM-kill your Collector, which means you lose observability exactly when you need it most. Set the limit to about 60 to 70% of the container's memory allocation.

Deployment Topologies

Two patterns dominate:

DaemonSet (agent mode): One Collector per node. Applications export to localhost, which is fast and doesn't cross the network. The DaemonSet Collector handles initial batching and can do per-node processing. Good for most workloads. Simpler operationally.

Sidecar mode: One Collector per pod. Useful when you need per-service configuration (different sampling rates per service) or when services produce very different volumes of telemetry. More resource overhead, but more granular control.

Many production setups combine both: a DaemonSet handles initial collection, then forwards to a centralized "gateway" Collector deployment that does tail-based sampling and final export. The gateway pattern is where you put the expensive processing steps that benefit from seeing data from multiple services at once.

Sampling Strategies That Actually Reduce Costs

Unsampled trace data at scale is expensive. A fleet of 100 microservices handling 10,000 requests per second produces millions of spans per minute. Storing all of them is neither affordable nor useful, because most successful requests look identical.

Head-Based Sampling

The simplest approach: decide at the start of a trace whether to keep it. Set a sampling rate (say, 10%) and only 1 in 10 traces gets recorded. The problem? You make this decision before you know whether the trace is interesting. A 10% sample rate means you drop 90% of error traces too.

Tail-Based Sampling

The smarter approach: wait until the trace is complete, then decide. The OTel Collector's tail_sampling processor can evaluate completed traces against rules: keep all traces with errors, keep all traces with latency above the 95th percentile, sample 5% of everything else. This is where the 5x to 10x storage savings come from without losing the data that actually matters.

Tail-based sampling requires the gateway Collector pattern. The gateway needs enough memory to hold trace fragments while it waits for them to complete (the decision_wait parameter, typically 10 to 30 seconds). Plan for this memory when sizing your Collector deployment. Undersizing it leads to incomplete traces being sampled randomly, which defeats the entire purpose.

Semantic Conventions: The Part Everyone Skips

OTel semantic conventions are standardized attribute names for common concepts: http.request.method, db.system, service.name, deployment.environment. They exist so that a dashboard built for Service A also works for Service B, because both services use the same attribute names for the same concepts.

Teams skip this step constantly. One service sets env: prod, another sets environment: production, a third sets deployment_env: prd. Six months later, building a cross-service dashboard requires mapping all three to a common dimension, and by that point there are 40 services with 12 different spellings of "environment."

What to standardize on day one:

• service.name and service.version (set via environment variables or SDK resource attributes)
• deployment.environment (dev, staging, production)
• service.namespace (if you have multiple business domains)
• Your own business-domain attributes, documented in a shared conventions file that lives in your platform team's repository

The resource detector plugins for Kubernetes, AWS, GCP, and Azure can auto-populate infrastructure attributes (pod name, node, region, instance ID). Enable them in the SDK configuration so every span carries infrastructure context without developers remembering to set it.

Enforce conventions at the Collector layer, not just at the SDK. The attributes processor can rename non-standard attributes, and the filter processor can drop spans missing required attributes. Catching these issues in the pipeline is cheaper than discovering them when a dashboard query returns incomplete data during an incident.

A Phased Rollout Plan for Production

Don't instrument everything at once. A phased approach lets you validate your Collector configuration, sampling policies, and backend integration before you're depending on OTel for production incident response.

1. Phase 1: Single Service (Week 1 to 2) Pick a non-critical service with moderate traffic. Deploy auto-instrumentation, set up a DaemonSet Collector, and export to your backend. Validate that traces appear correctly, latency overhead is acceptable (typically under 3%), and the Collector's resource usage is predictable. Document the configuration decisions you make.
2. Phase 2: Domain Boundary (Week 3 to 4) Expand to 3 to 5 services that communicate with each other. This is where you validate cross-service trace propagation. Check that context headers are preserved through your API gateway, message queues, and any async processing. Fix the propagation gaps before going wider.
3. Phase 3: Manual Instrumentation (Week 5 to 6) Add custom spans and business-context attributes to the services from Phase 2. Build at least one dashboard and one alert that uses this business context. Prove that the instrumentation is useful for actual debugging, not just pretty trace waterfalls.
4. Phase 4: Sampling and Gateway (Week 7 to 8) Deploy a gateway Collector. Implement tail-based sampling policies. Measure the cost reduction. Validate that error traces and high-latency traces are kept at 100%.
5. Phase 5: Fleet Rollout (Week 9+) Roll auto-instrumentation to remaining services in batches. Use your documented conventions and Collector configuration as a template. Have the platform team review manual instrumentation PRs for the first few services in each new team.

This timeline assumes a team that's already running some form of observability. If you're starting from scratch, double the time for Phase 1 and 2 to account for backend selection and dashboard building.

Compliance Angle: Observability Meets Audit Requirements

Distributed tracing with OpenTelemetry can serve double duty as an audit trail, if you design for it. SOC 2 Type II requires evidence that you can detect and respond to security events. HIPAA requires audit controls that record who accessed what protected health information and when. Comprehensive observability infrastructure is a prerequisite for meeting these requirements in microservices architectures.

Traces that carry user identity attributes (anonymized or tokenized, not raw PII) and business-operation context can demonstrate to auditors that your system records the provenance of every data access. The Collector's redaction processor can strip sensitive attributes before they reach the backend, satisfying both the "we record access" and "we protect sensitive data" requirements simultaneously.

Keep in mind that audit data has different retention requirements than operational telemetry. You might sample operational traces aggressively (keeping 5% of successes) while routing audit-relevant traces (those involving PII access, payment processing, or admin actions) unsampled to a separate, longer-retention backend. The Collector's routing capabilities make this a configuration problem, not a code change.

• OpenTelemetry
• Distributed Tracing
• OTel Collector
• Auto-Instrumentation
• Tail-Based Sampling
• Semantic Conventions
• Observability
• Microservices
• OTLP
• FinOps