Every Go backend eventually hits the moment where log.Printf stops being enough. A request is slow, but you don't know which handler, database call, or downstream service caused the delay. An error surfaces in production, but the log lines don't tell you which sequence of operations produced it. You need distributed tracing, and OpenTelemetry is the standard way to get it.

OpenTelemetry (OTel) is a CNCF project that provides vendor-neutral APIs, SDKs, and tools for generating telemetry data: traces, metrics, and logs. It's backed by every major observability vendor (Datadog, Grafana, Honeycomb, New Relic) and has become the de facto standard for instrumentation in the cloud-native ecosystem. The Go SDK was one of the first to reach stable status, and Go's explicit error handling and context.Context plumbing make it a natural fit for manual instrumentation.

This guide walks through setting up OpenTelemetry in a Go backend from scratch. We'll start with the manual approach using the OTel Go SDK, then look at how modern frameworks can eliminate most of this work entirely. By the end, you'll have distributed tracing with auto-instrumentation for HTTP and SQL, custom spans for business logic, context propagation across service boundaries, and a local collector with Jaeger for visualization.

Installing the SDK

OpenTelemetry's Go SDK is modular. There's no single package that does everything. You assemble what you need from a set of modules. For a typical HTTP API with PostgreSQL, here's the install:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/sdk/trace \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/otel/trace \
  go.opentelemetry.io/otel/semconv/v1.24.0 \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp \
  go.opentelemetry.io/contrib/instrumentation/database/sql/otelsql

That's eight modules, and this is the minimal set for tracing. Each serves a purpose:

• go.opentelemetry.io/otel: the core API, used to get tracers and set the global provider
• go.opentelemetry.io/otel/sdk: the SDK implementation that processes and exports spans
• go.opentelemetry.io/otel/sdk/trace: the trace pipeline (span processors, samplers)
• go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp: sends traces to a collector over HTTP
• go.opentelemetry.io/otel/trace: the tracing API you'll use for custom spans
• go.opentelemetry.io/otel/semconv/v1.24.0: standard attribute names so every tool speaks the same language
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: auto-instrumentation for net/http
• go.opentelemetry.io/contrib/instrumentation/database/sql/otelsql: auto-instrumentation for database/sql

If you also need gRPC instrumentation, add go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc. If you need metrics, add the metrics SDK and exporter modules. The dependency list grows with each concern.

Configuring the Tracer

OpenTelemetry needs a TracerProvider configured before any spans are created. In Go, you typically set this up in main() or in an initTracer() function called at startup. The provider ties together the exporter (where spans go), the resource (what service they belong to), and the span processor (how they're batched and sent).

Create a file called tracing.go:

package main

import (
	"context"
	"fmt"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer(ctx context.Context) (func(), error) {
	exporter, err := otlptracehttp.New(ctx,
		otlptracehttp.WithEndpoint("localhost:4318"),
		otlptracehttp.WithInsecure(),
	)
	if err != nil {
		return nil, fmt.Errorf("creating OTLP exporter: %w", err)
	}

	res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceName("my-api"),
			semconv.ServiceVersion("1.0.0"),
		),
	)
	if err != nil {
		return nil, fmt.Errorf("creating resource: %w", err)
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter,
			sdktrace.WithBatchTimeout(5*time.Second),
		),
		sdktrace.WithResource(res),
		sdktrace.WithSampler(sdktrace.AlwaysSample()),
	)

	otel.SetTracerProvider(tp)

	shutdown := func() {
		_ = tp.Shutdown(context.Background())
	}
	return shutdown, nil
}

Then call it at the start of main():

func main() {
	ctx := context.Background()
	shutdown, err := initTracer(ctx)
	if err != nil {
		log.Fatalf("failed to initialize tracer: %v", err)
	}
	defer shutdown()

	// rest of your application setup
}

A few things to note. The WithBatcher option buffers spans and sends them in batches, which is what you want in production. There's also WithSimpleSpanProcessor for development, which sends every span immediately but has higher overhead. The AlwaysSample sampler traces every request. In production, you'd switch to TraceIDRatioBased(0.1) or a parent-based sampler that respects upstream sampling decisions.

The resource attributes (ServiceName, ServiceVersion) identify your service in the trace backend. When you have 20 services sending traces to the same collector, these attributes are how you filter and search.

Auto-Instrumentation

Unlike Node.js, Go doesn't have a monkey-patching mechanism. You can't intercept all HTTP calls by loading a module at startup. Instead, Go's OTel instrumentation libraries provide wrappers that you apply explicitly.

HTTP with otelhttp

The otelhttp package provides middleware for incoming requests and a transport wrapper for outgoing requests.

For your HTTP server:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

mux := http.NewServeMux()
mux.HandleFunc("/orders", handleCreateOrder)
mux.HandleFunc("/orders/{id}", handleGetOrder)

// Wrap the entire mux
handler := otelhttp.NewHandler(mux, "my-api")
http.ListenAndServe(":8080", handler)

Every incoming request now gets a span with the HTTP method, route, status code, and duration. The span name defaults to the route pattern.

For outgoing HTTP calls (calling other services):

client := &http.Client{
	Transport: otelhttp.NewTransport(http.DefaultTransport),
}

resp, err := client.Do(req)

The transport wrapper creates a child span for each outgoing request and injects the traceparent header automatically. This is how trace context crosses service boundaries over HTTP.

SQL with otelsql

The otelsql package wraps database/sql to create spans for every query:

import (
	"database/sql"
	"go.opentelemetry.io/contrib/instrumentation/database/sql/otelsql"
	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
	_ "github.com/lib/pq"
)

db, err := otelsql.Open("postgres", "postgres://localhost:5432/mydb",
	otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
	otelsql.WithDBName("mydb"),
)

Now every db.QueryContext, db.ExecContext, and db.QueryRowContext call produces a span with the SQL statement and execution time.

Notice the Context suffix. The otelsql wrapper only instruments context-aware methods. If you call db.Query() (without a context), the query won't appear in your traces. This is a common gotcha: all your database calls need to pass a context.Context for tracing to work.

gRPC with otelgrpc

For gRPC services, otelgrpc provides interceptors for both servers and clients:

import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

// Server
server := grpc.NewServer(
	grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

// Client
conn, err := grpc.Dial(addr,
	grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

What auto-instrumentation covers and what it can't

These wrappers handle the transport layer: HTTP requests in and out, SQL queries, gRPC calls. They give you a skeleton of every request's journey through your system.

But they can't understand your business logic. An HTTP handler that validates an order, checks inventory, processes a payment, and sends a confirmation email looks like one long span with some database queries inside it. The meaningful steps are invisible. For that, you need custom spans.

Adding Custom Spans

Custom spans give you visibility into the operations that matter to your business. You create them using a Tracer obtained from the global provider.

import (
	"context"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/trace"
)

var tracer = otel.Tracer("order-service")

func processOrder(ctx context.Context, orderID string, items []Item) error {
	ctx, span := tracer.Start(ctx, "process-order",
		trace.WithAttributes(
			attribute.String("order.id", orderID),
			attribute.Int("order.item_count", len(items)),
		),
	)
	defer span.End()

	// Check inventory (creates a nested child span)
	if err := checkInventory(ctx, items); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "inventory check failed")
		return err
	}

	// Process payment (another child span)
	if err := chargePayment(ctx, orderID, totalPrice(items)); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "payment failed")
		return err
	}

	span.SetStatus(codes.Ok, "")
	return nil
}

func checkInventory(ctx context.Context, items []Item) error {
	ctx, span := tracer.Start(ctx, "check-inventory",
		trace.WithAttributes(
			attribute.Int("inventory.items_to_check", len(items)),
		),
	)
	defer span.End()

	// Database queries here will appear as children of this span
	// (if using otelsql and passing ctx)
	for _, item := range items {
		available, err := getStock(ctx, item.ProductID)
		if err != nil {
			span.RecordError(err)
			span.SetStatus(codes.Error, err.Error())
			return err
		}
		if available < item.Quantity {
			return fmt.Errorf("insufficient stock for product %s", item.ProductID)
		}
	}
	return nil
}

The pattern is consistent: tracer.Start(ctx, name) returns a new context and span. You pass the new context down to every subsequent call. defer span.End() ensures the span closes even on early returns or panics. Errors are recorded explicitly with RecordError and SetStatus.

The key thing Go gets right for tracing is that context.Context is already threaded through every function call. You don't need a separate mechanism to propagate trace context. The span lives in the context, and every function that receives ctx can create child spans.

Nested spans form automatically. When processOrder calls checkInventory and passes ctx, the inventory span becomes a child of the order span. When checkInventory makes a database query using that same ctx, the SQL span becomes a child of the inventory span. The hierarchy builds itself through the context chain.

Context Propagation

Go's context.Context is the foundation of OTel's context propagation within a single process. The active span is stored in the context, and any function that receives it can read the span or create children.

Across process boundaries, trace context travels as HTTP headers. The W3C traceparent header carries the trace ID, parent span ID, and sampling flags. When service A calls service B, the client injects these headers into the outgoing request, and service B's middleware extracts them to continue the trace.

If you're using otelhttp.NewTransport for outgoing calls and otelhttp.NewHandler for incoming requests, this happens automatically. But if you're using a custom HTTP client or a transport that otelhttp doesn't wrap, you need to handle it manually:

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
)

// Inject trace context into outgoing request headers
func injectTraceContext(ctx context.Context, req *http.Request) {
	otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))
}

// Extract trace context from incoming request headers
func extractTraceContext(ctx context.Context, req *http.Request) context.Context {
	return otel.GetTextMapPropagator().Extract(ctx, propagation.HeaderCarrier(req.Header))
}

You also need to register a propagator at startup. This is easy to forget and produces a subtle bug: traces appear to work locally but break across service boundaries.

import "go.opentelemetry.io/otel/propagation"

func initTracer(ctx context.Context) (func(), error) {
	// ... exporter and provider setup ...

	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.TraceContext{},
			propagation.Baggage{},
		),
	)

	// ... rest of setup ...
}

Without this, otel.GetTextMapPropagator() returns a no-op propagator, and no headers get injected or extracted. Your traces will show disconnected spans across services, and the cause won't be obvious from the code.

For Pub/Sub systems, context propagation requires more work. Message brokers don't natively carry traceparent headers. You need to inject trace context into message attributes on the publish side and extract it on the consume side. Some libraries like watermill have OTel plugins for this, but many require manual implementation.

Setting Up the Collector

Traces need somewhere to go. The OpenTelemetry Collector acts as a pipeline between your application and your observability backend. It receives telemetry, processes it, and exports it to one or more destinations.

Create a docker-compose.yml:

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    command: ["--config=/etc/otel/config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel/config.yaml
    ports:
      - "4317:4317" # gRPC receiver
      - "4318:4318" # HTTP receiver
    depends_on:
      - jaeger

  jaeger:
    image: jaegertracing/all-in-one:1.54
    ports:
      - "16686:16686" # Jaeger UI
      - "14268:14268" # Jaeger collector
    environment:
      - COLLECTOR_OTLP_ENABLED=true

Create otel-config.yaml for the collector:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, logging]

Start everything:

docker compose up -d

Run your Go application with the tracing configuration. Traces will flow from your app to the collector, through the batch processor, and into Jaeger. Open http://localhost:16686 to see your traces.

This is the minimum viable setup. In production, you'd add:

• Sampling to control volume (tail-based sampling in the collector catches slow and errored requests)
• Additional exporters for Datadog, Grafana Cloud, or other backends
• Resource detection to automatically add Kubernetes pod info, cloud provider metadata
• Health checks for the collector itself, since it becomes a critical component in your observability pipeline

The Maintenance Reality

Once you have this working, maintaining it is an ongoing commitment.

OpenTelemetry's Go modules release frequently, and the SDK and contrib packages need to stay compatible. The go.opentelemetry.io/otel core and go.opentelemetry.io/contrib instrumentation libraries have separate release cycles. Upgrading one without the other can produce build errors or subtle runtime issues. The semconv package versions (v1.21, v1.24, etc.) are particularly easy to get wrong, since different instrumentation libraries may expect different versions.

Every new library you adopt needs an instrumentation check. Adding Redis? You need otelredis. Switching from database/sql to pgx directly? The otelsql wrapper won't help; you need otelpgx or manual instrumentation. Using a custom RPC protocol? You're writing your own instrumentation.

Custom spans have a team cost. Every developer needs to know the OTel API, follow naming conventions, remember to pass context, handle errors with RecordError and SetStatus. In a larger codebase, inconsistent instrumentation is the norm. Some endpoints have detailed spans, others have none. Some spans record errors, others silently swallow them.

The context.Context discipline is Go's strength and OTel's requirement. If any function in the call chain drops the context or creates a new context.Background(), the trace breaks at that point. Everything downstream becomes a new root span, disconnected from the original request. This is correct Go practice regardless of tracing, but OTel makes the cost of getting it wrong visible.

None of this means OTel isn't worth it. Distributed tracing is essential for running backend services at any real scale. But the setup and maintenance cost falls on your team, and it accumulates with every service, every library, and every new hire who needs to learn the conventions.

An Alternative: Framework-Level Instrumentation

There's a different approach: move the instrumentation into the framework itself, so application code never touches tracing SDKs.

Encore.go takes this approach. Because Encore understands your application structure at compile time (the services, APIs, databases, Pub/Sub topics, and cron jobs), it instruments everything automatically. Tracing works out of the box from the moment you create your first endpoint.

Here's what an Encore service looks like:

package orders

import (
	"context"

	"encore.dev/storage/sqldb"
)

// Database declaration: Encore provisions and instruments it automatically
var db = sqldb.NewDatabase("orders", sqldb.DatabaseConfig{
	Migrations: "./migrations",
})

type CreateOrderRequest struct {
	UserID    string `json:"user_id"`
	ProductID string `json:"product_id"`
	Quantity  int    `json:"quantity"`
}

type Order struct {
	ID     string `json:"id"`
	Status string `json:"status"`
}

// API endpoint: traced automatically, including the database query inside
//
//encore:api public method=POST path=/orders
func CreateOrder(ctx context.Context, req *CreateOrderRequest) (*Order, error) {
	var order Order
	err := db.QueryRow(ctx,
		`INSERT INTO orders (user_id, product_id, quantity, status)
		 VALUES ($1, $2, $3, 'confirmed')
		 RETURNING id, status`,
		req.UserID, req.ProductID, req.Quantity,
	).Scan(&order.ID, &order.Status)
	if err != nil {
		return nil, err
	}
	return &order, nil
}

That's it. Every request to this endpoint is traced automatically, including the database query, with proper span hierarchy and timing. Service-to-service calls carry trace context without any propagation code. Pub/Sub messages maintain trace context across publish and subscribe boundaries.

You get a local development dashboard at localhost:9400 with a trace viewer that shows the full request lifecycle, including all database queries and their execution time, without configuring a collector or running Docker containers.

For production, Encore Cloud provides built-in observability with tracing, metrics, and logging — all accessible through Encore's own dashboard without needing to configure external backends.

Comparison

	Manual OpenTelemetry	Encore
Setup time	1-2 hours for basic tracing	Zero configuration
Packages to install	8+ (SDK, exporters, instrumentations)	None (built into the framework)
Auto-instrumentation	HTTP, SQL, gRPC (via explicit wrappers)	All APIs, databases, Pub/Sub, cron, service calls
Custom business logic	Manual spans with OTel API	Manual spans with OTel API (same)
Context propagation	Automatic for wrapped transports, manual otherwise	Automatic across all Encore primitives
Collector setup	Docker Compose + config file	Built into local dev, Encore Cloud for production
Maintenance	Keep 8+ modules in sync, verify on upgrades	Framework handles it
Flexibility	Any Go app, any architecture	Encore applications
Trace visualization	Via collector + external backend (Jaeger, Grafana, etc.)	Built-in tracing dashboard in local dev and Encore Cloud
context.Context	Required (good practice anyway)	Required (good practice anyway)

The trade-off is clear: manual OTel gives you maximum flexibility to instrument any Go application regardless of framework. Encore gives you comprehensive instrumentation with no setup cost, but you're building with the Encore framework. If you're starting a new Go project and want distributed tracing from day one, the framework-level approach removes an entire category of work from your plate.

Next Steps

If you're going the manual OTel route:

• Read the OpenTelemetry Go documentation for the full API reference
• Explore tail-based sampling in the collector to manage trace volume in production
• Look at the contrib instrumentation registry to find wrappers for the libraries you use
• Set up OTel metrics alongside traces for a more complete picture

If you want to skip the instrumentation setup:

• Install Encore and try the overview guide
• Check out the tracing features to see what built-in observability looks like
• Explore the Encore starter templates for working Go examples with full observability out of the box

Related Reading

• Distributed Tracing for Go APIs
• How to Debug Slow Go API Requests
• Best Tracing Tools for Go Backends
• Tracing vs Logging vs Metrics
• Distributed Tracing in Microservices
• Building Microservices in Go
• Best Go Backend Frameworks