Build And Deploy

Telemetry

Use OpenTelemetry-backed request ids, context logs, spans, and request metrics.

By Guillermo Alvarez - Published - Updated

OpenTelemetry orientation

GoLazy telemetry is aligned with OpenTelemetry, but it is not a complete SDK distribution. OpenTelemetry is the portable model for traces, metrics, logs, resources, context propagation, and OTLP exporters. Use these upstream references when wiring a production collector:

golazy.dev/lazytelemetry adds GoLazy behavior on top of that model. It is not only a wrapper around upstream packages: it owns request-id generation, safe correlation header handling, context-aware slog attributes, log-to-span events, low-dependency in-memory request metrics, Prometheus text exposition on the control plane, cache metric collection, and lazydev runtime trace artifacts.

The tradeoff is deliberate. The framework can expose useful local telemetry and standard OTEL_* configuration while keeping the app API small. GoLazy now installs the OpenTelemetry trace SDK and trace auto-exporter when trace export is configured. Metrics still use GoLazy's in-process registry and Prometheus control-plane endpoint, and logs are still process stdout logs for platform log collectors rather than a full OpenTelemetry logs SDK pipeline.

Activation

lazyapp.New always creates a lazytelemetry.Telemetry instance, stores it in the app context, and registers it as a dependency lifecycle service. It installs request telemetry middleware when the process has meaningful, known OpenTelemetry environment configuration.

This enables telemetry:

OTEL_SERVICE_NAME=sample_app ./sample-app
OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 ./sample-app
OTEL_METRICS_EXPORTER=prometheus ./sample-app
OTEL_LOGS_EXPORTER=console ./sample-app

This keeps telemetry disabled:

OTEL_SDK_DISABLED=true ./sample-app
OTEL_TRACES_EXPORTER=none OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none ./sample-app

OTEL_SDK_DISABLED=true wins over other telemetry variables. Setting OTEL_SDK_DISABLED=false by itself does not activate telemetry. Unknown OTEL_* variables do not activate telemetry by themselves.

Trace export is initialized when OTEL_TRACES_EXPORTER is set to an enabled exporter such as otlp or console, or when trace/global OTLP exporter configuration is present, such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT. OTEL_SERVICE_NAME alone activates request correlation and logging but does not start an OTLP trace exporter.

Applications can read the telemetry instance from dependency, job, request, and service contexts:

telemetry, ok := lazytelemetry.FromContext(ctx)
if ok {
    logger := telemetry.Logger()
    logger.InfoContext(ctx, "telemetry is available")
}

Most applications do not need to touch the instance directly. It owns the environment config, request logger, Prometheus metrics registry, optional OTel tracer provider, and exporter shutdown.

Request context

When telemetry is active, GoLazy adds one request middleware before the normal route dispatcher. It:

  • reads X-Request-ID or X-Correlation-ID when the value is safe;
  • generates a request id when no valid id is present;
  • writes X-Request-ID on the response;
  • preserves W3C traceparent and tracestate when present;
  • starts a lightweight request span;
  • attaches a request-aware slog.Logger to r.Context();
  • records request count and duration metrics.

Application code can read the request id and logger from context:

requestID := lazytelemetry.RequestID(r.Context())
logger := lazytelemetry.Logger(r.Context())
logger.InfoContext(r.Context(), "loaded post", "request_id", requestID)

Prefer lazytelemetry.WithLogAttrs, WithLogTags, and WithLogGroup when a service wants to carry structured context into downstream calls.

Logs

The default log backend is log/slog. Without a log exporter setting, GoLazy uses text logs on stdout. When a log exporter setting is configured, GoLazy uses structured JSON logs on stdout for this first implementation.

Logs created through lazytelemetry or lazylogs include request id, trace id, span id, method, path, and any attributes, tags, or groups already attached to the context. Logs emitted while a span is active are also recorded as span events, including the request id, so future exporters can include request-local log messages in traces. Go runtime traces also receive a lightweight log marker for those messages when request tracing is active.

GoLazy does not currently import go.opentelemetry.io/otel/log in the runtime path. That package is the upstream OpenTelemetry Logs API; the Go instrumentation docs recommend using bridges from existing log packages such as slog, zap, logrus, or logr into the logs SDK. Importing it becomes useful when GoLazy adds a real logs bridge and SDK/exporter setup. Until then, logs are normal process stdout JSON when OTEL_LOGS_EXPORTER is present, so the platform can collect them by stdout/file forwarding, usually into Loki in Kubernetes. OTEL_LOG_RECORD_TO_STRING is not a standard OpenTelemetry environment variable and is not read by GoLazy.

Spans and runtime trace

lazytelemetry.StartSpan starts a lightweight in-process span and attaches it to context. It is backed by the OpenTelemetry Go API, preserves incoming W3C trace ids, adds OpenTelemetry span attributes and events, and also uses Go runtime/trace tasks and regions for local diagnostics. GoLazy keeps small semantic helpers on top so framework code can name controller, view, cache, and request-id concepts consistently.

The request middleware is the top-level request task/span. When telemetry is active, GoLazy adds child regions for route matching, route dispatch, middleware calls, controller setup and BeforeAction, the concrete controller action, view rendering, layout rendering, and every partial helper/template render. Dynamic route middleware also records child regions for method override, downstream handler execution, ETag handling, and final buffered response flush so large cached responses can be distinguished from controller work. Region names include the middleware, controller/action, template, partial, or dispatch phase identifier so Go runtime traces are readable even without inspecting OpenTelemetry attributes. The request id is attached to the root request span and inherited by child regions; Go runtime traces also get a request_id log marker for the request task/region.

When trace export is configured, lazytelemetry.Telemetry calls the upstream OpenTelemetry auto-exporter for spans and installs an SDK tracer provider with a batch span processor. Supported standard trace exporter values are the auto-exporter values:

  • OTEL_TRACES_EXPORTER=none disables trace export;
  • OTEL_TRACES_EXPORTER=otlp exports OTLP traces;
  • OTEL_TRACES_EXPORTER=console writes spans to stdout.

For OTLP, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL or OTEL_EXPORTER_OTLP_PROTOCOL selects grpc or http/protobuf. Tempo does not need a GoLazy-specific exporter; point the standard OTLP trace endpoint at Tempo or an OpenTelemetry Collector that forwards to Tempo. The Go SDK reads standard resource, sampler, span-limit, and batch-span-processor variables such as OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG, and OTEL_BSP_*.

In lazydev builds, the request middleware can also record per-request development artifacts under the app root. Detailed request monitoring is off by default; enable it from the development panel's Requests or Traces controls:

.tmp/traces/<request-id>.trace
.tmp/traces/<request-id>.spans
.tmp/traces/<request-id>.log.json

The .trace file is the raw Go runtime trace. The .spans file is JSON with request metadata, span attributes and events, runtime details, goroutine counts, request-level allocation counts, memory deltas, system memory, stack, GC counters, and lazydev-only per-region allocation samples from runtime.ReadMemStats. The .log.json file is JSONL for request-local logs emitted through lazytelemetry or lazylogs. Because Go runtime tracing is process-wide, lazydev serializes traced requests while each trace file is being recorded.

Region allocation samples record allocation bytes, malloc counts, and free counts as process-wide deltas while each span is active. The sidecar also stores self duration and self allocation estimates by subtracting direct child regions. These values are intended for development diagnostics in the Traces tab, not exact runtime.mallocgc attribution.

Metrics

lazytelemetry/lazymetrics provides in-memory counter, gauge, and histogram vectors with Prometheus-like label APIs but no Prometheus dependency. The request middleware records those values locally and also records the same request count and duration through OpenTelemetry meter instruments when a meter provider is installed by the application or a telemetry adapter.

The request middleware records both http_server_requests_total and http_server_request_duration_seconds with method, route, status_class, controller, and action.

The route label uses the matched route pattern, not the raw request path. Controller routes also set controller and action from GoLazy route metadata. Non-controller routes and unmatched requests use unknown for those labels.

When OTEL_METRICS_EXPORTER=prometheus, lazyapp.New mounts /metrics on the control plane. The endpoint exposes request counters and duration histograms in Prometheus text format. It also exports the application cache stats from lazycache.Stats():

  • golazy_cache_enabled;
  • golazy_cache_entries;
  • golazy_cache_max_entries;
  • golazy_cache_hits_total;
  • golazy_cache_misses_total;
  • golazy_cache_sets_total;
  • golazy_cache_evictions_total.

/metrics is exposed only when CONTROL_PLANE_ADDR is set. If CONTROL_PLANE_ADDR points at a separate address, Prometheus should scrape that control-plane listener. If it is the same address as ADDR, GoLazy mounts the control-plane endpoints on the same listener before app routes. When CONTROL_PLANE_ADDR is unset, production builds do not expose /metrics on the public app listener. An app can still provide its own /metrics handler through lazycontrolplane.Config; explicit app control-plane metrics take precedence.

Keep metric labels low-cardinality. Do not use request id, raw query strings, user ids, or arbitrary route parameter values as labels.

GoLazy creates OpenTelemetry spans and meter measurements. Trace export is installed automatically from standard OTEL_* trace exporter variables. GoLazy does not yet install an OpenTelemetry metric reader/exporter automatically; metrics intended for Prometheus should use OTEL_METRICS_EXPORTER=prometheus and the control-plane /metrics endpoint.

Kubernetes example

A GoLazy app that exposes Prometheus metrics on a separate control-plane port usually needs three pieces in Kubernetes:

env:
  - name: CONTROL_PLANE_ADDR
    value: "0.0.0.0:3001"
  - name: OTEL_SERVICE_NAME
    value: "sample-app"
  - name: OTEL_METRICS_EXPORTER
    value: "prometheus"
  - name: OTEL_TRACES_EXPORTER
    value: "otlp"
  - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
    value: "grpc"
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "http://tempo.observability.svc.cluster.local:4317"
  - name: OTEL_TRACES_SAMPLER
    value: "parentbased_traceidratio"
  - name: OTEL_TRACES_SAMPLER_ARG
    value: "0.1"
  - name: OTEL_LOGS_EXPORTER
    value: "console"

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "3001"
    prometheus.io/path: "/metrics"
    prometheus.io/scheme: "http"

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    release: kube-prom-stack
spec:
  selector:
    matchLabels:
      app: sample-app
  podMetricsEndpoints:
    - port: control
      path: /metrics
      scheme: http

The prometheus.io/* keys are pod annotations for annotation-based scrapers. Prometheus Operator clusters commonly select PodMonitor or ServiceMonitor objects instead; in that setup the PodMonitor is the reliable scrape contract.

For ingress-nginx traces, enable OpenTelemetry on each app Ingress and configure the collector on the ingress controller ConfigMap:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-opentelemetry: "true"
    nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: "true"

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-load-balancer-microk8s-conf
  namespace: ingress
data:
  enable-opentelemetry: "true"
  otlp-collector-host: "tempo.observability.svc.cluster.local"
  otlp-collector-port: "4317"
  otel-sampler: "TraceIdRatioBased"
  otel-sampler-ratio: "0.1"
  otel-sampler-parent-based: "true"

otlp-collector-host, otlp-collector-port, and otel-sampler-ratio are ingress controller ConfigMap keys, not per-Ingress annotations.

Environment variables

GoLazy reads the following OpenTelemetry-style environment variables with the OTEL_ prefix. The configuration is loaded with lazyconfig.RemoveEnvNamePrefix("OTEL").

SDK and resources

  • OTEL_SDK_DISABLED
  • OTEL_ENTITIES
  • OTEL_RESOURCE_ATTRIBUTES
  • OTEL_SERVICE_NAME
  • OTEL_LOG_LEVEL
  • OTEL_PROPAGATORS
  • OTEL_TRACES_SAMPLER
  • OTEL_TRACES_SAMPLER_ARG

Exporter selection

  • OTEL_TRACES_EXPORTER
  • OTEL_METRICS_EXPORTER
  • OTEL_LOGS_EXPORTER

Known useful values include otlp, console, otlp/stdout, prometheus, and none. zipkin is trace-only, and prometheus is metrics-only.

OTLP exporter

  • OTEL_EXPORTER_OTLP_ENDPOINT
  • OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
  • OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
  • OTEL_EXPORTER_OTLP_INSECURE
  • OTEL_EXPORTER_OTLP_TRACES_INSECURE
  • OTEL_EXPORTER_OTLP_METRICS_INSECURE
  • OTEL_EXPORTER_OTLP_LOGS_INSECURE
  • OTEL_EXPORTER_OTLP_CERTIFICATE
  • OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE
  • OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE
  • OTEL_EXPORTER_OTLP_LOGS_CERTIFICATE
  • OTEL_EXPORTER_OTLP_CLIENT_KEY
  • OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY
  • OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY
  • OTEL_EXPORTER_OTLP_LOGS_CLIENT_KEY
  • OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE
  • OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE
  • OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE
  • OTEL_EXPORTER_OTLP_LOGS_CLIENT_CERTIFICATE
  • OTEL_EXPORTER_OTLP_HEADERS
  • OTEL_EXPORTER_OTLP_TRACES_HEADERS
  • OTEL_EXPORTER_OTLP_METRICS_HEADERS
  • OTEL_EXPORTER_OTLP_LOGS_HEADERS
  • OTEL_EXPORTER_OTLP_COMPRESSION
  • OTEL_EXPORTER_OTLP_TRACES_COMPRESSION
  • OTEL_EXPORTER_OTLP_METRICS_COMPRESSION
  • OTEL_EXPORTER_OTLP_LOGS_COMPRESSION
  • OTEL_EXPORTER_OTLP_TIMEOUT
  • OTEL_EXPORTER_OTLP_TRACES_TIMEOUT
  • OTEL_EXPORTER_OTLP_METRICS_TIMEOUT
  • OTEL_EXPORTER_OTLP_LOGS_TIMEOUT
  • OTEL_EXPORTER_OTLP_PROTOCOL
  • OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
  • OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
  • OTEL_EXPORTER_OTLP_LOGS_PROTOCOL

Batch processing

  • OTEL_BSP_SCHEDULE_DELAY
  • OTEL_BSP_EXPORT_TIMEOUT
  • OTEL_BSP_MAX_QUEUE_SIZE
  • OTEL_BSP_MAX_EXPORT_BATCH_SIZE
  • OTEL_BLRP_SCHEDULE_DELAY
  • OTEL_BLRP_EXPORT_TIMEOUT
  • OTEL_BLRP_MAX_QUEUE_SIZE
  • OTEL_BLRP_MAX_EXPORT_BATCH_SIZE

Limits

  • OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT
  • OTEL_ATTRIBUTE_COUNT_LIMIT
  • OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT
  • OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
  • OTEL_SPAN_EVENT_COUNT_LIMIT
  • OTEL_SPAN_LINK_COUNT_LIMIT
  • OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT
  • OTEL_LINK_ATTRIBUTE_COUNT_LIMIT
  • OTEL_LOGRECORD_ATTRIBUTE_VALUE_LENGTH_LIMIT
  • OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT

Metrics

  • OTEL_METRICS_EXEMPLAR_FILTER
  • OTEL_METRIC_EXPORT_INTERVAL
  • OTEL_METRIC_EXPORT_TIMEOUT

Other exporters and config files

  • OTEL_EXPORTER_ZIPKIN_ENDPOINT
  • OTEL_EXPORTER_ZIPKIN_TIMEOUT
  • OTEL_EXPORTER_ZIPKIN_PROTOCOL
  • OTEL_EXPORTER_PROMETHEUS_HOST
  • OTEL_EXPORTER_PROMETHEUS_PORT
  • OTEL_CONFIG_FILE
  • OTEL_EXPERIMENTAL_CONFIG_FILE

Legacy names

GoLazy also reads these obsolete OTLP names for compatibility:

  • OTEL_EXPORTER_OTLP_SPAN_INSECURE
  • OTEL_EXPORTER_OTLP_METRIC_INSECURE

Development command direction

The lazy development command does not force OTEL exporters into the child app. Lazydev builds install request telemetry unless OTEL_SDK_DISABLED=true is present, but detailed request monitoring is off by default. Enable it from the development panel to write the local .tmp/traces files described above.

The development proxy also assigns correlation before forwarding normal app traffic. It preserves safe incoming X-Request-ID and valid W3C traceparent headers, or generates both when they are missing, so the browser, lazy proxy, and app artifacts can share the same request id and trace id.