Telemetry

OpenTelemetry orientation

GoLazy telemetry is aligned with OpenTelemetry, but it is not a complete SDK distribution. OpenTelemetry is the portable model for traces, metrics, logs, resources, context propagation, and OTLP exporters. Use these upstream references when wiring a production collector:

• OpenTelemetry Go instrumentation for span, metric, and log-bridge concepts;
• OpenTelemetry SDK environment variables for service, sampler, and exporter selection variables;
• OpenTelemetry Protocol exporter configuration for OTLP endpoint, protocol, and signal-specific endpoint rules;
• ingress-nginx OpenTelemetry for Kubernetes ingress traces;
• go.opentelemetry.io/otel/log and go.opentelemetry.io/otel/sdk/log when adding a true OpenTelemetry logs SDK path.

golazy.dev/lazytelemetry adds GoLazy behavior on top of that model. It is not only a wrapper around upstream packages: it owns request-id generation, safe correlation header handling, context-aware slog attributes, log-to-span events, low-dependency in-memory request metrics, Prometheus text exposition on the control plane, cache metric collection, and lazydev runtime trace artifacts.

The tradeoff is deliberate. The framework can expose useful local telemetry and standard OTEL_* configuration without pulling every OpenTelemetry exporter into every GoLazy application. Full OTLP trace/log export still requires an application-mounted or adapter-provided OpenTelemetry SDK provider.

Activation

GoLazy telemetry is off by default. lazyapp.New installs telemetry middleware when the process has meaningful, known OpenTelemetry environment configuration.

This enables telemetry:

OTEL_SERVICE_NAME=sample_app ./sample-app
OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 ./sample-app
OTEL_METRICS_EXPORTER=prometheus ./sample-app
OTEL_LOGS_EXPORTER=otlp/stdout ./sample-app

This keeps telemetry disabled:

OTEL_SDK_DISABLED=true ./sample-app
OTEL_TRACES_EXPORTER=none OTEL_METRICS_EXPORTER=none OTEL_LOGS_EXPORTER=none ./sample-app

OTEL_SDK_DISABLED=true wins over other telemetry variables. Setting OTEL_SDK_DISABLED=false by itself does not activate telemetry. Unknown OTEL_* variables do not activate telemetry by themselves.

Request context

When telemetry is active, GoLazy adds one request middleware before the normal route dispatcher. It:

• reads X-Request-ID or X-Correlation-ID when the value is safe;
• generates a request id when no valid id is present;
• writes X-Request-ID on the response;
• preserves W3C traceparent and tracestate when present;
• starts a lightweight request span;
• attaches a request-aware slog.Logger to r.Context();
• records request count and duration metrics.

Application code can read the request id and logger from context:

requestID := lazytelemetry.RequestID(r.Context())
logger := lazytelemetry.Logger(r.Context())
logger.InfoContext(r.Context(), "loaded post", "request_id", requestID)

Prefer lazytelemetry.WithLogAttrs, WithLogTags, and WithLogGroup when a service wants to carry structured context into downstream calls.

Logs

The default log backend is log/slog. Without a log exporter setting, GoLazy uses text logs on stdout. When a log exporter setting is configured, GoLazy uses structured JSON logs on stdout for this first implementation.

Logs created through lazytelemetry or lazylogs include request id, trace id, span id, method, path, and any attributes, tags, or groups already attached to the context. Logs emitted while a span is active are also recorded as span events so future exporters can include request-local log messages in traces.

GoLazy does not currently import go.opentelemetry.io/otel/log in the runtime path. That package is the upstream OpenTelemetry Logs API; the Go instrumentation docs recommend using bridges from existing log packages such as slog, zap, logrus, or logr into the logs SDK. Importing it becomes useful when GoLazy adds a real logs bridge and SDK/exporter setup. Until then, logs are normal process stdout JSON when OTEL_LOGS_EXPORTER is present, so the platform can collect them by stdout/file forwarding. OTEL_LOG_RECORD_TO_STRING is not a standard OpenTelemetry environment variable and is not read by GoLazy.

Spans and runtime trace

lazytelemetry.StartSpan starts a lightweight in-process span and attaches it to context. It is backed by the OpenTelemetry Go API, preserves incoming W3C trace ids, adds OpenTelemetry span attributes and events, and also uses Go runtime/trace tasks and regions for local diagnostics. GoLazy keeps small semantic helpers on top so framework code can name controller, view, cache, and request-id concepts consistently.

In lazydev builds, when an OTEL exporter or OTLP exporter setting is present, the request middleware also records per-request development artifacts under the app root:

.tmp/traces/<request-id>.trace
.tmp/traces/<request-id>.spans
.tmp/traces/<request-id>.log.json

The .trace file is the raw Go runtime trace. The .spans file is JSON with request metadata, span attributes and events, runtime details, goroutine counts, allocation counts, memory deltas, system memory, stack, and GC counters from runtime.ReadMemStats. The .log.json file is JSONL for request-local logs emitted through lazytelemetry or lazylogs. Because Go runtime tracing is process-wide, lazydev serializes traced requests while each trace file is being recorded.

Metrics

lazytelemetry/lazymetrics provides in-memory counter, gauge, and histogram vectors with Prometheus-like label APIs but no Prometheus dependency. The request middleware records those values locally and also records the same request count and duration through OpenTelemetry meter instruments when a meter provider is installed by the application or a telemetry adapter.

The request middleware records:

• http_server_requests_total with method, route, and status_class;
• http_server_request_duration_seconds with the same labels.

When OTEL_METRICS_EXPORTER=prometheus, lazyapp.New mounts /metrics on the control plane. The endpoint exposes request counters and duration histograms in Prometheus text format. It also exports the application cache stats from lazycache.Stats():

• golazy_cache_enabled;
• golazy_cache_entries;
• golazy_cache_max_entries;
• golazy_cache_hits_total;
• golazy_cache_misses_total;
• golazy_cache_sets_total;
• golazy_cache_evictions_total.

/metrics is exposed only when CONTROL_PLANE_ADDR is set. If CONTROL_PLANE_ADDR points at a separate address, Prometheus should scrape that control-plane listener. If it is the same address as ADDR, GoLazy mounts the control-plane endpoints on the same listener before app routes. When CONTROL_PLANE_ADDR is unset, production builds do not expose /metrics on the public app listener. An app can still provide its own /metrics handler through lazycontrolplane.Config; explicit app control-plane metrics take precedence.

Keep metric labels low-cardinality. Do not use request id, raw query strings, user ids, or arbitrary route parameter values as labels.

GoLazy creates OpenTelemetry spans and meter measurements, but it does not yet install a full OpenTelemetry SDK exporter automatically. For now, exporting traces to an OTLP backend such as Tempo requires an application-mounted or adapter-provided OpenTelemetry tracer provider. The framework already parses standard trace exporter variables such as OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://tempo.observability:4317; automatic SDK/exporter setup is the next adapter boundary.

Kubernetes example

A GoLazy app that exposes Prometheus metrics on a separate control-plane port usually needs three pieces in Kubernetes:

env:
  - name: CONTROL_PLANE_ADDR
    value: "0.0.0.0:3001"
  - name: OTEL_SERVICE_NAME
    value: "sample-app"
  - name: OTEL_METRICS_EXPORTER
    value: "prometheus"
  - name: OTEL_TRACES_EXPORTER
    value: "otlp"
  - name: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
    value: "grpc"
  - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
    value: "http://tempo.observability.svc.cluster.local:4317"
  - name: OTEL_TRACES_SAMPLER
    value: "parentbased_traceidratio"
  - name: OTEL_TRACES_SAMPLER_ARG
    value: "0.1"
  - name: OTEL_LOGS_EXPORTER
    value: "console"

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "3001"
    prometheus.io/path: "/metrics"
    prometheus.io/scheme: "http"

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    release: kube-prom-stack
spec:
  selector:
    matchLabels:
      app: sample-app
  podMetricsEndpoints:
    - port: control
      path: /metrics
      scheme: http

The prometheus.io/* keys are pod annotations for annotation-based scrapers. Prometheus Operator clusters commonly select PodMonitor or ServiceMonitor objects instead; in that setup the PodMonitor is the reliable scrape contract.

For ingress-nginx traces, enable OpenTelemetry on each app Ingress and configure the collector on the ingress controller ConfigMap:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-opentelemetry: "true"
    nginx.ingress.kubernetes.io/opentelemetry-trust-incoming-span: "true"

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-load-balancer-microk8s-conf
  namespace: ingress
data:
  enable-opentelemetry: "true"
  otlp-collector-host: "tempo.observability.svc.cluster.local"
  otlp-collector-port: "4317"
  otel-sampler: "TraceIdRatioBased"
  otel-sampler-ratio: "0.1"
  otel-sampler-parent-based: "true"

otlp-collector-host, otlp-collector-port, and otel-sampler-ratio are ingress controller ConfigMap keys, not per-Ingress annotations.

Environment variables

GoLazy reads the following OpenTelemetry-style environment variables with the OTEL_ prefix. The configuration is loaded with lazyconfig.RemoveEnvNamePrefix("OTEL").

SDK and resources

• OTEL_SDK_DISABLED
• OTEL_ENTITIES
• OTEL_RESOURCE_ATTRIBUTES
• OTEL_SERVICE_NAME
• OTEL_LOG_LEVEL
• OTEL_PROPAGATORS
• OTEL_TRACES_SAMPLER
• OTEL_TRACES_SAMPLER_ARG

Exporter selection

• OTEL_TRACES_EXPORTER
• OTEL_METRICS_EXPORTER
• OTEL_LOGS_EXPORTER

Known useful values include otlp, console, otlp/stdout, prometheus, and none. zipkin is trace-only, and prometheus is metrics-only.

OTLP exporter

• OTEL_EXPORTER_OTLP_ENDPOINT
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
• OTEL_EXPORTER_OTLP_INSECURE
• OTEL_EXPORTER_OTLP_TRACES_INSECURE
• OTEL_EXPORTER_OTLP_METRICS_INSECURE
• OTEL_EXPORTER_OTLP_LOGS_INSECURE
• OTEL_EXPORTER_OTLP_CERTIFICATE
• OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE
• OTEL_EXPORTER_OTLP_METRICS_CERTIFICATE
• OTEL_EXPORTER_OTLP_LOGS_CERTIFICATE
• OTEL_EXPORTER_OTLP_CLIENT_KEY
• OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY
• OTEL_EXPORTER_OTLP_METRICS_CLIENT_KEY
• OTEL_EXPORTER_OTLP_LOGS_CLIENT_KEY
• OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE
• OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE
• OTEL_EXPORTER_OTLP_METRICS_CLIENT_CERTIFICATE
• OTEL_EXPORTER_OTLP_LOGS_CLIENT_CERTIFICATE
• OTEL_EXPORTER_OTLP_HEADERS
• OTEL_EXPORTER_OTLP_TRACES_HEADERS
• OTEL_EXPORTER_OTLP_METRICS_HEADERS
• OTEL_EXPORTER_OTLP_LOGS_HEADERS
• OTEL_EXPORTER_OTLP_COMPRESSION
• OTEL_EXPORTER_OTLP_TRACES_COMPRESSION
• OTEL_EXPORTER_OTLP_METRICS_COMPRESSION
• OTEL_EXPORTER_OTLP_LOGS_COMPRESSION
• OTEL_EXPORTER_OTLP_TIMEOUT
• OTEL_EXPORTER_OTLP_TRACES_TIMEOUT
• OTEL_EXPORTER_OTLP_METRICS_TIMEOUT
• OTEL_EXPORTER_OTLP_LOGS_TIMEOUT
• OTEL_EXPORTER_OTLP_PROTOCOL
• OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
• OTEL_EXPORTER_OTLP_METRICS_PROTOCOL
• OTEL_EXPORTER_OTLP_LOGS_PROTOCOL

Batch processing

• OTEL_BSP_SCHEDULE_DELAY
• OTEL_BSP_EXPORT_TIMEOUT
• OTEL_BSP_MAX_QUEUE_SIZE
• OTEL_BSP_MAX_EXPORT_BATCH_SIZE
• OTEL_BLRP_SCHEDULE_DELAY
• OTEL_BLRP_EXPORT_TIMEOUT
• OTEL_BLRP_MAX_QUEUE_SIZE
• OTEL_BLRP_MAX_EXPORT_BATCH_SIZE

Limits

• OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT
• OTEL_ATTRIBUTE_COUNT_LIMIT
• OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT
• OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
• OTEL_SPAN_EVENT_COUNT_LIMIT
• OTEL_SPAN_LINK_COUNT_LIMIT
• OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT
• OTEL_LINK_ATTRIBUTE_COUNT_LIMIT
• OTEL_LOGRECORD_ATTRIBUTE_VALUE_LENGTH_LIMIT
• OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT

Metrics

• OTEL_METRICS_EXEMPLAR_FILTER
• OTEL_METRIC_EXPORT_INTERVAL
• OTEL_METRIC_EXPORT_TIMEOUT

Other exporters and config files

• OTEL_EXPORTER_ZIPKIN_ENDPOINT
• OTEL_EXPORTER_ZIPKIN_TIMEOUT
• OTEL_EXPORTER_ZIPKIN_PROTOCOL
• OTEL_EXPORTER_PROMETHEUS_HOST
• OTEL_EXPORTER_PROMETHEUS_PORT
• OTEL_CONFIG_FILE
• OTEL_EXPERIMENTAL_CONFIG_FILE

Legacy names

GoLazy also reads these obsolete OTLP names for compatibility:

• OTEL_EXPORTER_OTLP_SPAN_INSECURE
• OTEL_EXPORTER_OTLP_METRIC_INSECURE

Development command direction

The lazy development command sets OTEL_SDK_DISABLED=false, OTEL_TRACES_EXPORTER=otlp, OTEL_LOGS_EXPORTER=otlp, and OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf for the child app. It does not set an OTLP endpoint yet; the first development transport is the local .tmp/traces files described above.

The development proxy also assigns correlation before forwarding normal app traffic. It preserves safe incoming X-Request-ID and valid W3C traceparent headers, or generates both when they are missing, so the browser, lazy proxy, and app artifacts can share the same request id and trace id.