OpenTelemetry: Trace, Metric and Log Correlation

1. The problem: three signals are not the same as diagnosability

A service may emit traces, metrics, and logs while operators still cannot prove which observations belong to the same request or failure path. Three dashboards only prove that data exists. Correlation requires stable identity, context, and a relationship that can be evidenced.

The owned output is Telemetry Correlation Evidence Record v1. It keeps direct evidence, inference, missing context, and unknowns separate. The goal is not “more telemetry”; it is one bounded, traceable diagnosis.

2. What OpenTelemetry is and is not

OpenTelemetry is an open-source, vendor-neutral framework and toolkit for instrumenting, generating, collecting, and exporting telemetry such as traces, metrics, and logs. OTel itself is not the storage or visualization backend, and a healthy pipeline does not prove instrumentation completeness.

In this lesson, a Collector or exporter is only one layer. The diagnostic question is whether the observations preserve enough identity and context to support the claimed relationship.

3. The three signals do different jobs

A trace usually exposes execution structure and parent/child relationships. A log captures an event or message at a point in execution. A metric summarizes behavior as aggregated or time-series measurements. They complement each other but are not interchangeable.

A detailed log may lack distributed topology. An aggregate metric may reveal an anomaly without identifying the exact request. A trace may show the path while an application log contains the error detail.

4. Establish resource identity before correlation

OpenTelemetry associates telemetry with Resource context. Fields such as service.name and, where supplied, service.instance.id can identify the origin of observations. A shared service name alone does not prove that two records came from the same instance or deployment.

Record resource identity independently. When identity is absent or inconsistent, leave that uncertainty explicit instead of filling it with a naming assumption.

5. Trace topology: a Trace ID is not the whole relationship

Distributed tracing also needs parentage. If Service A creates a span and Service B produces a downstream span with the same Trace ID and a Parent Span ID pointing to the upstream span, the relationship can be directly checked.

If the downstream span starts a new Trace ID or loses parentage, nearby timestamps do not repair the relationship. Record a propagation gap rather than inventing an edge.

6. Context propagation connects service boundaries

Context propagation carries execution context across process and service boundaries. For HTTP, trace context is commonly injected into a carrier and extracted by the receiver. Instrumentation often automates this, but propagation can still break.

One lab edge intentionally loses context. The correct response is not to infer the missing causal edge; keep it unproven and name the minimum instrumentation repair.

7. Log correlation uses trace and span context

The OpenTelemetry log data model can carry TraceId and SpanId on a LogRecord. When those identifiers match an observed span, the log can be correlated directly with that execution context.

A log that merely occurred in the same second is not direct correlation. Time is useful for search and triage; it is insufficient causal proof. The lab includes a deliberately nearby but context-free log.

8. Metric correlation and exemplars

Metrics are normally aggregates. OpenTelemetry defines exemplars as recorded measurements that may preserve context, including trace_id and span_id, allowing a concrete bridge from an aggregate metric to a trace.

A metric without an exemplar can still be valuable, but the aggregate alone cannot identify one exact request. The record separates aggregate evidence from trace-linked evidence.

9. Use the full key, not a partial-key shortcut

Do not equate a matching Span ID with a complete correlation key. Synthetic data can contain the same span identifier under a different Trace ID. If a log has the same span ID but a different Trace ID, reject the link.

The lab includes this probe specifically to invalidate the shortcut “span_id matches, therefore it is the same request.”

10. Pipeline health is not telemetry completeness

A Collector can successfully receive, process, and export what it is given while a library is uninstrumented, a route drops propagation, or a signal is missing on a critical path. Pipeline health only describes telemetry that actually entered the pipeline.

Do not convert pipeline healthy into application fully observable. Missing signal coverage remains an explicit instrumentation gap.

11. Bound cardinality and attributes by the evidence question

Adding many highly variable attributes may appear to make correlation easier, but it changes cost, volume, privacy exposure, and cardinality. Each attribute should answer a bounded diagnostic need.

Use appropriate semantic identity fields and keep secrets, credentials, and sensitive user data out of telemetry. The lesson fixtures contain no real user data.

12. Telemetry Correlation Evidence Record v1

The record includes: observation, signal type, resource/service identity, Trace ID, Span ID, parent relationship, log-context link, metric/exemplar link, direct correlation, inference, missing context, instrumentation gap, unknowns, and diagnostic decision.

This is a MAKNOW instructional synthesis, not a mandatory OpenTelemetry form. Its purpose is to keep observation and inference adjacent but distinct and to prevent gaps from silently disappearing.

13. Lab: eight fixtures and negative probes

The lab is fully local and synthetic. Eight fixtures cover a healthy path, broken propagation, a nearby context-free log, resource mismatch, a metric without exemplar linkage, a partial-key collision, incomplete signal coverage, and a transfer topology with a different broken edge.

For every fixture, classify eight dimensions independently. The hidden oracle checks relationships, not memorized prose. No external endpoint, live backend, or credential is required.

14. Transfer and acceptance criteria

In transfer, service names, instances, trace topology, and the broken edge change. First-case identifiers cannot be copied. Reconstruct topology and direct links from the second fixture.

PASS requires parentage without invented edges, log linkage only through valid context, correct exemplar-versus-aggregate separation, explicit resource mismatches, and a bounded actionable instrumentation gap.

15. Sources, freshness, and claim limits

Current official OpenTelemetry documentation still describes OTel as a vendor-neutral framework for traces, metrics, and logs. Context-propagation guidance explains how context enables correlation across processes and services. The logs specification defines trace-context fields for correlation, and the metrics specification describes exemplars as context-bearing measurements that can connect metrics with traces.

These sources were rechecked on 2026-09-19 and are due for review on 2026-10-17. Nothing here implies complete observability or mandates a specific backend; any link without adequate context remains OPEN.