[CAMEL-19667] camel-tracing: Context is not propagated from exchange header to OpenTelemetry Context

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.20.5, 3.20.6, 3.21.0, 4.0-M3
• Fix Version/s: 4.x
• Component/s: camel-opentelemetry, camel-tracing
• Labels: bug opentelemetry tracing
• Estimated Complexity: Moderate

Language

Java

• Assignee: Unassigned
• Reporter: Roman Kvasnytskyi
• Created: 27/Jul/23 12:30
• Updated: 22/Mar/24 14:42

Description

I am using OpenTelemetry Agent for tracing along with camel-opentelemetry-starter that configures OpenTelemetryTracer for Camel aligned with camel-tracing. 

I see my spans for Camel inside a single trace, but after control is passed to the Processor process method next spans are disassociated from the trace and are created in the separate trace.

The tracing context does not seem to get propagated, and the resulting spans end up being disassociated. For example:

from("timer:tick?period=5s)
    .process("myProcessor");    

public class MyProcessor implements Processor {

    private final HttpClient someClient = new HttpClient();

    @Override
    public void process(Exchange exchange) {
        // http client is instrumented and also produces spans

        someClient.get('/example');
    }
}

This results in 2 spans. One for timer:tick & another for a client call. The problem is that the parent span for client calls is not set, so they appear as 2 distinct traces. 

My exchange headers contain traceparent header with all data which should be put inside the OpenTelemetry context, but they do not.

I have come up with a workaround. The idea is trivial - get traceparent header if it is present in exchange, parse trace metadata from it, create a SpanContext object, and put it as a parent for the current OpenTelemetry context.

It looks like this:

public class TraceEnrichingProcessor implements Processor {

    private final Processor delegate;

    public TraceEnrichingProcessor(Processor delegate) {
        this.delegate = delegate;
    }

    @Override
    public void process(Exchange exchange) throws Exception {
        // Get the existing traceparent header from the Exchange
        String traceparent = exchange.getIn().getHeader("traceparent", String.class);
        if (traceparent != null && !traceparent.isEmpty()) {
            // Extract the traceId, parentSpanId and sampleFlag
            String[] parts = traceparent.split("-");
            String traceId = parts[1];
            String parentSpanId = parts[2];
            boolean isSampled = parts[3].equals("01");

            // Create the parent SpanContext
            SpanContext parentContext = SpanContext.create(
                traceId,
                parentSpanId,
                isSampled ? TraceFlags.getSampled() : TraceFlags.getDefault(),
                TraceState.getDefault()
            );

            // Attach the parent SpanContext to the current Context
            try (Scope scope = Context.current().with(Span.wrap(parentContext)).makeCurrent()) {
                // Now, the current Context has the parent SpanContext attached,
                // and any new spans created within this scope will use it as their parent
                
                // Pass control to the delegate processor
                delegate.process(exchange);
            }
        } else {
            // If no traceparent header is found, just delegate without modifying the Context
            delegate.process(exchange);
        }
    }
} 

Inside OpenTelemetry Agent, they dropped support of Camel 3.x+, because Camel provides its module for tracing, and they will not help with it. Link

I wonder if that can be done inside Camel to propagate context from the processor implicitly without manual propagation of context. 

UPD1: I have verified that these behaviour is consistent across Camel 3.20.5, 3.20.6, 3.21.0 and 4.0.0-RC1

Issue Links

• relates to: CAMEL-20603 camel-micrometer - Micrometer observation component losses spans during tracing
• relates to: CAMEL-19469 camel-opentelemetry: Context not propagated correctly when tracing bean or processor invocations with @WithSpan