[CAMEL-20603] camel-micrometer - Micrometer observation component losses spans during tracing

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Information Provided
• Affects Version/s: 4.4.0
• Fix Version/s: 4.x
• Component/s: camel-micrometer
• Estimated Complexity: Unknown
• Assignee: Unassigned
• Reporter: Kutorkin Artyom
• Created: 22/Mar/24 14:15
• Updated: Yesterday 17:34
• Resolved: Yesterday 17:34

Description

I've got an issue in my project related with Camel Observation within routes. Most of the time, during request processing previous spans are losing, so multiple traces can be created for one request. Also, for unknown reasons some of the spans that were created in current request, can join previous trace where spans were missed.

Workflow:

// 1. Spring controller producing the event to SEDA dictSync
producerTemplate.send(SEDA_DICT_SYNC_START.getUri(), exchange ->
    exchange.getIn().setBody(objectMapper.writeValueAsString(entities)));

// 2. Event-driven consumer consumes the event from SEDA dictSync and processes it in dictSyncStartProcessor
from(SEDA_DICT_SYNC_START)
    .routeId("DictSyncStart")
    .unmarshal().json(JsonLibrary.Jackson)
    .onException(Exception.class).handled(true)
        .process(exchange -> handleException((String) exchange.getIn().getBody(ArrayList.class).get(0), exchange))
    .end()
    .process(dictSyncStartProcessor);

// 3. dictSyncStartProcessor processes the event and send another to SEDA dictSyncSplit
producerTemplate.send(SEDA_DICT_SYNC_SPLIT.getUri(), exchange -> {
    Message message = exchange.getIn();

    entitySync.setUpdateDatetime(updateDatetime);
    message.setHeader("destination", destination.name());
    message.setHeader("entity", objectMapper.writeValueAsString(entitySync));
    message.setHeader("messageId", messageId);
    message.setBody(data);
});

// 4. dictSyncSplit consumes event, split its body, aggregate splitted records and send aggregated record to kafka
from(SEDA_DICT_SYNC_SPLIT)
    .routeId("DictSyncSplit")
    .onException(Exception.class).handled(true).useOriginalMessage()
        .process(exchange -> {
            var entitySync = objectMapper.readValue(exchange.getIn().getHeader("entity", String.class), EntitySync.class);
            handleException(entitySync.getEntity(), exchange);
        })
    .end()
    .split(body())
    .aggregate(header("messageId"), aggregationStrategy)
        .completionSize(20)
        .completionTimeout(1000L)
    .marshal().json(JsonLibrary.Jackson)
    .to(kafka(DICTIONARY_SYNC_TOPIC));

// 5. dictionarySync route consumes event from kafka and send another to proccesedEntitySync
producerTemplate.send(kafka(PROCESSED_ENTITY_SYNC_TOPIC).getUri(), ex ->
    ex.getIn().setBody(objectMapper.writeValueAsString(entitySync)));

 

First case

Actual unexpected results:

• dictSyncSplit route observations are missing (but will be associated during next request)
• processedEntitySync is separated into another trace

After invoking this request once again, new spans were associated with first request:

 

Second case

Actual unexpected results:

• dictSyncSplit is separated into another trace
• processedEntitySync is separated into another trace

 

Observation dependencies:

implementation 'org.springframework.boot:spring-boot-starter-aop'
implementation 'io.micrometer:micrometer-tracing-bridge-otel'
implementation 'io.opentelemetry:opentelemetry-exporter-zipkin'
implementation 'org.apache.camel.springboot:camel-observation-starter:4.4.0'

Spring boot version: 3.2.2

Issue Links

• is related to: CAMEL-19667 camel-tracing: Context is not propagated from exchange header to OpenTelemetry Context