[SOLR-5969] Enable distributed tracing of requests

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Gregg Donovan
• Created: 07/Apr/14 19:46
• Updated: 22/Sep/19 18:49
• Resolved: 22/Sep/19 18:49

Description

Enable users to add diagnostic information to requests and trace them in the logs across servers.

We have some metadata – e.g. a request UUID – that we log to every log line using Log4J's MDC. The UUID logging allows us to connect any log lines we have for a given request across servers. Sort of like Twitter's Zipkin.

Currently we're using EmbeddedSolrServer without sharding, so adding the UUID is fairly simple, since everything is in one process and one thread. But, we're testing a sharded HTTP implementation and running into some difficulties getting this data passed around in a way that lets us trace all log lines generated by a request to its UUID.

The first thing I tried was to add the UUID by adding it to the SolrParams. This achieves the goal of getting those values logged on the shards if a request is successful, but we miss having those values in the MDC if there are other log lines before the final log line. E.g. an Exception in a custom component.

My current thought is that sending HTTP headers with diagnostic information would be very useful. Those could be placed in the MDC even before handing off to work to SolrDispatchFilter, so that any Solr problem will have the proper logging.

I.e. every additional header added to a Solr request gets a "Solr-" prefix. On the server, we look for those headers and add them to the SLF4J MDC.

Issue Links

• duplicates: SOLR-13434 OpenTracing support for Solr
• is duplicated by: SOLR-9641 Emit distributed tracing information from Solr