Details
-
Bug
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
3.1.1
-
None
-
None
Description
Today ExternalBlockHandler exposes a few Timer metrics:
// Time latency for open block request in ms
private final Timer openBlockRequestLatencyMillis = new Timer();
// Time latency for executor registration latency in ms
private final Timer registerExecutorRequestLatencyMillis = new Timer();
// Time latency for processing fetch merged blocks meta request latency in ms
private final Timer fetchMergedBlocksMetaLatencyMillis = new Timer();
// Time latency for processing finalize shuffle merge request latency in ms
private final Timer finalizeShuffleMergeLatencyMillis = new Timer();
However these Dropwizard Timers by default use nanoseconds (documentation). It's certainly possible to extract milliseconds from them, but it seems misleading to have millis in the name here.
This causes YarnShuffleServiceMetrics to expose confusingly-named metrics like openBlockRequestLatencyMillis_count and openBlockRequestLatencyMillis_nanos. It should be up to the metrics exporter, like YarnShuffleServiceMetrics, to decide the unit and adjust the name accordingly, so the unit shouldn't be included in the name of the metric itself.
Attachments
Issue Links
- relates to
-
SPARK-35258 Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms
-
- Resolved
-
- links to
