Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
We recently noticed that the mapred.ShuffleMetrics.ShuffleConnections metric rises indefinitely (see attached graph), despite supposedly being a gauge measuring the number of currently open connections:
ShuffleHandler.java
@Metric("# of current shuffle connections")
MutableGaugeInt shuffleConnections;
It seems this is because the metric is incremented once for each map file sent, but decremented once for each request. Thus a request which fetches multiple map files permanently increments shuffleConnections by (mapsFetched - 1).
Attachments
Attachments
Issue Links
- is duplicated by
-
MAPREDUCE-6918 ShuffleMetrics.ShuffleConnections Gauge Metric Climbs Infinitely
- Resolved