Details
-
Umbrella
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
None
-
None
Description
Currently spark only provides little memory usage information (RDD cache on webUI) for the executors. User have no idea on what is the memory consumption when they are running spark applications with a lot of memory used in spark executors. Especially when they encounter the OOM, it’s really hard to know what is the cause of the problem. So it would be helpful to give out the detail memory consumption information for each part of spark, so that user can clearly have a picture of where the memory is exactly used.
The memory usage info to expose should include but not limited to shuffle, cache, network, serializer, etc.
User can optionally choose to open this functionality since this is mainly for debugging and tuning.
Attachments
Attachments
Issue Links
- is related to
-
SPARK-8735 Expose metrics for runtime memory usage
-
- Resolved
-
- relates to
-
-
- Open
-
- links to
1.
|
expose network layer memory usage |
|
Resolved | Saisai Shao |
2.
|
Add an additional WebUI Tab for Memory Usage |
|
Resolved | Unassigned |
3.
|
Log the memory usage info into history server |
|
Resolved | Unassigned |
4.
|
Include memory usage for each job & stage |
|
Resolved | Unassigned |
5.
|
Expose Kryo serializer buffer size |
|
Resolved | Unassigned |
6.
|
Dumping the memory info when an executor dies abnormally |
|
Resolved | Unassigned |
7.
|
update Netty version to "4.0.29.Final" for Netty Metrics |
|
Resolved | Zhang, Liye |
8.
|
Expose Netty memory usage via Metrics System |
|
Resolved | Saisai Shao |
9.
|
Exposing java.nio bufferedPool memory metrics to metrics system |
|
Resolved | Srinivas |