[SPARK-21352] Memory Usage in Spark Streaming

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.1.1
• Component/s: DStreams, Spark Core, Spark Submit, YARN
• Labels: newbie
• Flags: Important
• Assignee: Unassigned
• Reporter: Shubham Gupta
• Created: 09/Jul/17 10:07
• Updated: 17/May/20 18:13
• Resolved: 09/Jul/17 13:02

Description

I am trying to figure out the memory used by executors for a Spark Streaming job. For data I am using the rest endpoint for Spark AllExecutors and just summing up the metrics totalDuration * spark.executor.memory for every executor and then emitting the final sum as the memory usage.

But this is coming out to be very small for application which ran whole day , is something wrong with the logic.Also I am using dynamic allocation and executorIdleTimeout is 5 seconds.

Also I am also assuming that if some executor was removed for due to idle timeout and then was allocated to some other task then its totalDuration will be increased by the amount of time took by the executor to execute this new task.

https://stackoverflow.com/questions/44995212/spark-streaming-memory-usage-doubts