[SPARK-23738] Memory usage for executors

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.1.0
• Component/s: Spark Core, Spark Submit
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: assia ydroudj
• Created: 19/Mar/18 16:28
• Updated: 21/May/19 04:16
• Resolved: 21/May/19 04:16

Description

Hi, 

I'm running a spark cluster with 3 nodes (one as master+ worker) and 2 other workers. each worker has one executor. Then, I execute the pagerank example implemented in spark. I need to gather the memory usage by each executor while running the application and gather them into a file to further analysis. 

How can I do it please? 

One idea in my mind is to get the PID  of executor ans driver processes and then, use a linux command line to get this information..is it right?

please, guide me