[SPARK-16332] the history server of spark2.0-preview (may-24 build) consumes more than 1000% cpu

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.0
• Component/s: Spark Core

Environment

this is seen on both x86 (Intel(R) Xeon(R), E5-2699 ) and ppc platform IBM Power8 Habanero (Model: 8348-21C), Red Hat Enterprise Linux Server release 7.2 (Maipo), Spark2.0.0-preview (May-24, 2016)

• Flags: Important
• Assignee: Unassigned
• Reporter: Peter Liu
• Created: 30/Jun/16 19:16
• Updated: 16/Jul/16 20:05
• Resolved: 01/Jul/16 06:56

Description

the JVM instance of the Spark history server of spark2.0-preview (may-24 build) consumes more than 1000% cpu without the spark standalone cluster (of 6 nodes) being under load.
When under load (1TB input data for a SQL query scenario), the JVM instance of the Spark history server of spark2.0-preview consumes 2000% cpu (as seen with "top" on linux 3.10)
Note: can't see a proper Component selection here, surely not a Web GUI issue.

Issue Links

• duplicates: SPARK-16333 Excessive Spark history event/json data size (5GB each)