[SPARK-4584] 2x Performance regression for Spark-on-YARN

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: YARN
• Target Version/s: 1.2.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Nishkam Ravi
• Created: 25/Nov/14 02:01
• Updated: 28/Nov/14 20:17
• Resolved: 28/Nov/14 20:17

Description

Significant performance regression observed for Spark-on-YARN (upto 2x) after 1.2 rebase. The offending commit is: 70e824f750aa8ed446eec104ba158b0503ba58a9 from Oct 7th. Problem can be reproduced with JavaWordCount against a large enough input dataset in YARN cluster mode.

Issue Links

• links to: [Github] Pull Request #3484 (vanzin)