[SPARK-16827] Stop reporting spill metrics as shuffle metrics - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

Delete

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.0.0
Fix Version/s: None
Component/s: Shuffle, Spark Core
Labels:
- performance

Target Version/s:

2.0.3, 2.1.0

Description

One of our hive job which looks like this -

 SELECT  userid
     FROM  table1 a
     JOIN table2 b
      ON    a.ds = '2016-07-15'
      AND  b.ds = '2016-07-15'
      AND  a.source_id = b.id

After upgrade to Spark 2.0 the job is significantly slow. Digging a little into it, we found out that one of the stages produces excessive amount of shuffle data. Please note that this is a regression from Spark 1.6. Stage 2 of the job which used to produce 32KB shuffle data with 1.6, now produces more than 400GB with Spark 2.0. We also tried turning off whole stage code generation but that did not help.

PS - Even if the intermediate shuffle data size is huge, the job still produces accurate output.