Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16827

Stop reporting spill metrics as shuffle metrics

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • None
    • Shuffle, Spark Core

    Description

      One of our hive job which looks like this -

       SELECT  userid
           FROM  table1 a
           JOIN table2 b
            ON    a.ds = '2016-07-15'
            AND  b.ds = '2016-07-15'
            AND  a.source_id = b.id
      

      After upgrade to Spark 2.0 the job is significantly slow. Digging a little into it, we found out that one of the stages produces excessive amount of shuffle data. Please note that this is a regression from Spark 1.6. Stage 2 of the job which used to produce 32KB shuffle data with 1.6, now produces more than 400GB with Spark 2.0. We also tried turning off whole stage code generation but that did not help.

      PS - Even if the intermediate shuffle data size is huge, the job still produces accurate output.

      Attachments

        Activity

          People

            chobrian Brian Cho
            sitalkedia@gmail.com Sital Kedia
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: