Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21369

Don't use Scala classes in external shuffle service

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.1, 2.3.0
    • Component/s: Shuffle, YARN
    • Labels:
      None

      Description

      Right now the external shuffle service uses Scala Tuple2. However, the Scala library won't be shaded into the yarn shuffle assembly jar. Then when the codes are called, it will throw ClassNotFoundException.

      Right now it's safe because we disabled spark.reducer.maxReqSizeShuffleToMem by default. However, to allow using spark.reducer.maxReqSizeShuffleToMem for Yarn users, we should remove all usages of Tuples.

        Attachments

          Activity

            People

            • Assignee:
              zsxwing Shixiong Zhu
              Reporter:
              zsxwing Shixiong Zhu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: