Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20308

org.apache.spark.shuffle.FetchFailedException: Too large frame

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.1.0
    • None
    • Shuffle, Spark Core
    • None

    Description

      Spark uses custom frame decoder (TransportFrameDecoder) which does not support frames larger than 2G. This lead to fails when shuffling using large partitions.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. SPARK-5928 Remote Shuffle Blocks cannot be more than 2 GB

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              schernichkin Stanislav Chernichkin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: