Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1112

When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.9.0, 1.0.0
    • Fix Version/s: 0.9.2, 1.0.1, 1.1.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When I set the spark.akka.frameSize to something over 10, the messages sent from the executors to the driver completely block the execution if the message is bigger than 10MiB and smaller than the frameSize (if it's above the frameSize, it's ok)

      Workaround is to set the spark.akka.frameSize to 10. In this case, since 0.8.1, the blockManager deal with the data to be sent. It seems slower than akka direct message though.

      The configuration seems to be correctly read (see actorSystemConfig.txt), so I don't see where the 10MiB could come from

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mengxr Xiangrui Meng
                Reporter:
                guillaumepitel Guillaume Pitel
              • Votes:
                2 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: