Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3115 Improve task broadcast latency for small tasks
  3. SPARK-3137

Use finer grained locking in TorrentBroadcast.readObject

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      TorrentBroadcast.readObject uses a global lock so only one task can be fetching the blocks at the same time.

      This is not optimal if we are running multiple stages concurrently because they should be able to independently fetch their own blocks.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zsxwing Shixiong Zhu
                Reporter:
                rxin Reynold Xin
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: