Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27651

Avoid the network when block manager fetches shuffle blocks from the same host

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Block Manager, Spark Core
    • Labels:
      None

      Description

      When a shuffle block (content) is fetched the network is always used even when it is fetched from the external shuffle service running on the same host. This can be avoided by getting the local directories of the same host executors from the external shuffle service and accessing those blocks from the disk directly.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                attilapiros Attila Zsolt Piros
                Reporter:
                attilapiros Attila Zsolt Piros
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: