Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31924

Create remote shuffle service reference implementation

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Shuffle
    • None

    Description

      People in Spark Scalability & Reliability Sync Meeting have discussed a lot about remote (disaggregated) shuffle service, and plan to do a reference implementation to help demonstrate some basic design and pave the way for a future production grade remote  shuffle service.

       

      There are already two pull requests to enhance Spark shuffle metadata API to make it easy/possible to implement remote shuffle service (PR 28616, PR 28618). Creating a remote shuffle service reference implementation will help to validate those shuffle metadata API.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              bobyangbo BoYang
              Felix Cheung Felix Cheung
              Votes:
              3 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated: