Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33235 Push-based Shuffle Improvement Tasks
  3. SPARK-38064

Push-based shuffle's internal implementation details should not be exposed as API

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.1
    • None
    • Shuffle
    • None

    Description

      Currently the changes added in Dependency.scala for Push-based shuffle are all exposed as Developer API. For eg: `newShuffleMergeState`, `setMergerLocs`, `shuffleMergeFinalized` etc Some of these might be just internal implementation details which should not be leaked. This JIRA Is to appropriately mark them private as needed.

      cc mridulm80mshencsinghNgone51tgraves

      Attachments

        Activity

          People

            Unassigned Unassigned
            vsowrirajan Venkata krishnan Sowrirajan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: