Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31649

Spread partitions evenly to spark executors

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.4
    • Fix Version/s: None
    • Component/s: Spark Core
    • Labels:
      None

      Description

      The year is 2020 and I am still trying to find a solution to this. I totally understand what Thunder Stumpges was trying to achieve and I am trying to achieve the same. For a tool like spark, it is unacceptable not to be able to distribute the created partitions to the executors evenly. You know, we can create a custom partitioner to distribute the data to the partitions evenly by creating our own partition index. I was under the impression that a similar approach could be applied to spread these partitions to the executors evenly(using some sort of executor index for selection of executors during partition distribution). I have been googling this for a day now and I am very disappointed to say that up to now this seems to be not possible.

      Note: I am disappointed that the issue below was put into resolved state without actually doing anything about it.

      https://issues.apache.org/jira/browse/SPARK-19371

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              honor serdar onur
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: