Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.4.4
-
None
Description
The year is 2020 and I am still trying to find a solution to this. I totally understand what thunderstumpges was trying to achieve and I am trying to achieve the same. For a tool like spark, it is unacceptable not to be able to distribute the created partitions to the executors evenly. You know, we can create a custom partitioner to distribute the data to the partitions evenly by creating our own partition index. I was under the impression that a similar approach could be applied to spread these partitions to the executors evenly(using some sort of executor index for selection of executors during partition distribution). I have been googling this for a day now and I am very disappointed to say that up to now this seems to be not possible.
Note: I am disappointed that the issue below was put into resolved state without actually doing anything about it.