Details
-
Story
-
Status: Closed
-
Major
-
Resolution: Later
-
None
-
None
-
None
Description
In distributed query execution, there are two kinds of operators:
(1) operators that exchange data between different executors or threads: examples include broadcast, shuffle.
(2) operators that process data in a single thread: examples include project, filter, group by, etc.
This ticket proposes clearly differentiating them and creating local operators in Spark. This leads to a lot of benefits: easier to test, easier to optimize data exchange, better design (single responsibility), and potentially even having a hyper-optimized single-node version of DataFrame.
Attachments
Issue Links
- is blocked by
-
SPARK-13136 Data exchange (shuffle, broadcast) should only be handled by the exchange operator
- Resolved