Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.1
-
None
-
None
Description
Currently when both Push based shuffle and AQE is enabled and when partial set of map indexes are requested to MapOutputTracker this is delegated the regular shuffle instead of push based shuffle reading map blocks. This is because blocks from mapper in push based shuffle are merged out of order due to which its hard to only get the matching blocks of the reduce partition for the requested start and end map indexes.