Blocker Description
When push based shuffle is enabled, efficient fetch of merged mapper shuffle output happens.
Unfortunately, this currently interacts badly with spark.sql.adaptive.fetchShuffleBlocksInBatch, potentially causing shuffle fetch to hang and/or duplicate data to be fetched, causing correctness issues.
Given batch fetch does not benefit spark stages reading merged blocks when push based shuffle is enabled, ShuffleBlockFetcherIterator.doBatchFetch can be disabled when push based shuffle is enabled.
Thx to Ngone51 for surfacing this issue.
+CC Gengliang.Wang
