[SPARK-34091] Shuffle batch fetch can't be disabled once it's enabled previously

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.0
• Fix Version/s: 3.1.1
• Component/s: SQL
• Assignee: wuyi
• Reporter: wuyi
• Created: 12/Jan/21 13:31
• Updated: 12/Jan/21 15:46
• Resolved: 12/Jan/21 15:46

Description

  if (SQLConf.get.fetchShuffleBlocksInBatch) {
    dependency.rdd.context.setLocalProperty(
      SortShuffleManager.FETCH_SHUFFLE_BLOCKS_IN_BATCH_ENABLED_KEY, "true")
  }

The current code has a problem that once we set `fetchShuffleBlocksInBatch` to true first, we can never disable the batch fetch even if set `fetchShuffleBlocksInBatch` to false later.

 

Issue Links

• links to: [Github] Pull Request #31155 (Ngone51)