[CELEBORN-1625] Add new parameters to the pushOrMergeData method for gluten

• Type: New Feature
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 0.5.2
• Labels: pull-request-available
• Assignee: Kerwin Zhang
• Reporter: Kerwin Zhang
• Created: 27/Sep/24 08:10
• Updated: 30/Sep/24 01:51
• Resolved: 30/Sep/24 01:51

Description

Currently, in the scenario of Gluten Fallback, it is possible for a stage to have both Native and Java shuffles simultaneously. Since the ShuffleClient is a singleton, the compression settings for the ShuffleClient will only utilize either the Native or Java configuration. By adding new parameters, we aim to allow Gluten to control whether compression should be applied in this scenario.

Issue Links

• links to: GitHub Pull Request #2769