[TEZ-4521] Partition stats should be always uncompressed size

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.2
• Fix Version/s: 0.10.3
• Assignee: Shohei Okumiya
• Reporter: Shohei Okumiya
• Created: 27/Nov/23 15:24
• Updated: 28/Nov/23 15:34
• Resolved: 28/Nov/23 15:33

Description

We always put compressed size in ExternalSorter#partitionStats while we put uncompressed size in UnorderedPartitionedKVWriter#sizePerPartition. Those should have consistent semantics.

 

As far as I know, the uncompressed size is preferable because of some reasons.

1. The stats are used in FairShuffleVertexManager to configure the parallelism. The normal ShuffleVertexManager which is broadly used computes parallelism based on uncompressed size. Otherwise, we need to tune `tez.fair-shuffle-vertex-manager.desired-task-input-size` based on compressed size though `tez.shuffle-vertex-manager.desired-task-input-size` must be based on decompressed size
2. Ming pointed out we should use uncompressed size in TEZ-3206. Looks like, we missed creating a follow-up ticket

Issue Links

• links to: GitHub Pull Request #317