[HIVE-25615] Hive on tez will generate at least one MapContainer per 0 length file

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2, 4.0.0
• Component/s: Query Planning, Query Processor, Tez
• Labels: pull-request-available

Environment

hive-3.1.2
tez-0.10.1

• Assignee: ZhengBowen
• Reporter: JackYan
• Created: 15/Oct/21 03:23
• Updated: 01/Jan/24 00:22

Description

When tez read a table with many partitions and those partitions contain 0 length file only, ColumnarSplitSizeEstimator will return Integer.MAX_VALUE bytes length for every 0 length file.Then,TezSplitGrouper will treat those files as big files,and generate at least one MapContainer per 0 file to handle it.This is incorrect and even wasteful.

Issue Links

• links to: GitHub Pull Request #2723
• links to: GitHub Pull Request #4738
• links to: PR HIVE-25615