[HIVE-20210] Simple Fetch optimizer should lead to MapReduce when filter on non-partition column and conversion is minimal

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0, 2.3.1, 2.3.2
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Query Planning
• Assignee: Xilang Yan
• Reporter: Xilang Yan
• Due: 19/Jul/18
• Created: 19/Jul/18 07:27
• Updated: 17/Nov/22 08:55
• Resolved: 30/Jul/18 22:36

Description

When conversion is minimal, simple fetch can be used only when filter on partition column or no filter at all. But it lead to simple fetch even if filter on non-partition column. Unit test " select * from srcpart where key > 100 limit 10 " in the nonmr_fetch.q demonstration this issue – the unit test is not correct indeed(it should be Map Reduce  but in test it is Simple Fetch).

This issue lead to a serious problem when data size is huge. When conversion is more and filter on both partition column and non-partition column, it will not chech hive.fetch.task.conversion.threshold, which lead to the query to takes hours to finish. This issue doesn't exist in 1.2.1, how it works should be a magic...