[HIVE-9392] JoinStatsRule miscalculates join cardinality as incorrect NDV is used due to column names having duplicated fqColumnName

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Component/s: Physical Optimizer
• Assignee: Pengcheng Xiong
• Reporter: Mostafa Mokhtar
• Created: 15/Jan/15 20:37
• Updated: 23/May/15 21:03
• Resolved: 23/May/15 21:03

Description

In JoinStatsRule.process the join column statistics are stored in HashMap joinedColStats, the key used which is the ColStatistics.fqColName is duplicated between join column in the same vertex, as a result distinctVals ends up having duplicated values which negatively affects the join cardinality estimation.

The duplicate keys are usually named KEY.reducesinkkey0.

Issue Links

• is blocked by: HIVE-9512 HIVE-9327 causing regression in stats annotation
• is related to: HIVE-8769 Physical optimizer : Incorrect CE results in a shuffle join instead of a Map join (PK/FK pattern not detected)
• relates to: HIVE-10107 Union All : Vertex missing stats resulting in OOM and in-efficient plans
• links to: Review Board