[HIVE-12664] Bug in reduce deduplication optimization causing ArrayOutOfBoundException

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.1, 1.2.1
• Fix Version/s: 2.0.0
• Component/s: Hive
• Assignee: Johan Gustavsson
• Reporter: Johan Gustavsson
• Created: 14/Dec/15 03:01
• Updated: 16/Feb/16 23:51
• Resolved: 07/Jan/16 16:28

Description

The optimisation check for reduce deduplication only checks the first child node for join and the check itself also contains a major bug causing ArrayOutOfBoundException no matter what.

Sample data table form:

time	user	host	path	referer	code	agent	size	method
int	string	string	string	string	bigint	string	bigint	string

Sample query

SELECT 
  t1.host,
  COUNT(DISTINCT t1.`date`) AS login_count,
  MAX(t2.code) AS code,
  unix_timestamp() AS time
FROM (
    SELECT 
      HOST,
      MIN(time) AS DATE
    FROM
      www_access
    WHERE
      HOST IS NOT NULL
    GROUP BY
      HOST
  ) t1
JOIN (
    SELECT 
      HOST,
      MIN(time) AS code
    FROM
      www_access
    WHERE
      HOST IS NOT NULL
    GROUP BY
      HOST
  ) t2
  ON t1.host = t2.host
GROUP BY
  t1.host