[SPARK-6409] It is not necessary that avoid old inteface of hive, because this will make some UDAF can not work.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.3.1, 1.4.0
• Component/s: SQL
• Labels: starter
• Target Version/s: 1.3.1
• Assignee: Zhongshuai Pei
• Reporter: Zhongshuai Pei
• Created: 19/Mar/15 09:07
• Updated: 25/Mar/15 18:12
• Resolved: 25/Mar/15 18:12

Description

I run SQL like that

CREATE TEMPORARY FUNCTION test_avg AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage'; 
  
SELECT 
    test_avg(1), 
    test_avg(substr(value,5)) 
FROM src; 

then i get a exception

15/03/19 09:36:45 ERROR CliDriver: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 6, HPC-3): java.lang.ClassCastException: org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage$AverageAggregationBuffer cannot be cast to org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator$AbstractAggregationBuffer 
        at org.apache.spark.sql.hive.HiveUdafFunction.<init>(hiveUdfs.scala:369) 
        at org.apache.spark.sql.hive.HiveGenericUdaf.newInstance(hiveUdfs.scala:214) 
        at org.apache.spark.sql.hive.HiveGenericUdaf.newInstance(hiveUdfs.scala:188) 

i find that GenericUDAFAverage used a deprecated interface AggregationBuffer that has been instead by AbstractAggregationBuffer. and spark avoid the old interface AggregationBuffer , so GenericUDAFAverage can not work.I think it is not necessary.

code in spark

  // Cast required to avoid type inference selecting a deprecated Hive API.
  private val buffer =
    function.getNewAggregationBuffer.asInstanceOf[GenericUDAFEvaluator.AbstractAggregationBuffer]

Issue Links

• links to: [Github] Pull Request #5131 (DoingDone9)