[PIG-4633] Update hadoop version to enable Spark output statistics

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Xianda Ke
• Reporter: Xianda Ke
• Created: 11/Jul/15 15:54
• Updated: 21/Jun/17 09:18
• Resolved: 14/Jul/15 05:10

Description

Spark support output statistics from 1.3.0 (SPARK-3179. Add task OutputMetrics)

SparkHadoopUtil.scala

stats.map(Utils.invoke(classOf[Statistics], _, "getThreadStatistics"))

Spark invoke hadoop's function getThreadStatistics. But, this method was added into hadoop from version 2.5.0 (HADOOP-10688)

The version of hadoop in ivy/libraries.properties should be 2.5.0 +