[PIG-4592] Pig 0.15 stopped working with Hadoop 1.x

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.15.0
• Component/s: build

Environment

Hadoop 1.x and pig.jar retrieved from Maven

• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Costin Leau
• Created: 07/Jun/15 20:00
• Updated: 17/Jun/15 17:15
• Resolved: 16/Jun/15 22:22

Description

It looks like the Maven jar for Pig 0.15 is built against Hadoop 1.x instead of Hadoop 2.x.
Running Pig 0.15 against a Hadoop 1 environment (which works fine in 0.14) triggers a storm of class not found exceptions for classes in Hadoop 2 (like org/apache/hadoop/mapreduce/task/JobContextImpl).

The culprit seems to be the HadoopShim included in 0.15 which is different from 0.14 (and previous) - it's part of hadoop23 instead of hadoop20 package (whatever that means).

As it stands right now, using Pig 0.15 from Maven (common case) with Hadoop 1.x is impossible.