Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4592

Pig 0.15 stopped working with Hadoop 1.x

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.15.0
    • Fix Version/s: 0.15.0
    • Component/s: build
    • Labels:
      None
    • Environment:

      Hadoop 1.x and pig.jar retrieved from Maven

    • Hadoop Flags:
      Reviewed

      Description

      It looks like the Maven jar for Pig 0.15 is built against Hadoop 1.x instead of Hadoop 2.x.
      Running Pig 0.15 against a Hadoop 1 environment (which works fine in 0.14) triggers a storm of class not found exceptions for classes in Hadoop 2 (like org/apache/hadoop/mapreduce/task/JobContextImpl).

      The culprit seems to be the HadoopShim included in 0.15 which is different from 0.14 (and previous) - it's part of hadoop23 instead of hadoop20 package (whatever that means).

      As it stands right now, using Pig 0.15 from Maven (common case) with Hadoop 1.x is impossible.

        Attachments

        1. PIG-4592-1.patch
          0.9 kB
          Jianyong Dai

          Activity

            People

            • Assignee:
              daijy Jianyong Dai
              Reporter:
              costin Costin Manolache
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: