Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3930

"java.io.IOException: Cannot initialize Cluster" in local mode with hadoopversion=23 dependencies

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: build
    • Labels:
      None

      Description

      Steps to reproduce with script wc.pig

      A = load 'README.txt';
      B = foreach A generate flatten(TOKENIZE((chararray)$0)) as word;
      C = group B by word;
      D = foreach C generate COUNT(B), group;
      store D into 'wc.out';
      
      1. $ cd workspace/pig-trunk
      2. $ ant clean -Dhadoopversion=23 jar
      3. $ bin/pig -x local ~/wc.pig

      The problem is that in the dependency ./build/ivy/lib/Pig/hadoop-mapreduce-client-common-2.0.3-alpha.jar the file META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider lacks the entry org.apache.hadoop.mapred.LocalClientProtocolProvider.

      When the dependency is folded into fat pig.jar src/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider is overwritten.

      This can be fixed by using duplicate="preserve" in ant jar task.

        Attachments

        1. PIG-3930.v01.patch
          0.7 kB
          Gera Shegalov

          Activity

            People

            • Assignee:
              jira.shegalov Gera Shegalov
              Reporter:
              jira.shegalov Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: