Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2885

TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
    • Environment:

      Hadoop 1.0.3, CentOS 6.3 64 bit

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This makes hbase.jar and zookeeper.jar no longer be embedded in pig.jar, and thus, to use HBaseStorage with pig.jar, hbase.jar and zookeeper.jar must be present in classpath. To add them to classpath, the user can set either PIG_CLASSPATH=<path_to_hbase.jar:path_to_zookeeper.jar>, or HBASE_HOME=<hbase_home_dir> and ZOOKEEPER_HOME=<zookeeper_home_dir>.
      Show
      This makes hbase.jar and zookeeper.jar no longer be embedded in pig.jar, and thus, to use HBaseStorage with pig.jar, hbase.jar and zookeeper.jar must be present in classpath. To add them to classpath, the user can set either PIG_CLASSPATH=<path_to_hbase.jar:path_to_zookeeper.jar>, or HBASE_HOME=<hbase_home_dir> and ZOOKEEPER_HOME=<zookeeper_home_dir>.

      Description

      I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.

      • Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
      • The HTable constructor in HBase 0.94 changed:
        -        HTable table = new HTable(TESTTABLE_2);
        +        HTable table = new HTable(conf, TESTTABLE_2);
        
      • The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
        @@ -541,7 +543,7 @@ public class TestJobSubmission {
                 // use the estimation
                 Configuration conf = cluster.getConfiguration();
                 HBaseTestingUtility util = new HBaseTestingUtility(conf);
        -        util.startMiniZKCluster();
        +        int clientPort = util.startMiniZKCluster().getClientPort();
                 util.startMiniHBaseCluster(1, 1); 
             
                 String query = "a = load '/passwd';" + 
        @@ -553,6 +555,7 @@ public class TestJobSubmission {
             
                 pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
                 pc.getConf().setProperty("pig.exec.reducers.max", "10");
        +        pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
                 ConfigurationValidator.validatePigProperties(pc.getProperties());
                 conf = ConfigurationUtil.toConfiguration(pc.getProperties());
                 JobControlCompiler jcc = new JobControlCompiler(pc, conf);
        

      With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.

        Attachments

        1. PIG-2885.patch
          6 kB
          Cheolsoo Park
        2. PIG-2885-2.patch
          8 kB
          Cheolsoo Park
        3. PIG-2885-3.patch
          35 kB
          Cheolsoo Park
        4. PIG-2885-4.patch
          35 kB
          Cheolsoo Park

          Issue Links

            Activity

              People

              • Assignee:
                cheolsoo Cheolsoo Park
                Reporter:
                cheolsoo Cheolsoo Park
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: