Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2885

TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 0.12.0
    • None
    • Hadoop 1.0.3, CentOS 6.3 64 bit

    • Reviewed
    • Hide
      This makes hbase.jar and zookeeper.jar no longer be embedded in pig.jar, and thus, to use HBaseStorage with pig.jar, hbase.jar and zookeeper.jar must be present in classpath. To add them to classpath, the user can set either PIG_CLASSPATH=<path_to_hbase.jar:path_to_zookeeper.jar>, or HBASE_HOME=<hbase_home_dir> and ZOOKEEPER_HOME=<zookeeper_home_dir>.
      Show
      This makes hbase.jar and zookeeper.jar no longer be embedded in pig.jar, and thus, to use HBaseStorage with pig.jar, hbase.jar and zookeeper.jar must be present in classpath. To add them to classpath, the user can set either PIG_CLASSPATH=<path_to_hbase.jar:path_to_zookeeper.jar>, or HBASE_HOME=<hbase_home_dir> and ZOOKEEPER_HOME=<zookeeper_home_dir>.

    Description

      I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.

      • Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
      • The HTable constructor in HBase 0.94 changed:
        -        HTable table = new HTable(TESTTABLE_2);
        +        HTable table = new HTable(conf, TESTTABLE_2);
        
      • The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
        @@ -541,7 +543,7 @@ public class TestJobSubmission {
                 // use the estimation
                 Configuration conf = cluster.getConfiguration();
                 HBaseTestingUtility util = new HBaseTestingUtility(conf);
        -        util.startMiniZKCluster();
        +        int clientPort = util.startMiniZKCluster().getClientPort();
                 util.startMiniHBaseCluster(1, 1); 
             
                 String query = "a = load '/passwd';" + 
        @@ -553,6 +555,7 @@ public class TestJobSubmission {
             
                 pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
                 pc.getConf().setProperty("pig.exec.reducers.max", "10");
        +        pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
                 ConfigurationValidator.validatePigProperties(pc.getProperties());
                 conf = ConfigurationUtil.toConfiguration(pc.getProperties());
                 JobControlCompiler jcc = new JobControlCompiler(pc, conf);
        

      With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.

      Attachments

        1. PIG-2885.patch
          6 kB
          Cheolsoo Park
        2. PIG-2885-2.patch
          8 kB
          Cheolsoo Park
        3. PIG-2885-3.patch
          35 kB
          Cheolsoo Park
        4. PIG-2885-4.patch
          35 kB
          Cheolsoo Park

        Issue Links

          Activity

            People

              cheolsoo Cheolsoo Park
              cheolsoo Cheolsoo Park
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: