HBase
  1. HBase
  2. HBASE-4398

If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.4
    • Fix Version/s: 0.94.0, 0.95.1
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.

      We can reproduce the problem by the following instructions:

      - Add HRegionPartitioner.class to the 4th argument of
      
      TableMapReduceUtil#initTableReducerJob()
      
      at line around 133
      in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
      
      - Change or remove "hbase.zookeeper.property.clientPort" property
      
      in hbase-site.xml ( for example, changed to 12345 ).
      
      - run testMultiRegionTable()
      

      Then I got error messages as following:

      2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection opening connection to ZooKeeper with ensemble (localhost:12345)
      2011-09-12 22:28:51,022 INFO  [Thread-832] zookeeper.RecoverableZooKeeper(89): The identifier of this process is 43200@imac.local
      2011-09-12 22:28:51,123 WARN  [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      2011-09-12 22:28:51,123 INFO  [Thread-832] zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after sleeping 1000 ms
      
       =====
      
      2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d closed
      2011-09-12 22:29:02,422 WARN  [Thread-832] mapred.LocalJobRunner$Job(256): job_local_0001
      java.lang.NullPointerException
             at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128)
             at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
             at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
             at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527)
             at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
             at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
             at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
      

      I think HTable should connect to ZooKeeper at port 21818 configured at client side instead of 12345 in hbase-site.xml
      and It might be caused by "HBaseConfiguration.addHbaseResources(conf);" in HRegionPartitioner#setConf(Configuration).

      And this might mean that all of client side configurations, also configured in hbase-site.xml, are overwritten caused by this problem.

      1. HBASE-4398.patch
        11 kB
        Takuya Ueshin

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #157 (See https://builds.apache.org/job/HBase-TRUNK-security/157/)
        HBASE-5704 HBASE-4398 mistakenly rolled back on trunk (Revision 1308964)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #157 (See https://builds.apache.org/job/HBase-TRUNK-security/157/ ) HBASE-5704 HBASE-4398 mistakenly rolled back on trunk (Revision 1308964) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #104 (See https://builds.apache.org/job/HBase-0.92-security/104/)
        HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307118)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #104 (See https://builds.apache.org/job/HBase-0.92-security/104/ ) HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307118) Result = FAILURE stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2706 (See https://builds.apache.org/job/HBase-TRUNK/2706/)
        HBASE-5704 HBASE-4398 mistakenly rolled back on trunk (Revision 1308964)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2706 (See https://builds.apache.org/job/HBase-TRUNK/2706/ ) HBASE-5704 HBASE-4398 mistakenly rolled back on trunk (Revision 1308964) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Hide
        stack added a comment -

        @Takuya Sorry about that. A temporary disable while trying to figure why trunk build was broken got committed. I fixed trunk over in HBASE-5704. Thanks for noticing this.

        Show
        stack added a comment - @Takuya Sorry about that. A temporary disable while trying to figure why trunk build was broken got committed. I fixed trunk over in HBASE-5704 . Thanks for noticing this.
        Hide
        Takuya Ueshin added a comment -

        Rolled back in trunk?

        Show
        Takuya Ueshin added a comment - Rolled back in trunk?
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2698 (See https://builds.apache.org/job/HBase-TRUNK/2698/)
        HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307116)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2698 (See https://builds.apache.org/job/HBase-TRUNK/2698/ ) HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307116) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #6 (See https://builds.apache.org/job/HBase-0.94-security/6/)
        HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307117)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #6 (See https://builds.apache.org/job/HBase-0.94-security/6/ ) HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307117) Result = SUCCESS stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #154 (See https://builds.apache.org/job/HBase-TRUNK-security/154/)
        HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307116)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #154 (See https://builds.apache.org/job/HBase-TRUNK-security/154/ ) HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307116) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #344 (See https://builds.apache.org/job/HBase-0.92/344/)
        HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307118)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #344 (See https://builds.apache.org/job/HBase-0.92/344/ ) HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307118) Result = SUCCESS stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #67 (See https://builds.apache.org/job/HBase-0.94/67/)
        HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307117)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #67 (See https://builds.apache.org/job/HBase-0.94/67/ ) HBASE-4398 If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml (Revision 1307117) Result = SUCCESS stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/HRegionPartitioner.java
        Hide
        stack added a comment -

        Thanks for the patch Takuya. I applied it to 0.92, 0.94, and trunk.

        Show
        stack added a comment - Thanks for the patch Takuya. I applied it to 0.92, 0.94, and trunk.
        Hide
        stack added a comment -

        Good one Takuya. Thats broke. I would suggest that your test case while comprehensive, it is way overkill for your fix. Its also mostly a copy of another heavy-weight test we already run. I can't believe I'm saying this but I'm going to commit this fix w/o including the test.

        Show
        stack added a comment - Good one Takuya. Thats broke. I would suggest that your test case while comprehensive, it is way overkill for your fix. Its also mostly a copy of another heavy-weight test we already run. I can't believe I'm saying this but I'm going to commit this fix w/o including the test.
        Hide
        Takuya Ueshin added a comment -

        I attatched a patch file.
        Tnis includes a patch for HRegionPartitioner and a test case.

        Show
        Takuya Ueshin added a comment - I attatched a patch file. Tnis includes a patch for HRegionPartitioner and a test case.

          People

          • Assignee:
            Takuya Ueshin
            Reporter:
            Takuya Ueshin
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development