HBase
  1. HBase
  2. HBASE-4297

TableMapReduceUtil overwrites user supplied options

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.4
    • Fix Version/s: 0.90.5
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Job configuration is overwritten by hbase-default and hbase-site in TableMapReduceUtil.initTable(Mapper|Reducer)Job, causing unexpected behavior in the following code:

      Configuration conf = HBaseConfiguration.create();
      
      // change keyvalue size
      conf.setInt("hbase.client.keyvalue.maxsize", 20971520);
      
      Job job = new Job(conf, ...);
      
      TableMapReduceUtil.initTableMapperJob(...);
      
      // the job doesn't have the option changed, uses it from hbase-site or hbase-default
      job.submit();
      

      Although in this case it could be fixed by moving the set() after initTableMapperJob(), in case where user wants to change some option using GenericOptionsParser and -D this is impossible, making this cool feature useless.

      In the 0.20.x era this code behaved as expected. The solution of this problem should be that we don't overwrite the options, but just read them if they are missing.

        Activity

        Hide
        Jan Van Besien added a comment -

        I think this issue is actually not fixed in all code paths. There are still overloaded initTableMapperJob methods in TableMapReduceUtil that do not merge the configuration.

        I can still reproduce this issue with hbase-0.94.6.

        Show
        Jan Van Besien added a comment - I think this issue is actually not fixed in all code paths. There are still overloaded initTableMapperJob methods in TableMapReduceUtil that do not merge the configuration. I can still reproduce this issue with hbase-0.94.6.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2188 (See https://builds.apache.org/job/HBase-TRUNK/2188/)
        HBASE-4297 TableMapReduceUtil overwrites user supplied options

        stack :
        Files :

        • /hbase/trunk/CHANGES.txt
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2188 (See https://builds.apache.org/job/HBase-TRUNK/2188/ ) HBASE-4297 TableMapReduceUtil overwrites user supplied options stack : Files : /hbase/trunk/CHANGES.txt /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        Hide
        stack added a comment -

        Committed to TRUNK. Thank you for the patch Jan.

        Show
        stack added a comment - Committed to TRUNK. Thank you for the patch Jan.
        Hide
        Jan Lukavsky added a comment -

        Hi Stack,

        I've tested the patch against cdh3u1 and it works fine for us. I haven't seen any negative side affects so far.

        Show
        Jan Lukavsky added a comment - Hi Stack, I've tested the patch against cdh3u1 and it works fine for us. I haven't seen any negative side affects so far.
        Hide
        stack added a comment -

        +1 on the patch. Does it work for you Jan? Any side effects that you've seen?

        Show
        stack added a comment - +1 on the patch. Does it work for you Jan? Any side effects that you've seen?
        Hide
        Jan Lukavsky added a comment -

        Attaching patch for this issue.

        Show
        Jan Lukavsky added a comment - Attaching patch for this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Jan Lukavsky
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development