HBase
  1. HBase
  2. HBASE-4297

TableMapReduceUtil overwrites user supplied options

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.4
    • Fix Version/s: 0.90.5
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Job configuration is overwritten by hbase-default and hbase-site in TableMapReduceUtil.initTable(Mapper|Reducer)Job, causing unexpected behavior in the following code:

      Configuration conf = HBaseConfiguration.create();
      
      // change keyvalue size
      conf.setInt("hbase.client.keyvalue.maxsize", 20971520);
      
      Job job = new Job(conf, ...);
      
      TableMapReduceUtil.initTableMapperJob(...);
      
      // the job doesn't have the option changed, uses it from hbase-site or hbase-default
      job.submit();
      

      Although in this case it could be fixed by moving the set() after initTableMapperJob(), in case where user wants to change some option using GenericOptionsParser and -D this is impossible, making this cool feature useless.

      In the 0.20.x era this code behaved as expected. The solution of this problem should be that we don't overwrite the options, but just read them if they are missing.

        Activity

        Jan Lukavsky created issue -
        Jan Lukavsky made changes -
        Field Original Value New Value
        Attachment HBASE-4297.patch [ 12492210 ]
        Jan Lukavsky made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        stack made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Jonathan Hsieh made changes -
        Fix Version/s 0.90.5 [ 12317145 ]
        stack made changes -
        Fix Version/s 0.90.6 [ 12319200 ]
        Fix Version/s 0.90.5 [ 12317145 ]
        stack made changes -
        Fix Version/s 0.90.5 [ 12317145 ]
        Fix Version/s 0.90.6 [ 12319200 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Jan Lukavsky
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development