Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15129

Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Before HBASE-15129, if somehow hbase-default.xml is not on classpath, default values for hbase.fs.tmp.dir and hbase.bulkload.staging.dir are left empty. After HBASE-15129, default values of both properties are set to "/user/<user.name>/hbase-staging".
      Show
      Before HBASE-15129 , if somehow hbase-default.xml is not on classpath, default values for hbase.fs.tmp.dir and hbase.bulkload.staging.dir are left empty. After HBASE-15129 , default values of both properties are set to "/user/<user.name>/hbase-staging".

      Description

      One of our users has observed below error when our cluster upgrades from 0.98.12 to 1.1.2:

      java.lang.IllegalArgumentException: Can not create a Path from a null string
              at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
              at org.apache.hadoop.fs.Path.<init>(Path.java:134)
              at org.apache.hadoop.fs.Path.<init>(Path.java:88)
              at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:639)
              at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:489)
      

      And checking the application code:

        private Job createJob(Configuration jobConf) throws IOException {
          String tableName = jobConf.get(TABLE_NAME_KEY);
          Job job = new Job(jobConf);
          Configuration tableConf = HBaseConfiguration.create();
          HTable table = new HTable(tableConf, tableName);
          HFileOutputFormat2.configureIncrementalLoad(job, table);
          return job;
        }
      

      We could see the user has his own hbase-independent job configuration, so our current code in HFileOutputFormat2:

      Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" + UUID.randomUUID());
      

      will return a null string and cause the above exception.

      We propose to add default value in the code rather than depend on hbase-default.xml in this JIRA.

      This is an improvement for HBASE-13625

        Attachments

        1. HBASE-15129.patch
          6 kB
          Yu Li
        2. HBASE-15129_v2.patch
          4 kB
          Yu Li

          Activity

            People

            • Assignee:
              liyu Yu Li
              Reporter:
              liyu Yu Li
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: