Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8202

MultiTableOutputFormat should support writing to another HBase cluster

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mapreduce
    • Labels:
      None

      Description

      This was brought up by David Koch in thread 'hbase.mapred.output.quorum ignored in Mapper job with HDFS source and HBase sink' where he wanted to import a file on HDFS from one cluster A (source) into HBase
      tables on a different cluster B (destination) using a Mapper job with an
      HBase sink.

      Here is my analysis:

      MultiTableOutputFormat doesn't extend TableOutputFormat:

      public class MultiTableOutputFormat extends OutputFormat<ImmutableBytesWritable, Mutation> {
      

      Relevant configuration w.r.t. output quorum is setup in TableOutputFormat#setConf():

        public void setConf(Configuration otherConf) {
          this.conf = HBaseConfiguration.create(otherConf);
          String tableName = this.conf.get(OUTPUT_TABLE);
          if(tableName == null || tableName.length() <= 0) {
            throw new IllegalArgumentException("Must specify table name");
          }
          String address = this.conf.get(QUORUM_ADDRESS);
          int zkClientPort = conf.getInt(QUORUM_PORT, 0);
          String serverClass = this.conf.get(REGION_SERVER_CLASS);
          String serverImpl = this.conf.get(REGION_SERVER_IMPL);
          try {
            if (address != null) {
              ZKUtil.applyClusterKeyToConf(this.conf, address);
            }
      

        Attachments

          Activity

            People

            • Assignee:
              yuzhihong@gmail.com Ted Yu
              Reporter:
              yuzhihong@gmail.com Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: