Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15860

Improvements for HBASE-14280 - Fixing Bulkload for HDFS HA Clusters

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: util
    • Labels:
      None

      Description

      HBASE-14280 introduced fix for bulkload failures when referring a remote cluster name service id if "bulkloading" from a HA cluster.

      HBASE-14280 solution on FSHDFSUtils.getNNAddresses was to invoke DFSUtil.getNNServiceRpcAddressesForCluster instead of DFSUtil.getNNServiceRpcAddresses. This works for hadoop 2.6 and above.

      Proposed change here is to use "DFSUtil.getRpcAddressesForNameserviceId" instead, which already returns only addresses for specific nameservice informed. This is available since hadoop 2.4.

      Sample proposal on FSHDFSUtils.getNNAddresses:
      ...

           String nameServiceId = serviceName.split(":")[1];
              if (dfsUtilClazz == null) {
                dfsUtilClazz = Class.forName("org.apache.hadoop.hdfs.DFSUtil");
              }
              if (getNNAddressesMethod == null) {
                getNNAddressesMethod =
                    dfsUtilClazz.getMethod("getRpcAddressesForNameserviceId", Configuration.class,
                      String.class, String.class);
              }
      
              Map<String, InetSocketAddress> nnMap =
                  (Map<String, InetSocketAddress>) getNNAddressesMethod
                              .invoke(null, conf, nameServiceId, null);
              for (Map.Entry<String, InetSocketAddress> e2 : nnMap.entrySet()) {
                  InetSocketAddress addr = e2.getValue();
                  addresses.add(addr);
              }
      ...
      

      Will also add test conditions for FSHDFSUtils.isSameHdfs to verify scenario when multiple name service ids are defined.

        Attachments

        1. 0001-HBASE-15860.master.002.patch
          6 kB
          Wellington Chevreuil
        2. HBASE-15860.master.002.patch
          6 kB
          Wellington Chevreuil
        3. HBASE-15860.master.002.patch
          6 kB
          Wellington Chevreuil
        4. HBASE-15860.patch
          5 kB
          Wellington Chevreuil

          Activity

            People

            • Assignee:
              wchevreuil Wellington Chevreuil
              Reporter:
              wchevreuil Wellington Chevreuil
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: