Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2438

Use Class.cast when creating HiveConf object in ParquetJob

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.7
    • Fix Version/s: 1.5.0
    • Component/s: hive-integration
    • Labels:

      Description

      To create an instance of HiveMetaStoreClient, Sqoop needs to create an instance of the HiveConf class. That class is difficult to pass around, so we have specific code in the ParquetJob.addHiveDelegationToken method to create an instance of that class specifically. Using Class<?>.cast(...), we should be able to achieve the same result, with less code.

      // Need to use reflection since there's no compile time dependency on the client libs.
          Class<?> HiveConfClass;
          Class<?> HiveMetaStoreClientClass;
      
          try {
            HiveMetaStoreClientClass = Class.forName(HIVE_METASTORE_CLIENT_CLASS);
          } catch (ClassNotFoundException ex) {
            LOG.error("Could not load " + HIVE_METASTORE_CLIENT_CLASS
                + " when adding hive delegation token. "
                + "Make sure HIVE_CONF_DIR is set correctly.", ex);
            throw new RuntimeException("Couldn't fetch delegation token.", ex);
          }
      
          try {
            HiveConfClass = Class.forName(HiveConfig.HIVE_CONF_CLASS);
          } catch (ClassNotFoundException ex) {
            LOG.error("Could not load " + HiveConfig.HIVE_CONF_CLASS
                + " when adding hive delegation token."
                + " Make sure HIVE_CONF_DIR is set correctly.", ex);
            throw new RuntimeException("Couldn't fetch delegation token.", ex);
          }
      
          try {
            Object client = HiveMetaStoreClientClass.getConstructor(HiveConfClass).newInstance(
                HiveConfClass.getConstructor(Configuration.class, Class.class).newInstance(conf, Configuration.class)
            );
            // getDelegationToken(String kerberosPrincial)
            Method getDelegationTokenMethod = HiveMetaStoreClientClass.getMethod("getDelegationToken", String.class);
            Object tokenStringForm = getDelegationTokenMethod.invoke(client, UserGroupInformation.getLoginUser().getShortUserName());
      
            // Load token
            Token<DelegationTokenIdentifier> metastoreToken = new Token<DelegationTokenIdentifier>();
            metastoreToken.decodeFromUrlString(tokenStringForm.toString());
            conf.getCredentials().addToken(new Text(HIVE_METASTORE_TOKEN_ALIAS), metastoreToken);
      
            LOG.debug("Successfully fetched hive metastore delegation token. " + metastoreToken);
          } catch (Exception ex) {
            LOG.error("Couldn't fetch delegation token.", ex);
            throw new RuntimeException("Couldn't fetch delegation token.", ex);
          }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              abec Abraham Elmahrek
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: