Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14380

Queries on tables with remote HDFS paths fail in "encryption" checks.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: Encryption
    • Labels:
      None

      Description

      If a table has table/partition locations set to remote HDFS paths, querying them will cause the following IAException:

      2016-07-26 01:16:27,471 ERROR parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1867)) - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table is encrypted: java.lang.IllegalArgumentException: Wrong FS: hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table, expected: hdfs://bar.ygrid.yahoo.com:8020
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2204)
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2274)
      ...
      

      This is because of the following code in SessionState:

      SessionState.java
       public HadoopShims.HdfsEncryptionShim getHdfsEncryptionShim() throws HiveException {
          if (hdfsEncryptionShim == null) {
            try {
              FileSystem fs = FileSystem.get(sessionConf);
              if ("hdfs".equals(fs.getUri().getScheme())) {
                hdfsEncryptionShim = ShimLoader.getHadoopShims().createHdfsEncryptionShim(fs, sessionConf);
              } else {
                LOG.debug("Could not get hdfsEncryptionShim, it is only applicable to hdfs filesystem.");
              }
            } catch (Exception e) {
              throw new HiveException(e);
            }
          }
      
          return hdfsEncryptionShim;
        }
      

      When the FileSystem instance is created, using the sessionConf implies that the current HDFS is going to be used. This call should instead fetch the FileSystem instance corresponding to the path being checked.

      A fix is forthcoming...

      (Note to self: YHIVE-860)

        Attachments

        1. HIVE-14380.1.patch
          8 kB
          Mithun Radhakrishnan
        2. HIVE-14380.branch-1.2.patch
          17 kB
          Mithun Radhakrishnan

          Issue Links

            Activity

              People

              • Assignee:
                mithun Mithun Radhakrishnan
                Reporter:
                mithun Mithun Radhakrishnan
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: