[HIVE-14380] Queries on tables with remote HDFS paths fail in "encryption" checks.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Encryption
• Assignee: Mithun Radhakrishnan
• Reporter: Mithun Radhakrishnan
• Created: 29/Jul/16 00:14
• Updated: 11/May/20 09:33
• Resolved: 03/Aug/16 15:59

Description

If a table has table/partition locations set to remote HDFS paths, querying them will cause the following IAException:

2016-07-26 01:16:27,471 ERROR parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(1867)) - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to determine if hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table is encrypted: java.lang.IllegalArgumentException: Wrong FS: hdfs://foo.ygrid.yahoo.com:8020/projects/my_db/my_table, expected: hdfs://bar.ygrid.yahoo.com:8020
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2204)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2274)
...

This is because of the following code in SessionState:

SessionState.java

 public HadoopShims.HdfsEncryptionShim getHdfsEncryptionShim() throws HiveException {
    if (hdfsEncryptionShim == null) {
      try {
        FileSystem fs = FileSystem.get(sessionConf);
        if ("hdfs".equals(fs.getUri().getScheme())) {
          hdfsEncryptionShim = ShimLoader.getHadoopShims().createHdfsEncryptionShim(fs, sessionConf);
        } else {
          LOG.debug("Could not get hdfsEncryptionShim, it is only applicable to hdfs filesystem.");
        }
      } catch (Exception e) {
        throw new HiveException(e);
      }
    }

    return hdfsEncryptionShim;
  }

When the FileSystem instance is created, using the sessionConf implies that the current HDFS is going to be used. This call should instead fetch the FileSystem instance corresponding to the path being checked.

A fix is forthcoming...

(Note to self: YHIVE-860)

Issue Links

• duplicates: HIVE-14379 Queries on tables with remote HDFS paths fail in "encryption" checks.
• is duplicated by: HIVE-16874 qurey fail when try to read file from remote hdfs
• is related to: HIVE-11116 Can not select data from table which points to remote hdfs location
• is related to: HIVE-14688 Hive drop call fails in presence of TDE
• relates to: SPARK-31675 Fail to insert data to a table with remote location which causes by hive encryption check
• relates to: HIVE-16874 qurey fail when try to read file from remote hdfs