Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14688

Hive drop call fails in presence of TDE

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 1.2.1, 2.0.0
    • None
    • Security
    • None

    Description

      This should be committed to when Hive moves to Hadoop 2.8

      In Hadoop 2.8.0 TDE trash collection was fixed through HDFS-8831. This enables us to make drop table calls for Hive managed tables where Hive metastore warehouse directory is in encrypted zone. However even with the feature in HDFS, Hive drop table currently fail:

      $ hdfs crypto -listZones
      /apps/hive/warehouse  key2 
      $ hdfs dfs -ls /apps/hive/warehouse
      Found 1 items
      drwxrwxrwt   - hdfs hdfs          0 2016-09-01 02:54 /apps/hive/warehouse/.Trash
      hive> create table abc(a string, b int);
      OK
      Time taken: 5.538 seconds
      hive> dfs -ls /apps/hive/warehouse;
      Found 2 items
      drwxrwxrwt   - hdfs   hdfs          0 2016-09-01 02:54 /apps/hive/warehouse/.Trash
      drwxrwxrwx   - deepesh hdfs          0 2016-09-01 17:15 /apps/hive/warehouse/abc
      hive> drop table if exists abc;
      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Unable to drop default.abc because it is in an encryption zone and trash is enabled.  Use PURGE option to skip trash.)
      

      The problem lies here:

      metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
      private void checkTrashPurgeCombination(Path pathToData, String objectName, boolean ifPurge)
      ...
            if (trashEnabled) {
              try {
                HadoopShims.HdfsEncryptionShim shim =
                  ShimLoader.getHadoopShims().createHdfsEncryptionShim(FileSystem.get(hiveConf), hiveConf);
                if (shim.isPathEncrypted(pathToData)) {
                  throw new MetaException("Unable to drop " + objectName + " because it is in an encryption zone" +
                    " and trash is enabled.  Use PURGE option to skip trash.");
                }
              } catch (IOException ex) {
                MetaException e = new MetaException(ex.getMessage());
                e.initCause(ex);
                throw e;
              }
            }
      

      As we can see that we are making an assumption that delete wouldn't be successful in encrypted zone. We need to modify this logic.

      Attachments

        1. HIVE-14688.4.patch
          15 kB
          Wei Zheng
        2. HIVE-14688.3.patch
          8 kB
          Wei Zheng
        3. HIVE-14688.2.patch
          6 kB
          Wei Zheng
        4. HIVE-14688.1.patch
          1 kB
          Wei Zheng

        Issue Links

          Activity

            People

              wzheng Wei Zheng
              deepesh Deepesh Khandelwal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: