Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7623

hive partition rename fails if filesystem cache is disabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 0.13.1
    • Fix Version/s: 0.14.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      Seems to be similar issue https://issues.apache.org/jira/browse/HIVE-3815 when calling alterPartition (when renaming partitions)

      Setting fs.hdfs.impl.disable.cache=false and fs.file.impl.disable.cache=falseworks around this problem

      Error:
      =====
      2014-08-05 21:46:14,522 ERROR [pool-3-thread-1]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(143)) - InvalidOperationException(message:table new location hdfs://hadoop-namenode:8020/user/hive/warehouse/sample_logs/XX=AA/YY=123 is on a different file system than the old location hdfs://hadoop-namenode:8020/user/hive/warehouse/sample_logs/XX=AA/YY=456. This operation is not supported)
      at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartition(HiveAlterHandler.java:361)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:2629)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:2602)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:622)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
      at com.sun.proxy.$Proxy5.rename_partition(Unknown Source)
      at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$rename_partition.getResult(ThriftHiveMetastore.java:9057)
      at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$rename_partition.getResult(ThriftHiveMetastore.java:9041)
      at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
      at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:416)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)

      Looking at the code apache-hive-0.13.1-src/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java on line 361 see that its using != to compare filesystem objects

      // check that src and dest are on the same file system
      if (srcFs != destFs)

      { throw new InvalidOperationException("table new location " + destPath + " is on a different file system than the old location " + srcPath + ". This operation is not supported"); }

        Attachments

        1. HIVE-7623.1.patch.txt
          0.9 kB
          Navis Ryu

          Issue Links

            Activity

              People

              • Assignee:
                navis Navis Ryu
                Reporter:
                agateaaa Ashwin Agate
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: