Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20594

The staging directory should be appended with ".hive-staging" to avoid being deleted if we set hive.exec.stagingdir under the table directory without start with "."

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.1.0
    • Fix Version/s: 2.2.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      The staging directory should be appended with ".hive-staging" to avoid being deleted when we set hive.exec.stagingdir under the table directory without start with "."

      spark-sql> set hive.exec.stagingdir=./test;
      spark-sql> insert overwrite table test_table1 select * from test_table;

      we got error as follows:

      2017-05-04 15:21:06,948 INFO org.apache.hadoop.hive.common.FileUtils: deleting hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1
      2017-05-04 15:21:06,987 INFO org.apache.hadoop.fs.TrashPolicyDefault: Moved: 'hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1' to trash at: hdfs://nameservice/user/mr/.Trash/Current/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1
      2017-05-04 15:21:06,987 INFO org.apache.hadoop.hive.common.FileUtils: Moved to trash: hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1
      2017-05-04 15:21:07,001 ERROR org.apache.hadoop.hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
      2017-05-04 15:21:07,007 INFO hive.ql.metadata.Hive: Replacing src:hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1/-ext-10000/part-00000, dest: hdfs://nameservice/spark/ztb.db/test_table1/part-00000, Status:false
      2017-05-04 15:21:07,024 ERROR org.apache.spark.sql.hive.thriftserver.SparkSQLDriver: Failed in [insert overwrite table test_table1 select * from test_table]
      java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:633)
      at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:646)
      at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:646)
      at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:646)
      at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:280)
      at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:227)
      at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:226)
      at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:269)
      at org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:645)
      at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:290)
      at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:143)
      at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.executeCollect(InsertIntoHiveTable.scala:308)
      at org.apache.spark.sql.Dataset.<init>(Dataset.scala:186)
      at org.apache.spark.sql.Dataset.<init>(Dataset.scala:167)
      at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:65)
      at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:601)
      at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682)
      at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
      at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:331)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
      at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:247)
      at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move file/directory from hdfs://nameservice/spark/ztb.db/test_table1/test_hive_2017-05-04_15-21-05_972_7582740597864081934-1/-ext-10000/part-00000 to hdfs://nameservice/spark/ztb.db/test_table1/part-00000
      at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2899)
      at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1589)

        Attachments

          Activity

            People

            • Assignee:
              zuo.tingbing9 zuotingbing
              Reporter:
              zuo.tingbing9 zuotingbing
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: