Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23279

Unable to move source hdfs://xxx:8020/warehouse/.../.hive-staging_hive_... | 无法移动 hivestage 文件

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Blocker
    • Resolution: Unresolved
    • 2.1.1
    • None
    • HiveServer2

    Description

      error log from Cloudera Manager 6.2.1: | cdh 日志:

       

      [HiveServer2-Background-Pool: Thread-58]: Error running hive query:
      org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000 to destination hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01
      at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:329) ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:258) ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92) ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345) [hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
      at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_181]
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.2.1.jar:?]
      at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357) [hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000 to destination hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01
      at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3445) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3401) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:3396) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3693) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1614) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1525) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1489) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:501) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256) ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      ... 11 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000/000000_0 to destination hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/000000_0
      at org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:3418) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:3363) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3693) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1614) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1525) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1489) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:501) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1334) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256) ~[hive-service-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      ... 11 more
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000/000000_0 to destination hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/000000_0
      at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3445) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:3401) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive.access$200(Hive.java:175) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:3346) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:3331) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      ... 4 more
      Caused by: java.io.IOException: rename for src path: hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/.hive-staging_hive_2020-04-23_14-29-31_758_5380087337087305328-1/-ext-10000/000000_0 to dest path:hdfs://cdh231:8020/warehouse/szt.db/ads/ads_all_passengers_single_ride_spend_time_average/day=2018-09-01/000000_0 returned false
      at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:3343) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:3331) ~[hive-exec-2.1.1-cdh6.2.1.jar:2.1.1-cdh6.2.1]
      ... 4 more

       

      Additional notes:

      补充说明:

       

      My account for logging in to Cloudera Manager hue is hive, and the / warehouse directory has been authorized to hive, and I tried to set it up as follows: 

      我登录 Cloudera Manager  hue 的账户是 hive ,/warehouse 目录已授权给 hive ,并且尝试设置如下:

      ------------------------------------------------------------------------------------hive.warehouse.subdir.inherit.perms=false

      hive.exec.scratchdir=/warehouse/.hive-staging

      ------------------------------------------------------------------------------------

      or

      或者

      ------------------------------------------------------------------------------------hive.warehouse.subdir.inherit.perms=true

      ------------------------------------------------------------------------------------

      None of the above configurations can solve the problem

      I have been troubled by this problem for several weeks,
      I found that google and stackoverflow.com can't solve the problem

       

      以上配置全都无法解决问题
      我已经被这个问题困扰几周了,我查找了 google 和 stackoverflow.com 都无法解决问题

       

      My software version:

      我的软件版本:
      Cloudera Manager Agent 6.2.1Cloudera Manager Management Daemon 6.2.1Hadoop 3.0.0 cdh6.2.1HDFS 3.0.0 cdh6.2.1HttpFS 3.0.0 cdh6.2.1hadoop-kms 3.0.0 cdh6.2.1MapReduce 2 3.0.0 cdh6.2.1YARN 3.0.0 cdh6.2.1Hive 2.1.1 cdh6.2.1HCatalog 2.1.1 cdh6.2.1Hue 4.3.0 cdh6.2.1Impala 3.2.0 cdh6.2.1Java 8 java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)Parquet 1.9.0 cdh6.2.1spark 2.4.0 cdh6.2.1

       

       

      Attachments

        1. chrome_qy2E1vTR3j.png
          61 kB
          geekyouth
        2. chrome_k3C4YK0O5V.png
          129 kB
          geekyouth

        Activity

          People

            geekyouth geekyouth
            geekyouth geekyouth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: