Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20643

hive job was hang when execute aggregate calculation such as max()、count()

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      if i run "select * from table limit 10" in the hive, it can return successful. but when i was run "select max() from table" ,it is hang.

      the output massage :-----------------------------------

      Logging initialized using configuration in file:/app/apache-hive-2.3.0-bin/conf/hive-log4j2.properties Async: true
      WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
      Query ID = root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
      Total jobs = 1
      Launching Job 1 out of 1
      Number of reduce tasks determined at compile time: 1
      In order to change the average load for a reducer (in bytes):
      set hive.exec.reducers.bytes.per.reducer=<number>
      In order to limit the maximum number of reducers:
      set hive.exec.reducers.max=<number>
      In order to set a constant number of reducers:
      set mapreduce.job.reduces=<number>
      Cannot run job locally: Input Size (= 562011785) is larger than hive.exec.mode.local.auto.inputbytes.max (= 256000000)。

       

      The hive.log :---------------------------------------------------------------------

      ========"new HiveConf()"========

      2018-09-27T08:55:46,192 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: lockmgr.DbLockManager (DbLockManager.java:lock(104)) - Response to queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 LockResponse(lockid:19039, state:ACQUIRED)
      2018-09-27T08:55:46,193 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: lockmgr.DbTxnManager (DbTxnManager.java:startHeartbeat(507)) - Started heartbeat with delay/interval = 150000/150000 MILLISECONDS for query: root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
      2018-09-27T08:55:46,193 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: ql.Driver (Driver.java:execute(1734)) - Executing command(queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1): select count from tick2
      2018-09-27T08:55:46,195 WARN [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: ql.Driver (Driver.java:logMrWarning(2094)) - Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
      2018-09-27T08:55:46,195 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: ql.Driver (SessionState.java:printInfo(1111)) - WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
      2018-09-27T08:55:46,197 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: ql.Driver (SessionState.java:printInfo(1111)) - Query ID = root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
      2018-09-27T08:55:46,197 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: ql.Driver (SessionState.java:printInfo(1111)) - Total jobs = 1
      2018-09-27T08:55:46,209 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: ql.Driver (SessionState.java:printInfo(1111)) - Launching Job 1 out of 1
      2018-09-27T08:55:46,211 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]: ql.Driver (Driver.java:launchTask(2174)) - Starting task [Stage-1:MAPRED] in parallel
      2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task (SessionState.java:printInfo(1111)) - Number of reduce tasks determined at compile time: 1
      2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task (SessionState.java:printInfo(1111)) - In order to change the average load for a reducer (in bytes):
      2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task (SessionState.java:printInfo(1111)) - set hive.exec.reducers.bytes.per.reducer=<number>
      2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task (SessionState.java:printInfo(1111)) - In order to limit the maximum number of reducers:
      2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task (SessionState.java:printInfo(1111)) - set hive.exec.reducers.max=<number>
      2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task (SessionState.java:printInfo(1111)) - In order to set a constant number of reducers:
      2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task (SessionState.java:printInfo(1111)) - set mapreduce.job.reduces=<number>
      2018-09-27T08:55:46,225 INFO [Thread-6]: exec.Task (SessionState.java:printInfo(1111)) - Cannot run job locally: Input Size (= 562011785) is larger than hive.exec.mode.local.auto.inputbytes.max (= 256000000)
      2018-09-27T08:55:46,231 INFO [Thread-6]: ql.Context (Context.java:getMRScratchDir(454)) - New scratch dir is hdfs://cluster/app/apache-hive-2.3.0-bin/tmp/hive-root/root/5cf99469-4f8e-49b1-9108-b94b3928a2df/hive_2018-09-27_08-55-41_865_3811548964141108768-2
      2018-09-27T08:55:46,237 INFO [Thread-6]: mr.ExecDriver (ExecDriver.java:execute(287)) - Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
      2018-09-27T08:55:46,241 INFO [Thread-6]: exec.Utilities (Utilities.java:getInputPaths(3023)) - Processing alias tick2
      2018-09-27T08:55:46,241 INFO [Thread-6]: exec.Utilities (Utilities.java:getInputPaths(3052)) - Adding 1 inputs; the first input is hdfs://cluster/usr/hive/warehouse/tick2
      2018-09-27T08:55:46,242 INFO [Thread-6]: exec.Utilities (Utilities.java:isEmptyPath(2332)) - Content Summary hdfs://cluster/usr/hive/warehouse/tick2length: 562011785 num files: 1 num directories: 1
      2018-09-27T08:55:46,242 INFO [Thread-6]: ql.Context (Context.java:getMRScratchDir(454)) - New scratch dir is hdfs://cluster/app/apache-hive-2.3.0-bin/tmp/hive-root/root/5cf99469-4f8e-49b1-9108-b94b3928a2df/hive_2018-09-27_08-55-41_865_3811548964141108768-2
      2018-09-27T08:55:46,331 INFO [Thread-6]: exec.SerializationUtilities (SerializationUtilities.java:serializePlan(450)) - Serializing MapWork using kryo
      2018-09-27T08:55:46,471 INFO [Thread-6]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1173)) - mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
      2018-09-27T08:55:46,476 INFO [Thread-6]: exec.Utilities (Utilities.java:setBaseWork(616)) - Serialized plan (via FILE) - name: null size: 6.48KB
      2018-09-27T08:55:46,479 INFO [Thread-6]: exec.SerializationUtilities (SerializationUtilities.java:serializePlan(450)) - Serializing ReduceWork using kryo
      2018-09-27T08:55:46,499 INFO [Thread-6]: exec.Utilities (Utilities.java:setBaseWork(616)) - Serialized plan (via FILE) - name: null size: 5.67KB
      2018-09-27T08:55:46,736 INFO [Thread-6]: exec.Utilities (Utilities.java:getBaseWork(407)) - PLAN PATH = hdfs://cluster/app/apache-hive-2.3.0-bin/tmp/hive-root/root/5cf99469-4f8e-49b1-9108-b94b3928a2df/hive_2018-09-27_08-55-41_865_3811548964141108768-2/-mr-10005/0be9de35-7cf8-4896-8496-964bbeebc9c8/map.xml
      2018-09-27T08:55:46,736 INFO [Thread-6]: exec.Utilities (Utilities.java:getBaseWork(407)) - PLAN PATH = hdfs://cluster/app/apache-hive-2.3.0-bin/tmp/hive-root/root/5cf99469-4f8e-49b1-9108-b94b3928a2df/hive_2018-09-27_08-55-41_865_3811548964141108768-2/-mr-10005/0be9de35-7cf8-4896-8496-964bbeebc9c8/reduce.xml
      2018-09-27T08:58:16,195 INFO [Heartbeater-0]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:00:46,194 INFO [Heartbeater-0]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:03:16,194 INFO [Heartbeater-1]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:05:46,193 INFO [Heartbeater-0]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:08:16,194 INFO [Heartbeater-2]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:10:46,198 INFO [Heartbeater-2]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:13:16,193 INFO [Heartbeater-2]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:15:46,194 INFO [Heartbeater-2]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:18:16,193 INFO [Heartbeater-2]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0
      2018-09-27T09:20:46,194 INFO [Heartbeater-2]: lockmgr.DbTxnManager (DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1 txnid:0

       

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vincentzhao vincentzhao
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: