Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13500

Launching big queries fails with Out of Memory Exception

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There is a code snipped in Driver class in compile method:

            if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) ||
                 conf.isWebUiQueryInfoCacheEnabled()) {
              String explainOutput = getExplainOutput(sem, plan, tree);
              if (explainOutput != null) {
                if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT)) {
                  LOG.info("EXPLAIN output for queryid " + queryId + " : "
                    + explainOutput);
                }
                if (conf.isWebUiQueryInfoCacheEnabled()) {
                  queryDisplay.setExplainPlan(explainOutput);
                }
              }
            }
      

      This is the stack trace

      org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.OutOfMemoryError
              at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:178) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:216) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.cli.operation.Operation.run(Operation.java:325) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:456) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:433) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_72]
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_72]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_72]
              at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_72]
              at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_72]
              at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_72]
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) ~[hadoop-common-2.6.0.2.2.4.2-2.jar:?]
              at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at com.sun.proxy.$Proxy33.executeStatementAsync(Unknown Source) ~[?:?]
              at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:272) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:554) [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317) [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302) [hive-service-rpc-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) [hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_72]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_72]
              at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
      Caused by: java.lang.OutOfMemoryError
              at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123) ~[?:1.8.0_72]
              at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117) ~[?:1.8.0_72]
              at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[?:1.8.0_72]
              at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[?:1.8.0_72]
              at java.io.PrintStream.write(PrintStream.java:480) ~[?:1.8.0_72]
              at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) ~[?:1.8.0_72]
              at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) ~[?:1.8.0_72]
              at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104) ~[?:1.8.0_72]
              at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185) ~[?:1.8.0_72]
              at java.io.PrintStream.write(PrintStream.java:527) ~[?:1.8.0_72]
              at java.io.PrintStream.print(PrintStream.java:683) ~[?:1.8.0_72]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputMap(ExplainTask.java:420) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:737) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:581) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:765) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:581) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputMap(ExplainTask.java:508) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:737) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:581) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:765) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:581) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:826) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.outputStagePlans(ExplainTask.java:970) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.exec.ExplainTask.getJSONPlan(ExplainTask.java:223) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.Driver.getExplainOutput(Driver.java:597) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:525) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:318) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1192) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1179) ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:144) ~[hive-service-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
              ... 27 more
              
      

      Now, in plain English, in the compile phase, hive is trying to generate explain output and set that output in the query display. Generating the explain output fails with OOM.

      I propose that setting of the explain output in query display be made optional, so that queries can still launch.

        Attachments

        1. HIVE-13500.patch
          0.7 kB
          Rajat Khandelwal

          Issue Links

            Activity

              People

              • Assignee:
                prongs Rajat Khandelwal
                Reporter:
                prongs Rajat Khandelwal
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: