Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16426

Query cancel: improve the way to handle files

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:
      None

      Description

      1. Add data structure support to make it is easy to check query cancel status.
      2. Handle query cancel more gracefully. Remove possible file leaks caused by query cancel as shown in following stack:

      2017-04-11 09:57:30,727 WARN  org.apache.hadoop.hive.ql.exec.Utilities: [HiveServer2-Background-Pool: Thread-149]: Failed to clean-up tmp directories.
      java.io.InterruptedIOException: Call interrupted
              at org.apache.hadoop.ipc.Client.call(Client.java:1496)
              at org.apache.hadoop.ipc.Client.call(Client.java:1439)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
              at com.sun.proxy.$Proxy20.delete(Unknown Source)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
              at com.sun.proxy.$Proxy21.delete(Unknown Source)
              at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2059)
              at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
              at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:671)
              at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
              at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:671)
              at org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:277)
              at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:463)
              at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:142)
              at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
              at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
              at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1978)
              at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1691)
              at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1423)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1207)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1202)
              at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
              at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
              at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:303)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
              at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:316)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

      3. Add checkpoints to related file operations to improve response time for query cancelling.

        Attachments

        1. HIVE-16426.1.patch
          8 kB
          Yongzhi Chen

          Issue Links

            Activity

              People

              • Assignee:
                ychena Yongzhi Chen
                Reporter:
                ychena Yongzhi Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: