Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14784

Operation logs are disabled automatically if the parent directory does not exist.

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 2.3.0
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      Operation logging is disabled automatically for the query if for some reason the parent directory (named after the hive session id) that gets created when the session is established gets deleted (for any reason). For ex: if the operation logdir is /tmp which automatically can get purged at a configured interval by the OS.

      Running a query from that session leads to

      2016-09-15 15:09:16,723 WARN org.apache.hive.service.cli.operation.Operation: Unable to create operation log file: /tmp/hive/operation_logs/b8809985-6b38-47ec-a49b-6158a67cd9fc/d35414f7-2418-426c-8489-c6f643ca4599
      java.io.IOException: No such file or directory
      	at java.io.UnixFileSystem.createFileExclusively(Native Method)
      	at java.io.File.createNewFile(File.java:1012)
      	at org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195)
      	at org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237)
      	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:255)
      	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398)
      	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385)
      	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271)
      	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490)
      	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
      	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
      	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
      	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      

      This later leads to errors like (more prominent when using HUE as HUE does not close hive sessions and attempts to retrieve the operations logs days after they were created).

      WARN org.apache.hive.service.cli.thrift.ThriftCLIService: Error fetching results: 
      org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=d35414f7-2418-426c-8489-c6f643ca4599]
      	at org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet(OperationManager.java:259)
      	at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:701)
      	at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:451)
      	at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:676)
      	at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
      	at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
      	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
      	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
      	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
      	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745) 
      

        Attachments

        1. HIVE-14784.1.patch
          1 kB
          Naveen Gangam
        2. HIVE-14784.patch
          1 kB
          Naveen Gangam

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              ngangam Naveen Gangam Assign to me
              Reporter:
              ngangam Naveen Gangam

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment