Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: HiveServer2
    • Labels:

      Description

      HiveServer2 should support an API to retrieve query logs. This is particularly relevant because HiveServer2 supports async execution but doesn't provide a way to report progress. Providing an API to retrieve query logs will help report progress to the client.

      1. HIVE-4629.1.patch
        214 kB
        Shreepadma Venugopalan
      2. HIVE-4629.2.patch
        214 kB
        Shreepadma Venugopalan
      3. HIVE-4629.3.patch.txt
        137 kB
        Navis
      4. HIVE-4629.4.patch
        83 kB
        Dong Chen
      5. HIVE-4629.5.patch
        89 kB
        Dong Chen
      6. HIVE-4629.6.patch
        89 kB
        Dong Chen
      7. HIVE-4629.7.patch
        85 kB
        Thejas M Nair
      8. HIVE-4629.8.patch
        86 kB
        Dong Chen
      9. HIVE-4629.9.patch
        82 kB
        Brock Noland
      10. HIVE-4629-no_thrift.1.patch
        41 kB
        Shreepadma Venugopalan

        Issue Links

          Activity

          Hide
          Carl Steinbach added a comment -

          @Shreepadma: before making any code changes I think it would be a good idea to get feedback on the changes you plan to make to TCLIService and CLIService.

          Show
          Carl Steinbach added a comment - @Shreepadma: before making any code changes I think it would be a good idea to get feedback on the changes you plan to make to TCLIService and CLIService.
          Hide
          Shreepadma Venugopalan added a comment -

          @Carl: The proposed addition to TCLIService.thrift is the following new API and structs,

          // GetLog()
          // Fetch operation log from the server corresponding to
          // a particular OperationHandle.
          
          struct TGetLogReq {
            // Operation whose log is requested
            1: required TOperationHandle operationHandle
          }
          
          struct TGetLogResp {
            1: required TStatus status
            2: required string log
          }
          
          service TCLIService {
          ...
          ...
          TGetLogResp GetLog(1:TGetLogReq req);
          }
          
          Show
          Shreepadma Venugopalan added a comment - @Carl: The proposed addition to TCLIService.thrift is the following new API and structs, // GetLog() // Fetch operation log from the server corresponding to // a particular OperationHandle. struct TGetLogReq { // Operation whose log is requested 1: required TOperationHandle operationHandle } struct TGetLogResp { 1: required TStatus status 2: required string log } service TCLIService { ... ... TGetLogResp GetLog(1:TGetLogReq req); }
          Hide
          Shreepadma Venugopalan added a comment -

          Carl Steinbach: Can you look at this? Thanks!

          Show
          Shreepadma Venugopalan added a comment - Carl Steinbach : Can you look at this? Thanks!
          Hide
          Carl Steinbach added a comment -

          Here are some questions I had while looking at the API:

          • When am I allowed to call GetLog()? For example, what happens if I call it after calling CloseOperation or CancelOperation? If it's ok to call after CloseOperation()/CancelOperation(), what does it return, and what does it return if I first call a function that also returns an operationhandle?
          • What happens if I call GetLog() twice during the same operation? Will it always return the entire log collected up to that point or just a piece of it?
          • Do you think we log (or should log) anything of interest between the OpenSession() call and creation of the first OperationHandle, or between CloseSession() and the previous call to CancelOperation()/CloseOperation()?

          I also thought of a possible alternate approach and wanted to get your opinion on it:

          • Provide access to logs on a per-session basis instead of per-operation.
          • Provide a LogHandle (either hanging off of the SessionHandle or provided via GetLogHandle()) and make it fetchable with FetchResults()
          • We can provide per-operation log retrieval by supporting scrollable log cursors, e.g. scrolling to FETCH_LAST between operations will advance the log stream to the start of the next operation's log.
          Show
          Carl Steinbach added a comment - Here are some questions I had while looking at the API: When am I allowed to call GetLog()? For example, what happens if I call it after calling CloseOperation or CancelOperation? If it's ok to call after CloseOperation()/CancelOperation(), what does it return, and what does it return if I first call a function that also returns an operationhandle? What happens if I call GetLog() twice during the same operation? Will it always return the entire log collected up to that point or just a piece of it? Do you think we log (or should log) anything of interest between the OpenSession() call and creation of the first OperationHandle, or between CloseSession() and the previous call to CancelOperation()/CloseOperation()? I also thought of a possible alternate approach and wanted to get your opinion on it: Provide access to logs on a per-session basis instead of per-operation. Provide a LogHandle (either hanging off of the SessionHandle or provided via GetLogHandle()) and make it fetchable with FetchResults() We can provide per-operation log retrieval by supporting scrollable log cursors, e.g. scrolling to FETCH_LAST between operations will advance the log stream to the start of the next operation's log.
          Hide
          Shreepadma Venugopalan added a comment -

          This API gets the operation log aka the per query log. Clients are mostly interested in the operation/per query log rather than the session log. Furthermore, session log will contain log output for a given query interspersed with the log output of other queries that may be executing concurrently in the same session. Note that executing multiple queries in a session is possible with async execution. The main purpose of this JIRA is to support an API to retrieve per query log. If there's enough interest, adding an API to support retrieving session log can be done in a follow on JIRA.

          Show
          Shreepadma Venugopalan added a comment - This API gets the operation log aka the per query log. Clients are mostly interested in the operation/per query log rather than the session log. Furthermore, session log will contain log output for a given query interspersed with the log output of other queries that may be executing concurrently in the same session. Note that executing multiple queries in a session is possible with async execution. The main purpose of this JIRA is to support an API to retrieve per query log. If there's enough interest, adding an API to support retrieving session log can be done in a follow on JIRA.
          Hide
          Shreepadma Venugopalan added a comment -
          • Operation logs don't persist beyond the life of an operation. If getLog(opHandle) is called after a CloseOperation() or CancelOperation(), an exception is raised.
          • If getLog() is called on the same operationHandle twice, the log up to the point will be returned. Note that the log is maintained as a circular buffer and loops back. The default size is 128 KB, but is configurable. We have not come across a case where 128 KB was insufficient.
          Show
          Shreepadma Venugopalan added a comment - Operation logs don't persist beyond the life of an operation. If getLog(opHandle) is called after a CloseOperation() or CancelOperation(), an exception is raised. If getLog() is called on the same operationHandle twice, the log up to the point will be returned. Note that the log is maintained as a circular buffer and loops back. The default size is 128 KB, but is configurable. We have not come across a case where 128 KB was insufficient.
          Hide
          Shreepadma Venugopalan added a comment -
          Show
          Shreepadma Venugopalan added a comment - Review board: https://reviews.apache.org/r/14326/
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12604928/HIVE-4629.1.patch

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/886/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/886/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-886/source-prep.txt
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java'
          ++ egrep -v '^X|^Performing status on external'
          ++ awk '{print $2}'
          ++ svn status --no-ignore
          + rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen ql/src/test/results/clientpositive/cast_to_int.q.out ql/src/test/queries/clientpositive/cast_to_int.q
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1526135.
          
          At revision 1526135.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0 to p2
          + exit 1
          '
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12604928/HIVE-4629.1.patch Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/886/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/886/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-886/source-prep.txt + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen ql/src/test/results/clientpositive/cast_to_int.q.out ql/src/test/queries/clientpositive/cast_to_int.q + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1526135. At revision 1526135. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0 to p2 + exit 1 ' This message is automatically generated.
          Hide
          Shreepadma Venugopalan added a comment -

          I'm able to apply the patch with -p0 to the tip of trunk. I've re-attached the patch to trigger a run.

          Show
          Shreepadma Venugopalan added a comment - I'm able to apply the patch with -p0 to the tip of trunk. I've re-attached the patch to trigger a run.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12605136/HIVE-4629.2.patch

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/907/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/907/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-907/source-prep.txt
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'build-common.xml'
          Reverted 'ql/src/test/queries/clientnegative/invalid_columns.q'
          Reverted 'ql/src/test/queries/clientnegative/deletejar.q'
          Reverted 'ql/src/test/queries/clientpositive/input16_cc.q'
          Reverted 'ql/src/test/queries/clientpositive/alter1.q'
          Reverted 'ql/src/test/queries/clientpositive/input16.q'
          ++ awk '{print $2}'
          ++ egrep -v '^X|^Performing status on external'
          ++ svn status --no-ignore
          + rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1526404.
          
          At revision 1526404.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0 to p2
          + exit 1
          '
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12605136/HIVE-4629.2.patch Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/907/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/907/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-907/source-prep.txt + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'build-common.xml' Reverted 'ql/src/test/queries/clientnegative/invalid_columns.q' Reverted 'ql/src/test/queries/clientnegative/deletejar.q' Reverted 'ql/src/test/queries/clientpositive/input16_cc.q' Reverted 'ql/src/test/queries/clientpositive/alter1.q' Reverted 'ql/src/test/queries/clientpositive/input16.q' ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1526404. At revision 1526404. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0 to p2 + exit 1 ' This message is automatically generated.
          Hide
          Brock Noland added a comment -

          I think it makes sense to do a scrollable log in a future jira. Carl do you have any more concerns?

          Show
          Brock Noland added a comment - I think it makes sense to do a scrollable log in a future jira. Carl do you have any more concerns?
          Hide
          Carl Steinbach added a comment -

          Brock Noland Making the logs scrollable wasn't the point of my suggestion. I'm more concerned about leveraging the existing fetch functions and patterns to satisfy a use case which fundamentally looks very similar to fetching a query result set. CLIService is a public interface. We're permanently stuck with any changes that are made to it. I'd like to avoid cluttering it with a mishmash of logging RPCs if it's possible to reduce all of these use cases to a single pattern.

          In the future I hope we can finish discussing changes like this before the first patch is posted.

          Shreepadma Venugopalan I left some comments on reviewboard. Thanks.

          Show
          Carl Steinbach added a comment - Brock Noland Making the logs scrollable wasn't the point of my suggestion. I'm more concerned about leveraging the existing fetch functions and patterns to satisfy a use case which fundamentally looks very similar to fetching a query result set. CLIService is a public interface. We're permanently stuck with any changes that are made to it. I'd like to avoid cluttering it with a mishmash of logging RPCs if it's possible to reduce all of these use cases to a single pattern. In the future I hope we can finish discussing changes like this before the first patch is posted. Shreepadma Venugopalan I left some comments on reviewboard. Thanks.
          Hide
          Brock Noland added a comment -

          Carl, I see, as you know I took your comments as a suggestion for a scrollable log. I apologize for mis-understanding your comments.

          Show
          Brock Noland added a comment - Carl, I see, as you know I took your comments as a suggestion for a scrollable log. I apologize for mis-understanding your comments.
          Hide
          Prasad Mujumdar added a comment -

          Shreepadma Venugopalan I guess the patch is out of sync and won't apply. There are a bunch of changes to HS2 interface, the build framework changed from ant to maven etc. Would you mind rebasing it to the latest and update the review.

          If you want I can take care of the rebase.

          Show
          Prasad Mujumdar added a comment - Shreepadma Venugopalan I guess the patch is out of sync and won't apply. There are a bunch of changes to HS2 interface, the build framework changed from ant to maven etc. Would you mind rebasing it to the latest and update the review. If you want I can take care of the rebase.
          Hide
          Gordon Wang added a comment -

          What about the status of this jira?
          Does anyone try to rebase it to the latest trunk?
          I think it is a useful feature especially when doing some testing about hql.

          Show
          Gordon Wang added a comment - What about the status of this jira? Does anyone try to rebase it to the latest trunk? I think it is a useful feature especially when doing some testing about hql.
          Hide
          Carl Steinbach added a comment -

          Does the new version of the patch address any of the API design issues I mentioned earlier?

          Show
          Carl Steinbach added a comment - Does the new version of the patch address any of the API design issues I mentioned earlier?
          Hide
          Suhas Satish added a comment -

          It would be great to have this jira accepted into hive trunk, even I am waiting on this from a long time.

          Show
          Suhas Satish added a comment - It would be great to have this jira accepted into hive trunk, even I am waiting on this from a long time.
          Hide
          Ashu Pachauri added a comment -

          Any estimate on when this will be accepted into trunk?

          Show
          Ashu Pachauri added a comment - Any estimate on when this will be accepted into trunk?
          Hide
          Ravi Prakash added a comment -

          This patch no longer applies on HIVE trunk. Is there a plan to upmerge it? And incorporate Carl's comments?

          Show
          Ravi Prakash added a comment - This patch no longer applies on HIVE trunk. Is there a plan to upmerge it? And incorporate Carl's comments?
          Hide
          Navis added a comment -

          Just rebased to trunk. Let this get in.

          Show
          Navis added a comment - Just rebased to trunk. Let this get in.
          Hide
          Brock Noland added a comment -

          Agreed, let's get this in.

          Show
          Brock Noland added a comment - Agreed, let's get this in.
          Hide
          Ravi Prakash added a comment -

          Thanks a lot Navis for the updated patch!

          @Brock ++ Are there any plans to incorporate Carl's comments and provide an updated patch?

          @Carl: We need this functionality in Hive to continue providing Hue to our users. Hue is a pretty useful application, so we would like to continue providing it. Would you please consider accepting the patch?

          Show
          Ravi Prakash added a comment - Thanks a lot Navis for the updated patch! @Brock ++ Are there any plans to incorporate Carl's comments and provide an updated patch? @Carl: We need this functionality in Hive to continue providing Hue to our users. Hue is a pretty useful application, so we would like to continue providing it. Would you please consider accepting the patch?
          Hide
          Brock Noland added a comment -

          Hi Ravi and Carl,

          The current patch does not implement the changes requested by Carl. I do think the changes described by Carl would make the API more flexible. However, the current design does work well and thus I feel we should move forward with the current design.

          Brock

          Show
          Brock Noland added a comment - Hi Ravi and Carl, The current patch does not implement the changes requested by Carl. I do think the changes described by Carl would make the API more flexible. However, the current design does work well and thus I feel we should move forward with the current design. Brock
          Hide
          Thejas M Nair added a comment -

          Please give me a day to review this patch.
          I had an initial pass on this, one concern I have is the 1 MB of log being returned on every call to getLog. I believe this is for use cases such as Hue. In such use cases there is only one consumer for the log, and giving out incremental logs would be better. ie , after a call to getLog, clear the buffer and the next getLog call returns only the new log lines. This will reduce the memory footprint in HiveServer2 and the load that getLog call puts on it. It will also make applications like Hue more responsive, and be able to refresh the log more frequently.

          Show
          Thejas M Nair added a comment - Please give me a day to review this patch. I had an initial pass on this, one concern I have is the 1 MB of log being returned on every call to getLog. I believe this is for use cases such as Hue. In such use cases there is only one consumer for the log, and giving out incremental logs would be better. ie , after a call to getLog, clear the buffer and the next getLog call returns only the new log lines. This will reduce the memory footprint in HiveServer2 and the load that getLog call puts on it. It will also make applications like Hue more responsive, and be able to refresh the log more frequently.
          Hide
          Carl Steinbach added a comment -

          Does anyone think this is the right way to implement this feature?

          Show
          Carl Steinbach added a comment - Does anyone think this is the right way to implement this feature?
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12647299/HIVE-4629.3.patch.txt

          ERROR: -1 due to 7 failed/errored test(s), 5468 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/343/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/343/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-343/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 7 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12647299

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12647299/HIVE-4629.3.patch.txt ERROR: -1 due to 7 failed/errored test(s), 5468 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/343/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/343/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-343/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12647299
          Hide
          Thejas M Nair added a comment -

          I have added some comments on review board .

          I think this feature is very useful. A lot of other people have asked for it in the jira as well.
          But I think we should address the review board comments and comments in this jira first.

          We should first agree on the public api before committing the patch. The suggestion from Carl about using the FetchResults api for this makes sense. As the use cases for this evolves, I think it the getLog api is likely to start looking more like the FetchResults api.
          The TFetchResultsReq can have an additional resulttype parameter (OUTPUT vs LOG).

          Also, the circular in-memory buffer does not gurantee that you get the whole log, it also puts burden on the HS2 memory requirements. I think we should explore using the approach in HIVE-5924 for this instead of an inmemory pool. But I am OK with having such changes as part of follow up jiras, as long as we don't have the in-memory logging enabled by default in this patch.

          Show
          Thejas M Nair added a comment - I have added some comments on review board . I think this feature is very useful. A lot of other people have asked for it in the jira as well. But I think we should address the review board comments and comments in this jira first. We should first agree on the public api before committing the patch. The suggestion from Carl about using the FetchResults api for this makes sense. As the use cases for this evolves, I think it the getLog api is likely to start looking more like the FetchResults api. The TFetchResultsReq can have an additional resulttype parameter (OUTPUT vs LOG). Also, the circular in-memory buffer does not gurantee that you get the whole log, it also puts burden on the HS2 memory requirements. I think we should explore using the approach in HIVE-5924 for this instead of an inmemory pool. But I am OK with having such changes as part of follow up jiras, as long as we don't have the in-memory logging enabled by default in this patch.
          Hide
          Navis added a comment -

          I'm not intended to take this job(I'm not good at interface design) but just rebased the patch. Will be really appreciated if anyone take this work, which will be huge contribution to hive community.

          Carl Steinbach Any idea on how to retrieve operation logs via currently existing APIs in ICliService? Should we extend select statement for that? (select _hive_operation_log from operation-handler, for example)

          Show
          Navis added a comment - I'm not intended to take this job(I'm not good at interface design) but just rebased the patch. Will be really appreciated if anyone take this work, which will be huge contribution to hive community. Carl Steinbach Any idea on how to retrieve operation logs via currently existing APIs in ICliService? Should we extend select statement for that? (select _hive_operation_log from operation-handler, for example)
          Hide
          Ravi Prakash added a comment -

          Is this something complicated? Can a hive beginner take up this task?

          Show
          Ravi Prakash added a comment - Is this something complicated? Can a hive beginner take up this task?
          Hide
          Vaibhav Gumashta added a comment -

          Ravi Prakash This can definitely be taken up. There is some good feedback on the jira too regarding the modifications. Let me know if you plan to take it up and need more help while you work on it. Thanks a lot!

          Show
          Vaibhav Gumashta added a comment - Ravi Prakash This can definitely be taken up. There is some good feedback on the jira too regarding the modifications. Let me know if you plan to take it up and need more help while you work on it. Thanks a lot!
          Hide
          Ravi Prakash added a comment -

          Hi Vaibhav! Sorry for the delay. I wish I had enough cycles to take this up, but that's just not happening.

          Show
          Ravi Prakash added a comment - Hi Vaibhav! Sorry for the delay. I wish I had enough cycles to take this up, but that's just not happening.
          Hide
          Dong Chen added a comment -

          A patch HIVE-4629.4.patch is attached, which trying to address the comments in this jira and review board, to support an API to retrieve query logs.

          This patch is developed based on HIVE-4629.3.patch code in this jira and refer to the method of saving operation log to files in HIVE-5924.

          RB entry: https://reviews.apache.org/r/24293/

          The design and changes are as follows:
          1. Reuse FetchResults() API and add a TFetchType in the param. The type could be QUERY_OUTPUT and LOG.
          2. Original LogManager code is modified and moved in OperationManager. Now OM is responsible for dispatching log updating and fetching request.
          3. OperatinLog class wraps the actual operation log files. Each Operation create, register, and unregister the OperationLog around its run() method.
          4. Make OperationLog to ThreadLocal, so that we don't have to bind and maintain them with threads in some Maps.
          5. The actual operation log files are maintained in the hierarchy like:
          rootDir/sessionDir/operationLogFiles
          The rootDir is configurable, created when SessionManager init, deleted when stop.
          The sessionDir is created when session is opened, and deleted during closing.
          The operationLogFiles is create before operation runs, and deleted when it is closed.

          Show
          Dong Chen added a comment - A patch HIVE-4629 .4.patch is attached, which trying to address the comments in this jira and review board, to support an API to retrieve query logs. This patch is developed based on HIVE-4629 .3.patch code in this jira and refer to the method of saving operation log to files in HIVE-5924 . RB entry: https://reviews.apache.org/r/24293/ The design and changes are as follows: 1. Reuse FetchResults() API and add a TFetchType in the param. The type could be QUERY_OUTPUT and LOG. 2. Original LogManager code is modified and moved in OperationManager. Now OM is responsible for dispatching log updating and fetching request. 3. OperatinLog class wraps the actual operation log files. Each Operation create, register, and unregister the OperationLog around its run() method. 4. Make OperationLog to ThreadLocal, so that we don't have to bind and maintain them with threads in some Maps. 5. The actual operation log files are maintained in the hierarchy like: rootDir/sessionDir/operationLogFiles The rootDir is configurable, created when SessionManager init, deleted when stop. The sessionDir is created when session is opened, and deleted during closing. The operationLogFiles is create before operation runs, and deleted when it is closed.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12659815/HIVE-4629.4.patch

          ERROR: -1 due to 5 failed/errored test(s), 5854 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_mixed_case
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode
          org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/178/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/178/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-178/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12659815

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12659815/HIVE-4629.4.patch ERROR: -1 due to 5 failed/errored test(s), 5854 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_mixed_case org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/178/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/178/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-178/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12659815
          Hide
          Dong Chen added a comment -

          Update the patch (v6) to trigger test. It addresses review comments and fixes one failed case related with this patch in HIVE QA.

          Show
          Dong Chen added a comment - Update the patch (v6) to trigger test. It addresses review comments and fixes one failed case related with this patch in HIVE QA.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12660420/HIVE-4629.6.patch

          ERROR: -1 due to 6 failed/errored test(s), 5875 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/221/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/221/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-221/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12660420

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12660420/HIVE-4629.6.patch ERROR: -1 due to 6 failed/errored test(s), 5875 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/221/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/221/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-221/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12660420
          Hide
          Brock Noland added a comment -

          Nice work Dong Chen!!

          Thejas M Nair Carl Steinbach you two had some good feedback on the earlier design. Can you take a look at the latest patch? Romain Rigaux, I know Hue uses this API, do you want to take a look?

          Show
          Brock Noland added a comment - Nice work Dong Chen !! Thejas M Nair Carl Steinbach you two had some good feedback on the earlier design. Can you take a look at the latest patch? Romain Rigaux , I know Hue uses this API, do you want to take a look?
          Hide
          Romain Rigaux added a comment -

          I mostly looked at the TCLIService.thrift. The new API looks good.

          Is FETCH_FIRST also implemented for the TFetchOrientation? (would still be useful in case a user refresh the page or wants all the logs again)

          Show
          Romain Rigaux added a comment - I mostly looked at the TCLIService.thrift. The new API looks good. Is FETCH_FIRST also implemented for the TFetchOrientation? (would still be useful in case a user refresh the page or wants all the logs again)
          Hide
          Dong Chen added a comment -

          Hi Romain Rigaux, thanks for your review. Yes, the FETCH_FIRST is implemented.

          Show
          Dong Chen added a comment - Hi Romain Rigaux , thanks for your review. Yes, the FETCH_FIRST is implemented.
          Hide
          Thejas M Nair added a comment -

          Dong Chen Thanks for the patch! I will review it in a few days.

          Show
          Thejas M Nair added a comment - Dong Chen Thanks for the patch! I will review it in a few days.
          Hide
          Brock Noland added a comment - - edited

          Hi Dong,

          I tried posting this on RB but it went down. Thank you very much for removing the thrift enum compatibility problem! I had another comment with regards to the method signature which I think I did not explain well. I think the new method should be:

          FetchResultsResponse fetchResults(FetchResultsRequest) throws ...
          

          The problem with how we've defined RPC methods to date has led to an explosion of RPC methods which is problematic. This is described in more detail in this thread.

          Let me know what you think!!

          Cheers,
          Brock

          Show
          Brock Noland added a comment - - edited Hi Dong, I tried posting this on RB but it went down. Thank you very much for removing the thrift enum compatibility problem! I had another comment with regards to the method signature which I think I did not explain well. I think the new method should be: FetchResultsResponse fetchResults(FetchResultsRequest) throws ... The problem with how we've defined RPC methods to date has led to an explosion of RPC methods which is problematic. This is described in more detail in this thread . Let me know what you think!! Cheers, Brock
          Hide
          Thejas M Nair added a comment -

          Dong Chen Thanks for the clean and well documented code! I have added some minor comments in review board.

          Show
          Thejas M Nair added a comment - Dong Chen Thanks for the clean and well documented code! I have added some minor comments in review board.
          Hide
          Thejas M Nair added a comment -

          Dong Chen Earlier patch also had a method in HiveStatement to get the log. I think that will be convenient for many users, though we need to be careful and specify that is the only non jdbc function that is part of a public API in it. But this can also be done as follow up work in separate jira.

          Show
          Thejas M Nair added a comment - Dong Chen Earlier patch also had a method in HiveStatement to get the log. I think that will be convenient for many users, though we need to be careful and specify that is the only non jdbc function that is part of a public API in it. But this can also be done as follow up work in separate jira.
          Hide
          Thejas M Nair added a comment -

          HIVE-4629.7.patch - Dong's patch from reviewboard, just to know how it is doing wrt tests.

          Dong Chen Looking forward to updated patch from you!

          Show
          Thejas M Nair added a comment - HIVE-4629 .7.patch - Dong's patch from reviewboard, just to know how it is doing wrt tests. Dong Chen Looking forward to updated patch from you!
          Hide
          Carl Steinbach added a comment -

          I tried posting this on RB but it went down. Thank you very much for removing the thrift enum compatibility problem! I had another comment with regards to the method signature which I think I did not explain well. I think the new method should be...

          Brock Noland, I totally agree with this, but I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else?

          Show
          Carl Steinbach added a comment - I tried posting this on RB but it went down. Thank you very much for removing the thrift enum compatibility problem! I had another comment with regards to the method signature which I think I did not explain well. I think the new method should be... Brock Noland , I totally agree with this, but I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else?
          Hide
          Brock Noland added a comment -

          I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else?

          You are right...I was confused between ICLIService and TCLIService. The patch only adds a new option member to TFetchResultsReq which is exactly what I wanted.

          Nevermind me

          Show
          Brock Noland added a comment - I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else? You are right...I was confused between ICLIService and TCLIService. The patch only adds a new option member to TFetchResultsReq which is exactly what I wanted. Nevermind me
          Hide
          Dong Chen added a comment -

          Hi, Thejas M Nair, Brock Noland, Carl Steinbach, thanks for your precious comments. I am working on the updated patch to address these comments, and will let you know when the updated patch is uploaded.

          Earlier patch also had a method in HiveStatement to get the log. I think that will be convenient for many users, though we need to be careful and specify that is the only non jdbc function that is part of a public API in it. But this can also be done as follow up work in separate jira.

          Thejas M Nair, you are right, adding a method in HiveStatement to get the log is convenient for users. I filed a Jira HIVE-7615 and planed to add the getting log API in jdbc level there.

          I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else?

          Carl Steinbach, Brock Noland, the latest patch still does not fulfill the comments about backward compatibility. I will update the patch soon and let you know.
          The Thrift level interface TCLIService is OK.
          For client and service layer interface ICLIService, although it is not RPC and is not a public API of Hive, I think making it follow the single request/response struct mode is also good. Will make the new fetchResults method follow the single request/response struct model. Then remove those old fetchResults methods.

          Show
          Dong Chen added a comment - Hi, Thejas M Nair , Brock Noland , Carl Steinbach , thanks for your precious comments. I am working on the updated patch to address these comments, and will let you know when the updated patch is uploaded. Earlier patch also had a method in HiveStatement to get the log. I think that will be convenient for many users, though we need to be careful and specify that is the only non jdbc function that is part of a public API in it. But this can also be done as follow up work in separate jira. Thejas M Nair , you are right, adding a method in HiveStatement to get the log is convenient for users. I filed a Jira HIVE-7615 and planed to add the getting log API in jdbc level there. I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else? Carl Steinbach , Brock Noland , the latest patch still does not fulfill the comments about backward compatibility. I will update the patch soon and let you know. The Thrift level interface TCLIService is OK. For client and service layer interface ICLIService, although it is not RPC and is not a public API of Hive, I think making it follow the single request/response struct mode is also good. Will make the new fetchResults method follow the single request/response struct model. Then remove those old fetchResults methods.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12663483/HIVE-4629.7.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/455/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/455/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-455/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
          + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-455/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriterV2.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/SerializationUtils.java'
          ++ egrep -v '^X|^Performing status on external'
          ++ awk '{print $2}'
          ++ svn status --no-ignore
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target accumulo-handler/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1619731.
          
          At revision 1619731.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12663483

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12663483/HIVE-4629.7.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/455/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/455/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-455/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-455/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriterV2.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/SerializationUtils.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target accumulo-handler/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1619731. At revision 1619731. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12663483
          Hide
          Brock Noland added a comment -

          Hi Dong,

          The latest patch fails to apply to HEAD and will need to be rebased.

          the latest patch still does not fulfill the comments about backward compatibility.

          I am very sorry for the confusion, I believe the patch does meet the backward compatibility requirement.

          For client and service layer interface ICLIService, although it is not RPC and is not a public API of Hive, I think making it follow the single request/response struct mode is also good. Will make the new fetchResults method follow the single request/response struct model. Then remove those old fetchResults methods.

          I do not feel this is required. The current patch works exactly as we requested!

          Thank you very much Dong!!

          Show
          Brock Noland added a comment - Hi Dong, The latest patch fails to apply to HEAD and will need to be rebased. the latest patch still does not fulfill the comments about backward compatibility. I am very sorry for the confusion, I believe the patch does meet the backward compatibility requirement. For client and service layer interface ICLIService, although it is not RPC and is not a public API of Hive, I think making it follow the single request/response struct mode is also good. Will make the new fetchResults method follow the single request/response struct model. Then remove those old fetchResults methods. I do not feel this is required. The current patch works exactly as we requested! Thank you very much Dong!!
          Hide
          Dong Chen added a comment -

          Update patch V8 to address the precious comments from Thejas M Nair, Carl Steinbach, and Brock Noland. Thanks very much for your review to make this patch better.

          And rebase it from latest trunk.

          Show
          Dong Chen added a comment - Update patch V8 to address the precious comments from Thejas M Nair , Carl Steinbach , and Brock Noland . Thanks very much for your review to make this patch better. And rebase it from latest trunk.
          Hide
          Brock Noland added a comment -

          Nice work Dong! From my side this looks great and I think we should commit.

          +1 pending tests

          Thejas or Carl, please let me know if you'd like to see additional work before commit.

          Show
          Brock Noland added a comment - Nice work Dong! From my side this looks great and I think we should commit. +1 pending tests Thejas or Carl, please let me know if you'd like to see additional work before commit.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12664360/HIVE-4629.8.patch

          ERROR: -1 due to 1 failed/errored test(s), 6119 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/505/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/505/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-505/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12664360

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12664360/HIVE-4629.8.patch ERROR: -1 due to 1 failed/errored test(s), 6119 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/505/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/505/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-505/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12664360
          Hide
          Thejas M Nair added a comment -

          +1 .
          Thanks Dong, great work!
          Looking forward to your contributions in HIVE-7615, it would be great to have that in hive 0.14 release !

          Show
          Thejas M Nair added a comment - +1 . Thanks Dong, great work! Looking forward to your contributions in HIVE-7615 , it would be great to have that in hive 0.14 release !
          Hide
          Brock Noland added a comment -

          I have committed this to trunk! Thank you Dong! This is truly a long awaited contribution!

          FYI - I had to do one trivial rebase in SessionManager when committing so I've uploaded that patch as v9. I verified it does not break the build.

          Show
          Brock Noland added a comment - I have committed this to trunk! Thank you Dong! This is truly a long awaited contribution! FYI - I had to do one trivial rebase in SessionManager when committing so I've uploaded that patch as v9. I verified it does not break the build.
          Hide
          Lefty Leverenz added a comment -

          Does this need to be documented in the wiki?

          Of course, two new parameters need to go in Configuration Properties (hive.server2.logging.operation.enabled, hive.server2.logging.operation.log.location), but should general guidance be given in one of the HS2 wikidocs? The logging section in Getting Started could also mention this.

          Show
          Lefty Leverenz added a comment - Does this need to be documented in the wiki? Of course, two new parameters need to go in Configuration Properties ( hive.server2.logging.operation.enabled , hive.server2.logging.operation.log.location ), but should general guidance be given in one of the HS2 wikidocs? The logging section in Getting Started could also mention this. HiveServer2 Clients Setting Up HiveServer2 Getting Started – Error Logs Configuration Properties – HiveServer2
          Hide
          Dong Chen added a comment -

          Lefty Leverenz, Thanks for your comments. For the document, I think adding the two new parameters in Configuration Properties might be enough now.
          Since this patch only impact the Thrift interface, maybe the general guidance is not needed for users. The getting log API of JDBC layer, which is an upper level of Thrift layer, will be added in HIVE-7615. I think after that Jira is done, a general guidance about how to use the query logs api could be introduced then.

          Show
          Dong Chen added a comment - Lefty Leverenz , Thanks for your comments. For the document, I think adding the two new parameters in Configuration Properties might be enough now. Since this patch only impact the Thrift interface, maybe the general guidance is not needed for users. The getting log API of JDBC layer, which is an upper level of Thrift layer, will be added in HIVE-7615 . I think after that Jira is done, a general guidance about how to use the query logs api could be introduced then.
          Hide
          Lefty Leverenz added a comment -

          Doc note: HIVE-8785 extends the description of hive.server2.logging.operation.enabled to "When true, HS2 will save operation logs and make them available for clients" (emphasis added).

          Show
          Lefty Leverenz added a comment - Doc note: HIVE-8785 extends the description of hive.server2.logging.operation.enabled to "When true, HS2 will save operation logs and make them available for clients " (emphasis added).
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Lefty Leverenz added a comment -

          Doc note: hive.server2.logging.operation.enabled and hive.server2.logging.operation.log.location are now documented in the wiki, but I need help with the location's default value. (I tried to generate a template file for 0.14.0, but failed because my new laptop didn't have Maven or JDK and I hit an installation snag.)

          In the patch, the default location is "$system:java.io.tmpdir" + File.separator + "$system:user.name" + File.separator + "operation_logs" – braces omitted – but surely that can be simplified for the doc.

          Also, now that HIVE-7615 has been committed we need some guidance about how to use the query logs API.

          Show
          Lefty Leverenz added a comment - Doc note: hive.server2.logging.operation.enabled and hive.server2.logging.operation.log.location are now documented in the wiki, but I need help with the location's default value. (I tried to generate a template file for 0.14.0, but failed because my new laptop didn't have Maven or JDK and I hit an installation snag.) In the patch, the default location is "$system:java.io.tmpdir" + File.separator + "$system:user.name" + File.separator + "operation_logs" – braces omitted – but surely that can be simplified for the doc. hive.server2.logging.operation.enabled hive.server2.logging.operation.log.location Also, now that HIVE-7615 has been committed we need some guidance about how to use the query logs API.
          Hide
          Dong Chen added a comment -

          The location's value in template is ${system:java.io.tmpdir}/${system:user.name}/operation_logs. And for doc, I think it can be more simplified as ${java.io.tmpdir}/${user.name}/operation_logs

          Show
          Dong Chen added a comment - The location's value in template is ${system:java.io.tmpdir}/${system:user.name}/operation_logs . And for doc, I think it can be more simplified as ${java.io.tmpdir}/${user.name}/operation_logs
          Hide
          Lefty Leverenz added a comment -

          Done, thanks Dong Chen.

          Guidance about configuring and using the query logs could be added to one of the HiveServer2 docs (probably Setting Up HS2), with links to & from the logging section in Getting Started. Also see HIVE-8785 for hive.server2.logging.operation.verbose.

          Show
          Lefty Leverenz added a comment - Done, thanks Dong Chen . Guidance about configuring and using the query logs could be added to one of the HiveServer2 docs (probably Setting Up HS2), with links to & from the logging section in Getting Started. Also see HIVE-8785 for hive.server2.logging.operation.verbose . Setting Up HiveServer2 HiveServer2 Clients Getting Started – Error Logs

            People

            • Assignee:
              Dong Chen
              Reporter:
              Shreepadma Venugopalan
            • Votes:
              5 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development