@Shreepadma: before making any code changes I think it would be a good idea to get feedback on the changes you plan to make to TCLIService and CLIService.

@Carl: The proposed addition to TCLIService.thrift is the following new API and structs,

// GetLog()
// Fetch operation log from the server corresponding to
// a particular OperationHandle.

struct TGetLogReq {
  // Operation whose log is requested
  1: required TOperationHandle operationHandle
}

struct TGetLogResp {
  1: required TStatus status
  2: required string log
}

service TCLIService {
...
...
TGetLogResp GetLog(1:TGetLogReq req);
}

Carl Steinbach: Can you look at this? Thanks!

Here are some questions I had while looking at the API:

• When am I allowed to call GetLog()? For example, what happens if I call it after calling CloseOperation or CancelOperation? If it's ok to call after CloseOperation()/CancelOperation(), what does it return, and what does it return if I first call a function that also returns an operationhandle?
• What happens if I call GetLog() twice during the same operation? Will it always return the entire log collected up to that point or just a piece of it?
• Do you think we log (or should log) anything of interest between the OpenSession() call and creation of the first OperationHandle, or between CloseSession() and the previous call to CancelOperation()/CloseOperation()?

I also thought of a possible alternate approach and wanted to get your opinion on it:

• Provide access to logs on a per-session basis instead of per-operation.
• Provide a LogHandle (either hanging off of the SessionHandle or provided via GetLogHandle()) and make it fetchable with FetchResults()
• We can provide per-operation log retrieval by supporting scrollable log cursors, e.g. scrolling to FETCH_LAST between operations will advance the log stream to the start of the next operation's log.

This API gets the operation log aka the per query log. Clients are mostly interested in the operation/per query log rather than the session log. Furthermore, session log will contain log output for a given query interspersed with the log output of other queries that may be executing concurrently in the same session. Note that executing multiple queries in a session is possible with async execution. The main purpose of this JIRA is to support an API to retrieve per query log. If there's enough interest, adding an API to support retrieving session log can be done in a follow on JIRA.

• Operation logs don't persist beyond the life of an operation. If getLog(opHandle) is called after a CloseOperation() or CancelOperation(), an exception is raised.

• If getLog() is called on the same operationHandle twice, the log up to the point will be returned. Note that the log is maintained as a circular buffer and loops back. The default size is 128 KB, but is configurable. We have not come across a case where 128 KB was insufficient.

Review board: https://reviews.apache.org/r/14326/

Overall: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12604928/HIVE-4629.1.patch

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/886/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/886/console

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-886/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen ql/src/test/results/clientpositive/cast_to_int.q.out ql/src/test/queries/clientpositive/cast_to_int.q
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1526135.

At revision 1526135.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'

This message is automatically generated.

I'm able to apply the patch with -p0 to the tip of trunk. I've re-attached the patch to trigger a run.

Overall: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12605136/HIVE-4629.2.patch

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/907/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/907/console

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-907/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'build-common.xml'
Reverted 'ql/src/test/queries/clientnegative/invalid_columns.q'
Reverted 'ql/src/test/queries/clientnegative/deletejar.q'
Reverted 'ql/src/test/queries/clientpositive/input16_cc.q'
Reverted 'ql/src/test/queries/clientpositive/alter1.q'
Reverted 'ql/src/test/queries/clientpositive/input16.q'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1526404.

At revision 1526404.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'

This message is automatically generated.

I think it makes sense to do a scrollable log in a future jira. Carl do you have any more concerns?

Brock Noland Making the logs scrollable wasn't the point of my suggestion. I'm more concerned about leveraging the existing fetch functions and patterns to satisfy a use case which fundamentally looks very similar to fetching a query result set. CLIService is a public interface. We're permanently stuck with any changes that are made to it. I'd like to avoid cluttering it with a mishmash of logging RPCs if it's possible to reduce all of these use cases to a single pattern.

In the future I hope we can finish discussing changes like this before the first patch is posted.

Shreepadma Venugopalan I left some comments on reviewboard. Thanks.

Carl, I see, as you know I took your comments as a suggestion for a scrollable log. I apologize for mis-understanding your comments.

Shreepadma Venugopalan I guess the patch is out of sync and won't apply. There are a bunch of changes to HS2 interface, the build framework changed from ant to maven etc. Would you mind rebasing it to the latest and update the review.

If you want I can take care of the rebase.

What about the status of this jira?
Does anyone try to rebase it to the latest trunk?
I think it is a useful feature especially when doing some testing about hql.

Does the new version of the patch address any of the API design issues I mentioned earlier?

It would be great to have this jira accepted into hive trunk, even I am waiting on this from a long time.

Any estimate on when this will be accepted into trunk?

This patch no longer applies on HIVE trunk. Is there a plan to upmerge it? And incorporate Carl's comments?

Just rebased to trunk. Let this get in.

Agreed, let's get this in.

Thanks a lot Navis for the updated patch!

@Brock ++ Are there any plans to incorporate Carl's comments and provide an updated patch?

@Carl: We need this functionality in Hive to continue providing Hue to our users. Hue is a pretty useful application, so we would like to continue providing it. Would you please consider accepting the patch?

Hi Ravi and Carl,

The current patch does not implement the changes requested by Carl. I do think the changes described by Carl would make the API more flexible. However, the current design does work well and thus I feel we should move forward with the current design.

Brock

Please give me a day to review this patch.
I had an initial pass on this, one concern I have is the 1 MB of log being returned on every call to getLog. I believe this is for use cases such as Hue. In such use cases there is only one consumer for the log, and giving out incremental logs would be better. ie , after a call to getLog, clear the buffer and the next getLog call returns only the new log lines. This will reduce the memory footprint in HiveServer2 and the load that getLog call puts on it. It will also make applications like Hue more responsive, and be able to refresh the log more frequently.

Does anyone think this is the right way to implement this feature?

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12647299/HIVE-4629.3.patch.txt

ERROR: -1 due to 7 failed/errored test(s), 5468 tests executed
Failed tests:

org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/343/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/343/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-343/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed

This message is automatically generated.

ATTACHMENT ID: 12647299

I have added some comments on review board .

I think this feature is very useful. A lot of other people have asked for it in the jira as well.
But I think we should address the review board comments and comments in this jira first.

We should first agree on the public api before committing the patch. The suggestion from Carl about using the FetchResults api for this makes sense. As the use cases for this evolves, I think it the getLog api is likely to start looking more like the FetchResults api.
The TFetchResultsReq can have an additional resulttype parameter (OUTPUT vs LOG).

Also, the circular in-memory buffer does not gurantee that you get the whole log, it also puts burden on the HS2 memory requirements. I think we should explore using the approach in HIVE-5924 for this instead of an inmemory pool. But I am OK with having such changes as part of follow up jiras, as long as we don't have the in-memory logging enabled by default in this patch.

I'm not intended to take this job(I'm not good at interface design) but just rebased the patch. Will be really appreciated if anyone take this work, which will be huge contribution to hive community.

Carl Steinbach Any idea on how to retrieve operation logs via currently existing APIs in ICliService? Should we extend select statement for that? (select _hive_operation_log from operation-handler, for example)

Is this something complicated? Can a hive beginner take up this task?

Ravi Prakash This can definitely be taken up. There is some good feedback on the jira too regarding the modifications. Let me know if you plan to take it up and need more help while you work on it. Thanks a lot!

Hi Vaibhav! Sorry for the delay. I wish I had enough cycles to take this up, but that's just not happening.

A patch HIVE-4629.4.patch is attached, which trying to address the comments in this jira and review board, to support an API to retrieve query logs.

This patch is developed based on HIVE-4629.3.patch code in this jira and refer to the method of saving operation log to files in HIVE-5924.

RB entry: https://reviews.apache.org/r/24293/

The design and changes are as follows:
1. Reuse FetchResults() API and add a TFetchType in the param. The type could be QUERY_OUTPUT and LOG.
2. Original LogManager code is modified and moved in OperationManager. Now OM is responsible for dispatching log updating and fetching request.
3. OperatinLog class wraps the actual operation log files. Each Operation create, register, and unregister the OperationLog around its run() method.
4. Make OperationLog to ThreadLocal, so that we don't have to bind and maintain them with threads in some Maps.
5. The actual operation log files are maintained in the hierarchy like:
rootDir/sessionDir/operationLogFiles
The rootDir is configurable, created when SessionManager init, deleted when stop.
The sessionDir is created when session is opened, and deleted during closing.
The operationLogFiles is create before operation runs, and deleted when it is closed.

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12659815/HIVE-4629.4.patch

ERROR: -1 due to 5 failed/errored test(s), 5854 tests executed
Failed tests:

org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_mixed_case
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/178/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/178/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-178/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed

This message is automatically generated.

ATTACHMENT ID: 12659815

Update the patch (v6) to trigger test. It addresses review comments and fixes one failed case related with this patch in HIVE QA.

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12660420/HIVE-4629.6.patch

ERROR: -1 due to 6 failed/errored test(s), 5875 tests executed
Failed tests:

org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/221/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/221/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-221/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed

This message is automatically generated.

ATTACHMENT ID: 12660420

Nice work Dong Chen!!

Thejas M Nair Carl Steinbach you two had some good feedback on the earlier design. Can you take a look at the latest patch? Romain Rigaux, I know Hue uses this API, do you want to take a look?

I mostly looked at the TCLIService.thrift. The new API looks good.

Is FETCH_FIRST also implemented for the TFetchOrientation? (would still be useful in case a user refresh the page or wants all the logs again)

Hi Romain Rigaux, thanks for your review. Yes, the FETCH_FIRST is implemented.

Dong Chen Thanks for the patch! I will review it in a few days.

Hi Dong,

I tried posting this on RB but it went down. Thank you very much for removing the thrift enum compatibility problem! I had another comment with regards to the method signature which I think I did not explain well. I think the new method should be:

FetchResultsResponse fetchResults(FetchResultsRequest) throws ...

The problem with how we've defined RPC methods to date has led to an explosion of RPC methods which is problematic. This is described in more detail in this thread.

Let me know what you think!!

Cheers,
Brock

Dong Chen Thanks for the clean and well documented code! I have added some minor comments in review board.

Dong Chen Earlier patch also had a method in HiveStatement to get the log. I think that will be convenient for many users, though we need to be careful and specify that is the only non jdbc function that is part of a public API in it. But this can also be done as follow up work in separate jira.

HIVE-4629.7.patch - Dong's patch from reviewboard, just to know how it is doing wrt tests.

Dong Chen Looking forward to updated patch from you!

I tried posting this on RB but it went down. Thank you very much for removing the thrift enum compatibility problem! I had another comment with regards to the method signature which I think I did not explain well. I think the new method should be...

Brock Noland, I totally agree with this, but I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else?

I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else?

You are right...I was confused between ICLIService and TCLIService. The patch only adds a new option member to TFetchResultsReq which is exactly what I wanted.

Nevermind me

Hi, Thejas M Nair, Brock Noland, Carl Steinbach, thanks for your precious comments. I am working on the updated patch to address these comments, and will let you know when the updated patch is uploaded.

Earlier patch also had a method in HiveStatement to get the log. I think that will be convenient for many users, though we need to be careful and specify that is the only non jdbc function that is part of a public API in it. But this can also be done as follow up work in separate jira.

Thejas M Nair, you are right, adding a method in HiveStatement to get the log is convenient for users. I filed a Jira HIVE-7615 and planed to add the getting log API in jdbc level there.

I didn't see this in the patch. Are you referring to something in the Thrift IDL file or something else?

Carl Steinbach, Brock Noland, the latest patch still does not fulfill the comments about backward compatibility. I will update the patch soon and let you know.
The Thrift level interface TCLIService is OK.
For client and service layer interface ICLIService, although it is not RPC and is not a public API of Hive, I think making it follow the single request/response struct mode is also good. Will make the new fetchResults method follow the single request/response struct model. Then remove those old fetchResults methods.

Overall: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12663483/HIVE-4629.7.patch

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/455/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/455/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-455/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-455/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriterV2.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/SerializationUtils.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target accumulo-handler/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1619731.

At revision 1619731.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'

This message is automatically generated.

ATTACHMENT ID: 12663483

Hi Dong,

The latest patch fails to apply to HEAD and will need to be rebased.

the latest patch still does not fulfill the comments about backward compatibility.

I am very sorry for the confusion, I believe the patch does meet the backward compatibility requirement.

For client and service layer interface ICLIService, although it is not RPC and is not a public API of Hive, I think making it follow the single request/response struct mode is also good. Will make the new fetchResults method follow the single request/response struct model. Then remove those old fetchResults methods.

I do not feel this is required. The current patch works exactly as we requested!

Thank you very much Dong!!

Update patch V8 to address the precious comments from Thejas M Nair, Carl Steinbach, and Brock Noland. Thanks very much for your review to make this patch better.

And rebase it from latest trunk.

Nice work Dong! From my side this looks great and I think we should commit.

+1 pending tests

Thejas or Carl, please let me know if you'd like to see additional work before commit.

Overall: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664360/HIVE-4629.8.patch

ERROR: -1 due to 1 failed/errored test(s), 6119 tests executed
Failed tests:

org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/505/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/505/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-505/

Messages:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed

This message is automatically generated.

ATTACHMENT ID: 12664360

+1 .
Thanks Dong, great work!
Looking forward to your contributions in HIVE-7615, it would be great to have that in hive 0.14 release !

I have committed this to trunk! Thank you Dong! This is truly a long awaited contribution!

FYI - I had to do one trivial rebase in SessionManager when committing so I've uploaded that patch as v9. I verified it does not break the build.

Does this need to be documented in the wiki?

Of course, two new parameters need to go in Configuration Properties (hive.server2.logging.operation.enabled, hive.server2.logging.operation.log.location), but should general guidance be given in one of the HS2 wikidocs? The logging section in Getting Started could also mention this.

• HiveServer2 Clients
• Setting Up HiveServer2
• Getting Started – Error Logs
• Configuration Properties – HiveServer2

Lefty Leverenz, Thanks for your comments. For the document, I think adding the two new parameters in Configuration Properties might be enough now.
Since this patch only impact the Thrift interface, maybe the general guidance is not needed for users. The getting log API of JDBC layer, which is an upper level of Thrift layer, will be added in HIVE-7615. I think after that Jira is done, a general guidance about how to use the query logs api could be introduced then.

Doc note: HIVE-8785 extends the description of hive.server2.logging.operation.enabled to "When true, HS2 will save operation logs and make them available for clients" (emphasis added).

This has been fixed in 0.14 release. Please open new jira if you see any issues.

Doc note: hive.server2.logging.operation.enabled and hive.server2.logging.operation.log.location are now documented in the wiki, but I need help with the location's default value. (I tried to generate a template file for 0.14.0, but failed because my new laptop didn't have Maven or JDK and I hit an installation snag.)

In the patch, the default location is "$system:java.io.tmpdir" + File.separator + "$system:user.name" + File.separator + "operation_logs" – braces omitted – but surely that can be simplified for the doc.

• hive.server2.logging.operation.enabled
• hive.server2.logging.operation.log.location

Also, now that HIVE-7615 has been committed we need some guidance about how to use the query logs API.

The location's value in template is ${system:java.io.tmpdir}/${system:user.name}/operation_logs. And for doc, I think it can be more simplified as ${java.io.tmpdir}/${user.name}/operation_logs

Done, thanks Dong Chen.

Guidance about configuring and using the query logs could be added to one of the HiveServer2 docs (probably Setting Up HS2), with links to & from the logging section in Getting Started. Also see HIVE-8785 for hive.server2.logging.operation.verbose.

• Setting Up HiveServer2
• HiveServer2 Clients
• Getting Started – Error Logs

As far as I understand, the logs are removed once the operation completes. So if my operation finished with an error and I want to see the logs after it fails, how do I proceed?