Hive
  1. Hive
  2. HIVE-7224

Set incremental printing to true by default in Beeline

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 1.0.0, 1.2.0, 1.1.0
    • Fix Version/s: 2.2.0
    • Component/s: Beeline, Clients, JDBC
    • Labels:

      Description

      See HIVE-7221.

      By default beeline tries to buffer the entire output relation before printing it on stdout. This can cause OOM when the output relation is large. However, beeline has the option of incremental prints. We should keep that as the default.

      1. HIVE-7224.1.patch
        1 kB
        Vaibhav Gumashta
      2. HIVE-7224.2.patch
        0.6 kB
        Xuefu Zhang
      3. HIVE-7224.2.patch
        0.6 kB
        Vaibhav Gumashta
      4. HIVE-7224.3.patch
        0.6 kB
        Sahil Takiar
      5. HIVE-7224.4.patch
        0.6 kB
        Sahil Takiar
      6. HIVE-7224.5.patch
        0.6 kB
        Sahil Takiar

        Issue Links

          Activity

          Hide
          Vaibhav Gumashta added a comment -
          Show
          Vaibhav Gumashta added a comment - cc Xuefu Zhang Thejas M Nair
          Hide
          Lefty Leverenz added a comment -

          [~vaibhavgumashta], thanks for documenting "--incremental" in the wiki. We can add the default with version information after this jira commits.

          Do you have time to deal with other Beeline doc issues that I raised in a comment on HIVE-6173?

          Show
          Lefty Leverenz added a comment - [~vaibhavgumashta] , thanks for documenting "--incremental" in the wiki. We can add the default with version information after this jira commits. Do you have time to deal with other Beeline doc issues that I raised in a comment on HIVE-6173 ? HiveServer2 Clients: Beeline Command Options (--incremental) HIVE-6173 comment: Beeline doc issues
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12650083/HIVE-7224.1.patch

          ERROR: -1 due to 6 failed/errored test(s), 5535 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty
          org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.templeton.tool.TestTempletonUtils.testPropertiesParsing
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/453/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/453/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-453/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12650083

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12650083/HIVE-7224.1.patch ERROR: -1 due to 6 failed/errored test(s), 5535 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.templeton.tool.TestTempletonUtils.testPropertiesParsing Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/453/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/453/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-453/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12650083
          Hide
          Thejas M Nair added a comment -

          Vaibhav Gumashta It looks like the way the column sizes are estimated is different in case of incremental and buffered mode. In case of incremental mode, it seems to look at the ResultSetMetadata.getColumnDisplaySize to estimate how much width is to be given for each column. Can you check how different the formatting is with these two modes ?

          Show
          Thejas M Nair added a comment - Vaibhav Gumashta It looks like the way the column sizes are estimated is different in case of incremental and buffered mode. In case of incremental mode, it seems to look at the ResultSetMetadata.getColumnDisplaySize to estimate how much width is to be given for each column. Can you check how different the formatting is with these two modes ?
          Hide
          Thejas M Nair added a comment -

          I am more concerned about the column width used because the table output format tends to truncate results . See HIVE-6928.

          Show
          Thejas M Nair added a comment - I am more concerned about the column width used because the table output format tends to truncate results . See HIVE-6928 .
          Hide
          Vaibhav Gumashta added a comment -

          Thejas M Nair Sure, let me look at that. If that is the case, I think we should have it consistent for both incremental/non-incremental fetches.

          On the column truncation issue, I think we should not truncate columns by default. What do you think?

          Show
          Vaibhav Gumashta added a comment - Thejas M Nair Sure, let me look at that. If that is the case, I think we should have it consistent for both incremental/non-incremental fetches. On the column truncation issue, I think we should not truncate columns by default. What do you think?
          Hide
          Thejas M Nair added a comment -

          On the column truncation issue, I think we should not truncate columns by default. What do you think?

          I agree, truncation is bad. That is the opinion I express in HIVE-6928 also.

          Show
          Thejas M Nair added a comment - On the column truncation issue, I think we should not truncate columns by default. What do you think? I agree, truncation is bad. That is the opinion I express in HIVE-6928 also.
          Hide
          Xingwen Zhang added a comment -

          We would like to report that "--incremental=true" in Beeline is not compatible with Binary columns (at least for queries like "select * from some_table_with_a_binary_column limit 10;"). The error trace we got was:
          Error: Invalid column type: -2 (state=,code=0)
          java.sql.SQLException: Invalid column type: -2
          at org.apache.hive.jdbc.JdbcColumn.columnDisplaySize(JdbcColumn.java:94)
          at org.apache.hive.jdbc.HiveResultSetMetaData.getColumnDisplaySize(HiveResultSetMetaData.java:60)
          at org.apache.hive.beeline.IncrementalRows.<init>(IncrementalRows.java:55)
          at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1570)
          at org.apache.hive.beeline.Commands.execute(Commands.java:745)
          at org.apache.hive.beeline.Commands.sql(Commands.java:657)
          at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:794)
          at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:661)
          at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:366)
          at org.apache.hive.beeline.BeeLine.main(BeeLine.java:349)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

          The error was gone when we reverted back to " "--incremental=false".

          BTW: we are using Hive 0.12.

          Show
          Xingwen Zhang added a comment - We would like to report that "--incremental=true" in Beeline is not compatible with Binary columns (at least for queries like "select * from some_table_with_a_binary_column limit 10;"). The error trace we got was: Error: Invalid column type: -2 (state=,code=0) java.sql.SQLException: Invalid column type: -2 at org.apache.hive.jdbc.JdbcColumn.columnDisplaySize(JdbcColumn.java:94) at org.apache.hive.jdbc.HiveResultSetMetaData.getColumnDisplaySize(HiveResultSetMetaData.java:60) at org.apache.hive.beeline.IncrementalRows.<init>(IncrementalRows.java:55) at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1570) at org.apache.hive.beeline.Commands.execute(Commands.java:745) at org.apache.hive.beeline.Commands.sql(Commands.java:657) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:794) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:661) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:366) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:349) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) The error was gone when we reverted back to " "--incremental=false". BTW: we are using Hive 0.12.
          Hide
          Thejas M Nair added a comment -

          Xingwen Zhang Thanks for the information. Looks like we have another issue to be fixed here. Please open another jira to track this if you can.

          Show
          Thejas M Nair added a comment - Xingwen Zhang Thanks for the information. Looks like we have another issue to be fixed here. Please open another jira to track this if you can.
          Hide
          Xingwen Zhang added a comment -
          Show
          Xingwen Zhang added a comment - Ticket file. See https://issues.apache.org/jira/browse/HIVE-7448
          Hide
          Harsh J added a comment -

          HIVE-7448 is resolved in 0.13 via HIVE-3611.

          Can this patch go in? I got no review authority, but the patch change appears good enough to me.

          (Except the unwanted changes such as whitespace fixes, which should ideally just be done across all the source files with a direct, single commit than by polluting random commits with such changes – but thats just IMHO).

          Show
          Harsh J added a comment - HIVE-7448 is resolved in 0.13 via HIVE-3611 . Can this patch go in? I got no review authority, but the patch change appears good enough to me. (Except the unwanted changes such as whitespace fixes, which should ideally just be done across all the source files with a direct, single commit than by polluting random commits with such changes – but thats just IMHO).
          Hide
          Sushanth Sowmyan added a comment -

          Removing fix version of 1.2.0 in preparation of release, since this is not a blocker for 1.2.0.

          Show
          Sushanth Sowmyan added a comment - Removing fix version of 1.2.0 in preparation of release, since this is not a blocker for 1.2.0.
          Hide
          Thejas M Nair added a comment -

          Vaibhav Gumashta can you please rebase ?

          Show
          Thejas M Nair added a comment - Vaibhav Gumashta can you please rebase ?
          Hide
          Vaibhav Gumashta added a comment -

          Rebased patch.

          Show
          Vaibhav Gumashta added a comment - Rebased patch.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12750158/HIVE-7224.2.patch

          ERROR: -1 due to 13 failed/errored test(s), 9348 tests executed
          Failed tests:

          org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes
          org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput
          org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes
          org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput
          org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes
          org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty
          org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg
          org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output
          org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes
          org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput
          org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes
          org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery
          org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4941/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4941/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4941/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 13 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12750158 - PreCommit-HIVE-TRUNK-Build

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12750158/HIVE-7224.2.patch ERROR: -1 due to 13 failed/errored test(s), 9348 tests executed Failed tests: org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4941/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4941/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4941/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed This message is automatically generated. ATTACHMENT ID: 12750158 - PreCommit-HIVE-TRUNK-Build
          Hide
          Xuefu Zhang added a comment -

          Reattached the same patch to trigger precommit test run.

          Show
          Xuefu Zhang added a comment - Reattached the same patch to trigger precommit test run.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12755464/HIVE-7224.2.patch

          ERROR: -1 due to 14 failed/errored test(s), 9424 tests executed
          Failed tests:

          org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
          org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes
          org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput
          org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes
          org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput
          org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes
          org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty
          org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg
          org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output
          org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes
          org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput
          org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes
          org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
          org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5255/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5255/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5255/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 14 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12755464 - PreCommit-HIVE-TRUNK-Build

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12755464/HIVE-7224.2.patch ERROR: -1 due to 14 failed/errored test(s), 9424 tests executed Failed tests: org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers org.apache.hive.beeline.TestBeeLineWithArgs.testCSV2OutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testCSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testDSVOutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2Output org.apache.hive.beeline.TestBeeLineWithArgs.testTSV2OutputWithDoubleQuotes org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutput org.apache.hive.beeline.TestBeeLineWithArgs.testTSVOutputWithDoubleQuotes org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.jdbc.TestSSL.testSSLFetchHttp Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5255/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5255/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5255/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed This message is automatically generated. ATTACHMENT ID: 12755464 - PreCommit-HIVE-TRUNK-Build
          Hide
          Xuefu Zhang added a comment -

          Vaibhav Gumashta, it looks like that the patch has caused some test failure. Could you please investigate? Thanks.

          Show
          Xuefu Zhang added a comment - Vaibhav Gumashta , it looks like that the patch has caused some test failure. Could you please investigate? Thanks.
          Hide
          Vaibhav Gumashta added a comment -

          Xuefu Zhang Sure, I'll look into the test failures.

          Show
          Vaibhav Gumashta added a comment - Xuefu Zhang Sure, I'll look into the test failures.
          Hide
          Sahil Takiar added a comment -

          Attaching a new, re-based patch in order to trigger build and see if the test failures are still present.

          Show
          Sahil Takiar added a comment - Attaching a new, re-based patch in order to trigger build and see if the test failures are still present.
          Hide
          Sahil Takiar added a comment -

          Vaibhav Gumashta I wanted to help moving this ticket forward, it seems there were some test failures related to this change. Is it ok if I assign this ticket to myself and start working on resolving the test failures?

          Show
          Sahil Takiar added a comment - Vaibhav Gumashta I wanted to help moving this ticket forward, it seems there were some test failures related to this change. Is it ok if I assign this ticket to myself and start working on resolving the test failures?
          Hide
          Vaibhav Gumashta added a comment -

          Sahil Takiar Sure, thanks for taking it up.

          Show
          Vaibhav Gumashta added a comment - Sahil Takiar Sure, thanks for taking it up.
          Hide
          Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12813155/HIVE-7224.3.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 23 failed/errored test(s), 10288 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join1
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_10
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_deleteAnalyze
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_complex
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform1
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_join
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_udf
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby6
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_mapjoin1
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_17
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_limit
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_math_funcs
          org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-330/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 23 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12813155 - PreCommit-HIVE-MASTER-Build

          Show
          Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12813155/HIVE-7224.3.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 23 failed/errored test(s), 10288 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_10 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_deleteAnalyze org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part_all_complex org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_udf org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby6 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_mapjoin1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_17 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_limit org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_math_funcs org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-330/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 23 tests failed This message is automatically generated. ATTACHMENT ID: 12813155 - PreCommit-HIVE-MASTER-Build
          Hide
          Sahil Takiar added a comment -

          None of these test failures seem related to the given change. All Beeline tests are passing.

          The new patch I attached is a rebased version of the previous patch.

          Is there anything else blocking this from being merged?

          Show
          Sahil Takiar added a comment - None of these test failures seem related to the given change. All Beeline tests are passing. The new patch I attached is a rebased version of the previous patch. Is there anything else blocking this from being merged?
          Hide
          Vaibhav Gumashta added a comment -

          Sahil Takiar Thanks a lot for taking this up. In the previous iterations, we've had issues with TestBeeLineWithArgs which was failing. From the tests listed here: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/testReport, doesn't look like those tests were run. Can you verify?

          Show
          Vaibhav Gumashta added a comment - Sahil Takiar Thanks a lot for taking this up. In the previous iterations, we've had issues with TestBeeLineWithArgs which was failing. From the tests listed here: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/testReport , doesn't look like those tests were run. Can you verify?
          Show
          Vaibhav Gumashta added a comment - Actually it does seem those tests were run and pass now: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/330/testReport/org.apache.hive.beeline/TestBeeLineWithArgs/
          Hide
          Vaibhav Gumashta added a comment -

          Sahil Takiar I'll do some quick manual testing and post my feedback. I think this looks good to go in.

          Show
          Vaibhav Gumashta added a comment - Sahil Takiar I'll do some quick manual testing and post my feedback. I think this looks good to go in.
          Hide
          Sahil Takiar added a comment -

          Thanks Vaibhav Gumashta! Let me know if you need any help with the manual testing.

          Show
          Sahil Takiar added a comment - Thanks Vaibhav Gumashta ! Let me know if you need any help with the manual testing.
          Hide
          Vaibhav Gumashta added a comment - - edited

          Sahil Takiar There seems to be an issue with the column width estimation when incremental printing is enabled by default. An example (output of describe table):
          With incremental true:

          | search_engine   | string          |                 |
          | exclude_hit     | string          |                 |
          | hier1           | string          |                 |
          | hier2           | string          |                 |
          | hier3           | string          |                 |
          | hier4           | string          |                 |
          | hier5           | string          |                 |
          | browser         | string          |                 |
          | post_browser_height | string          |                 |
          | post_browser_width | string          |                 |
          | post_cookies    | string          |                 |
          | post_java_enabled | string          |                 |
          | post_persistent_cookie | string          |                 |
          | color           | string          |                 |
          | connection_type | string          |                 |
          | country         | string          |                 |
          | domain          | string          |                 |
          | post_t_time_info | string          |                 |
          | javascript      | string          |                 |
          | language        | string          |                 |
          | os              | string          |                 |
          | plugins         | string          |                 |
          | resolution      | string          |                 |
          | last_hit_time_gmt | string          |                 |
          | first_hit_time_gmt | string          |                 |
          | visit_start_time_gmt | string          |                 |
          | last_purchase_time_gmt | string          |                 |
          +-----------------+-----------------+-----------------+--+
          |    col_name     |    data_type    |     comment     |
          +-----------------+-----------------+-----------------+--+
          | last_purchase_num | string          |                 |
          | first_hit_page_url | string          |                 |
          | first_hit_pagename | string          |                 |
          | visit_start_page_url | string          |                 |
          | visit_start_pagename | string          |                 |
          | first_hit_referrer | string          |                 |
          | visit_referrer  | string          |                 |
          | visit_search_engine | string          |                 |
          | visit_num       | string          |                 |
          | visit_page_num  | string          |                 |
          | prev_page       | string          |                 |
          | geo_city        | string          |                 |
          | geo_country     | string          |                 |
          | geo_region      | string          |                 |
          | duplicate_purchase | string          |                 |
          

          With incremental false:

          | search_engine            | string                |                       |
          | exclude_hit              | string                |                       |
          | hier1                    | string                |                       |
          | hier2                    | string                |                       |
          | hier3                    | string                |                       |
          | hier4                    | string                |                       |
          | hier5                    | string                |                       |
          | browser                  | string                |                       |
          | post_browser_height      | string                |                       |
          | post_browser_width       | string                |                       |
          | post_cookies             | string                |                       |
          | post_java_enabled        | string                |                       |
          | post_persistent_cookie   | string                |                       |
          | color                    | string                |                       |
          | connection_type          | string                |                       |
          | country                  | string                |                       |
          | domain                   | string                |                       |
          | post_t_time_info         | string                |                       |
          | javascript               | string                |                       |
          | language                 | string                |                       |
          | os                       | string                |                       |
          | plugins                  | string                |                       |
          | resolution               | string                |                       |
          | last_hit_time_gmt        | string                |                       |
          | first_hit_time_gmt       | string                |                       |
          | visit_start_time_gmt     | string                |                       |
          | last_purchase_time_gmt   | string                |                       |
          +--------------------------+-----------------------+-----------------------+--+
          |         col_name         |       data_type       |        comment        |
          +--------------------------+-----------------------+-----------------------+--+
          | last_purchase_num        | string                |                       |
          | first_hit_page_url       | string                |                       |
          | first_hit_pagename       | string                |                       |
          | visit_start_page_url     | string                |                       |
          | visit_start_pagename     | string                |                       |
          | first_hit_referrer       | string                |                       |
          | visit_referrer           | string                |                       |
          | visit_search_engine      | string                |                       |
          | visit_num                | string                |                       |
          | visit_page_num           | string                |                       |
          | prev_page                | string                |                       |
          | geo_city                 | string                |                       |
          | geo_country              | string                |                       |
          | geo_region               | string                |                       |
          | duplicate_purchase       | string                |                       |
          

          I think we'll need to fix this before we can get this in.

          Show
          Vaibhav Gumashta added a comment - - edited Sahil Takiar There seems to be an issue with the column width estimation when incremental printing is enabled by default. An example (output of describe table): With incremental true: | search_engine | string | | | exclude_hit | string | | | hier1 | string | | | hier2 | string | | | hier3 | string | | | hier4 | string | | | hier5 | string | | | browser | string | | | post_browser_height | string | | | post_browser_width | string | | | post_cookies | string | | | post_java_enabled | string | | | post_persistent_cookie | string | | | color | string | | | connection_type | string | | | country | string | | | domain | string | | | post_t_time_info | string | | | javascript | string | | | language | string | | | os | string | | | plugins | string | | | resolution | string | | | last_hit_time_gmt | string | | | first_hit_time_gmt | string | | | visit_start_time_gmt | string | | | last_purchase_time_gmt | string | | +-----------------+-----------------+-----------------+--+ | col_name | data_type | comment | +-----------------+-----------------+-----------------+--+ | last_purchase_num | string | | | first_hit_page_url | string | | | first_hit_pagename | string | | | visit_start_page_url | string | | | visit_start_pagename | string | | | first_hit_referrer | string | | | visit_referrer | string | | | visit_search_engine | string | | | visit_num | string | | | visit_page_num | string | | | prev_page | string | | | geo_city | string | | | geo_country | string | | | geo_region | string | | | duplicate_purchase | string | | With incremental false: | search_engine | string | | | exclude_hit | string | | | hier1 | string | | | hier2 | string | | | hier3 | string | | | hier4 | string | | | hier5 | string | | | browser | string | | | post_browser_height | string | | | post_browser_width | string | | | post_cookies | string | | | post_java_enabled | string | | | post_persistent_cookie | string | | | color | string | | | connection_type | string | | | country | string | | | domain | string | | | post_t_time_info | string | | | javascript | string | | | language | string | | | os | string | | | plugins | string | | | resolution | string | | | last_hit_time_gmt | string | | | first_hit_time_gmt | string | | | visit_start_time_gmt | string | | | last_purchase_time_gmt | string | | +--------------------------+-----------------------+-----------------------+--+ | col_name | data_type | comment | +--------------------------+-----------------------+-----------------------+--+ | last_purchase_num | string | | | first_hit_page_url | string | | | first_hit_pagename | string | | | visit_start_page_url | string | | | visit_start_pagename | string | | | first_hit_referrer | string | | | visit_referrer | string | | | visit_search_engine | string | | | visit_num | string | | | visit_page_num | string | | | prev_page | string | | | geo_city | string | | | geo_country | string | | | geo_region | string | | | duplicate_purchase | string | | I think we'll need to fix this before we can get this in.
          Hide
          Sahil Takiar added a comment -

          Thanks Vaibhav Gumashta I'll dig into it some more and see what I can find.

          Show
          Sahil Takiar added a comment - Thanks Vaibhav Gumashta I'll dig into it some more and see what I can find.
          Hide
          Sahil Takiar added a comment -

          Vaibhav Gumashta is seems the behavior you are seeing is by design. Looking at https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommandOptions the following explanation of the --incremental property suggests that this is expected:

          Defaults to false. When set to false, the entire result set is fetched and buffered before being displayed, yielding optimal display column sizing. When set to true, result rows are displayed immediately as they are fetched, yielding lower latency and memory usage at the price of extra display column padding. Setting --incremental=true is recommended if you encounter an OutOfMemory on the client side (due to the fetched result set size being large).

          So it seems there is a tradeoff when using --incremental that the column padding won't be optimal, but memory usage will be better. This makes sense since the IncrementalRows class that controls this logic doesn't do any buffering of rows, so it cannot predict what the optimal column width should be since it only looks at one row at a time.

          I think a better approach for the IncrementalRows class would be to instead buffer 1000 rows at a time (by default, this value can be configurable), this way it can optimally set the column width for each set of 1000 rows. This shouldn't introduce memory issues unless each row is huge, in which case the use can decrease the buffer size to say 100 or 10.

          What do you think?

          Show
          Sahil Takiar added a comment - Vaibhav Gumashta is seems the behavior you are seeing is by design. Looking at https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommandOptions the following explanation of the --incremental property suggests that this is expected: Defaults to false. When set to false, the entire result set is fetched and buffered before being displayed, yielding optimal display column sizing. When set to true, result rows are displayed immediately as they are fetched, yielding lower latency and memory usage at the price of extra display column padding. Setting --incremental=true is recommended if you encounter an OutOfMemory on the client side (due to the fetched result set size being large). So it seems there is a tradeoff when using --incremental that the column padding won't be optimal, but memory usage will be better. This makes sense since the IncrementalRows class that controls this logic doesn't do any buffering of rows, so it cannot predict what the optimal column width should be since it only looks at one row at a time. I think a better approach for the IncrementalRows class would be to instead buffer 1000 rows at a time (by default, this value can be configurable), this way it can optimally set the column width for each set of 1000 rows. This shouldn't introduce memory issues unless each row is huge, in which case the use can decrease the buffer size to say 100 or 10. What do you think?
          Hide
          Vaibhav Gumashta added a comment -

          Sahil Takiar I agree that buffering would improve usability. Would you like to take a shot at it?

          Show
          Vaibhav Gumashta added a comment - Sahil Takiar I agree that buffering would improve usability. Would you like to take a shot at it?
          Hide
          Thejas M Nair added a comment -

          I think a better approach for the IncrementalRows class would be to instead buffer 1000 rows at a time (by default, this value can be configurable), this way it can optimally set the column width for each set of 1000 rows.

          I think it is better to keep the column width to be consistent across all rows in the output, rather than adjust it every 1000 rows. The purpose of using optimal display size is primarily to make it easier to read. If we change the column width periodically, it doesn't help with that. Also, if there is some application that assumes the column width remains the same across rows, that might break.

          ie, beeline could buffer only the first 1000 rows and use that to determine the optimal column width, and then output the rest unbuffered. Not having to buffer the remaining rows could also be better for performance.

          Show
          Thejas M Nair added a comment - I think a better approach for the IncrementalRows class would be to instead buffer 1000 rows at a time (by default, this value can be configurable), this way it can optimally set the column width for each set of 1000 rows. I think it is better to keep the column width to be consistent across all rows in the output, rather than adjust it every 1000 rows. The purpose of using optimal display size is primarily to make it easier to read. If we change the column width periodically, it doesn't help with that. Also, if there is some application that assumes the column width remains the same across rows, that might break. ie, beeline could buffer only the first 1000 rows and use that to determine the optimal column width, and then output the rest unbuffered. Not having to buffer the remaining rows could also be better for performance.
          Hide
          Sahil Takiar added a comment -

          I think it is better to keep the column width to be consistent across all rows in the output, rather than adjust it every 1000 rows.

          Thanks for the input Thejas M Nair! To clarify, what would happen if Beeline uses the first 1000 rows to calculate the width, but then row 1001th is longer than that width. In that case the width would have to be widened for that row, right? I think that is what the current implementation of --incremental is doing.

          I like the idea of keeping the width the same for all rows, even in incremental mode, but I'm not sure how it would work. Thoughts?

          Show
          Sahil Takiar added a comment - I think it is better to keep the column width to be consistent across all rows in the output, rather than adjust it every 1000 rows. Thanks for the input Thejas M Nair ! To clarify, what would happen if Beeline uses the first 1000 rows to calculate the width, but then row 1001th is longer than that width. In that case the width would have to be widened for that row, right? I think that is what the current implementation of --incremental is doing. I like the idea of keeping the width the same for all rows, even in incremental mode, but I'm not sure how it would work. Thoughts?
          Hide
          Thejas M Nair added a comment -

          To clarify, what would happen if Beeline uses the first 1000 rows to calculate the width, but then row 1001th is longer than that width.

          If 1001th row has column larger than the precomputed column width, that particular row would have the column with larger width to accommodate it. This would mean some rows have the separator "|" out of alignment with previous row. However, even if we recompute every 1000 rows, we could still have misalignment every 1000 rows.

          I looked at where the Row width gets used. The width is getting used only when --outputformat=table (ie TableOutputFormat class) is used .
          If someone is working on very large outputs, it is likely to be processed by other applications and not human eyes, and a *sv (eg csv) format is likely to be used. It doesn't make any sense waste cpu cycles computing the width in those cases. This is also the case where performance impact of this computation would be more visible.

          ie, If we can selectively enable buffering and width calculation only for TableOutputFormat, I don't think it would matter if we stick to column width based on first 1000 rows or recompute every 1000 rows.
          Looks like the Row subclasses have access to beeline options and would be able to determine what the output format is.

          Show
          Thejas M Nair added a comment - To clarify, what would happen if Beeline uses the first 1000 rows to calculate the width, but then row 1001th is longer than that width. If 1001th row has column larger than the precomputed column width, that particular row would have the column with larger width to accommodate it. This would mean some rows have the separator "|" out of alignment with previous row. However, even if we recompute every 1000 rows, we could still have misalignment every 1000 rows. I looked at where the Row width gets used. The width is getting used only when --outputformat=table (ie TableOutputFormat class) is used . If someone is working on very large outputs, it is likely to be processed by other applications and not human eyes, and a *sv (eg csv) format is likely to be used. It doesn't make any sense waste cpu cycles computing the width in those cases. This is also the case where performance impact of this computation would be more visible. ie, If we can selectively enable buffering and width calculation only for TableOutputFormat, I don't think it would matter if we stick to column width based on first 1000 rows or recompute every 1000 rows. Looks like the Row subclasses have access to beeline options and would be able to determine what the output format is.
          Hide
          Sahil Takiar added a comment -

          Thanks for the explanation Thejas M Nair!

          So if my understanding is correct, one approach would be to:

          • Modify the logic so that the width is only calculated if TableOutputFormat is used
            • This will save CPU cycles when the outputformat = a delimiter separated value format
          • Modify IncrementalRows so that it re-calculates the width every 1000 rows
            • I think the advantage of doing it every 1000 rows is just for human readability purposes

          Does that sound reasonable to you?

          Show
          Sahil Takiar added a comment - Thanks for the explanation Thejas M Nair ! So if my understanding is correct, one approach would be to: Modify the logic so that the width is only calculated if TableOutputFormat is used This will save CPU cycles when the outputformat = a delimiter separated value format Modify IncrementalRows so that it re-calculates the width every 1000 rows I think the advantage of doing it every 1000 rows is just for human readability purposes Does that sound reasonable to you?
          Hide
          Thejas M Nair added a comment -

          Sahil Takiar
          That sounds good to me.

          Show
          Thejas M Nair added a comment - Sahil Takiar That sounds good to me.
          Hide
          Thejas M Nair added a comment -

          You can probably add a method like isWidthFormatted in OutputFormat which TableOutputFormat would then set to true. Then pass the value to the Row constructors.

          Show
          Thejas M Nair added a comment - You can probably add a method like isWidthFormatted in OutputFormat which TableOutputFormat would then set to true. Then pass the value to the Row constructors.
          Hide
          Sahil Takiar added a comment -

          Sounds good! Thanks for all the suggestions! I just created two sub-tasks to encapsulate this work.

          Show
          Sahil Takiar added a comment - Sounds good! Thanks for all the suggestions! I just created two sub-tasks to encapsulate this work.
          Hide
          Sahil Takiar added a comment -

          Thejas M Nair and Vaibhav Gumashta the subtasks we discussed for this JIRA have been resolved. I've reattached the patch to see if there are new test failures reported by Hive QA. Assuming there are no test failures, are there any other items needed before merging this change?

          Show
          Sahil Takiar added a comment - Thejas M Nair and Vaibhav Gumashta the subtasks we discussed for this JIRA have been resolved. I've reattached the patch to see if there are new test failures reported by Hive QA. Assuming there are no test failures, are there any other items needed before merging this change?
          Hide
          Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12827105/HIVE-7224.4.patch

          ERROR: -1 due to build exiting with an error

          Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1110/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1110/console
          Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1110/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
          + export JAVA_HOME=/usr/java/jdk1.8.0_25
          + JAVA_HOME=/usr/java/jdk1.8.0_25
          + export PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          + PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-1110/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ git = \s\v\n ]]
          + [[ git = \g\i\t ]]
          + [[ -z master ]]
          + [[ -d apache-github-source-source ]]
          + [[ ! -d apache-github-source-source/.git ]]
          + [[ ! -d apache-github-source-source ]]
          + cd apache-github-source-source
          + git fetch origin
          + git reset --hard HEAD
          HEAD is now at e870ae1 HIVE-14697: Can not access kerberized HS2 Web UI (Chaoyu Tang, reviewed by Jimmy Xiang)
          + git clean -f -d
          Removing ql/src/test/queries/clientpositive/double_precision.q
          Removing ql/src/test/results/clientpositive/double_precision.q.out
          + git checkout master
          Already on 'master'
          Your branch is up-to-date with 'origin/master'.
          + git reset --hard origin/master
          HEAD is now at e870ae1 HIVE-14697: Can not access kerberized HS2 Web UI (Chaoyu Tang, reviewed by Jimmy Xiang)
          + git merge --ff-only origin/master
          Already up-to-date.
          + git gc
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          error: patch failed: beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java:71
          error: beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java: patch does not apply
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12827105 - PreCommit-HIVE-MASTER-Build

          Show
          Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12827105/HIVE-7224.4.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1110/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1110/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1110/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]] + export JAVA_HOME=/usr/java/jdk1.8.0_25 + JAVA_HOME=/usr/java/jdk1.8.0_25 + export PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-1110/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at e870ae1 HIVE-14697: Can not access kerberized HS2 Web UI (Chaoyu Tang, reviewed by Jimmy Xiang) + git clean -f -d Removing ql/src/test/queries/clientpositive/double_precision.q Removing ql/src/test/results/clientpositive/double_precision.q.out + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at e870ae1 HIVE-14697: Can not access kerberized HS2 Web UI (Chaoyu Tang, reviewed by Jimmy Xiang) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch error: patch failed: beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java:71 error: beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java: patch does not apply The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12827105 - PreCommit-HIVE-MASTER-Build
          Hide
          Sahil Takiar added a comment -

          Rebasing patch

          Show
          Sahil Takiar added a comment - Rebasing patch
          Hide
          Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12827107/HIVE-7224.5.patch

          ERROR: -1 due to no test(s) being added or modified.

          ERROR: -1 due to 6 failed/errored test(s), 10443 tests executed
          Failed tests:

          TestBeeLineWithArgs - did not produce a TEST-*.xml file
          TestHiveCli - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1111/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1111/console
          Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1111/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12827107 - PreCommit-HIVE-MASTER-Build

          Show
          Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12827107/HIVE-7224.5.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 6 failed/errored test(s), 10443 tests executed Failed tests: TestBeeLineWithArgs - did not produce a TEST-*.xml file TestHiveCli - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1111/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1111/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1111/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12827107 - PreCommit-HIVE-MASTER-Build
          Hide
          Sahil Takiar added a comment -

          Sergio Peña can this be merged? The change simply sets Beeline's incremental property to true. There were a few subtasks that needed to be completed before this could be merged, but they are all resolved now.

          Show
          Sahil Takiar added a comment - Sergio Peña can this be merged? The change simply sets Beeline's incremental property to true. There were a few subtasks that needed to be completed before this could be merged, but they are all resolved now.
          Hide
          Sergio Peña added a comment -

          Sure.
          +1

          Show
          Sergio Peña added a comment - Sure. +1
          Hide
          Lefty Leverenz added a comment -

          Doc note: The change of default for --incremental needs to be documented in the wiki for Hive 2.2.0.

          Added a TODOC2.2 label.

          Show
          Lefty Leverenz added a comment - Doc note: The change of default for --incremental needs to be documented in the wiki for Hive 2.2.0. HiveServer2 Clients – Beeline Command Options Added a TODOC2.2 label.
          Hide
          Sahil Takiar added a comment -

          Hey Lefty Leverenz could you grant me permission to edit the wiki page? Username is "stakiar"

          Show
          Sahil Takiar added a comment - Hey Lefty Leverenz could you grant me permission to edit the wiki page? Username is "stakiar"
          Hide
          Lefty Leverenz added a comment -

          Done. Welcome to the Hive wiki team, Sahil Takiar.

          Show
          Lefty Leverenz added a comment - Done. Welcome to the Hive wiki team, Sahil Takiar .

            People

            • Assignee:
              Sahil Takiar
              Reporter:
              Vaibhav Gumashta
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development