Hive
  1. Hive
  2. HIVE-7224

Set incremental printing to true by default in Beeline

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.13.0
    • Fix Version/s: 1.2.0
    • Component/s: Clients, JDBC
    • Labels:

      Description

      See HIVE-7221.

      By default beeline tries to buffer the entire output relation before printing it on stdout. This can cause OOM when the output relation is large. However, beeline has the option of incremental prints. We should keep that as the default.

      1. HIVE-7224.1.patch
        1 kB
        Vaibhav Gumashta

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          HIVE-7448 is resolved in 0.13 via HIVE-3611.

          Can this patch go in? I got no review authority, but the patch change appears good enough to me.

          (Except the unwanted changes such as whitespace fixes, which should ideally just be done across all the source files with a direct, single commit than by polluting random commits with such changes – but thats just IMHO).

          Show
          Harsh J added a comment - HIVE-7448 is resolved in 0.13 via HIVE-3611 . Can this patch go in? I got no review authority, but the patch change appears good enough to me. (Except the unwanted changes such as whitespace fixes, which should ideally just be done across all the source files with a direct, single commit than by polluting random commits with such changes – but thats just IMHO).
          Hide
          Xingwen Zhang added a comment -
          Show
          Xingwen Zhang added a comment - Ticket file. See https://issues.apache.org/jira/browse/HIVE-7448
          Hide
          Thejas M Nair added a comment -

          Xingwen Zhang Thanks for the information. Looks like we have another issue to be fixed here. Please open another jira to track this if you can.

          Show
          Thejas M Nair added a comment - Xingwen Zhang Thanks for the information. Looks like we have another issue to be fixed here. Please open another jira to track this if you can.
          Hide
          Xingwen Zhang added a comment -

          We would like to report that "--incremental=true" in Beeline is not compatible with Binary columns (at least for queries like "select * from some_table_with_a_binary_column limit 10;"). The error trace we got was:
          Error: Invalid column type: -2 (state=,code=0)
          java.sql.SQLException: Invalid column type: -2
          at org.apache.hive.jdbc.JdbcColumn.columnDisplaySize(JdbcColumn.java:94)
          at org.apache.hive.jdbc.HiveResultSetMetaData.getColumnDisplaySize(HiveResultSetMetaData.java:60)
          at org.apache.hive.beeline.IncrementalRows.<init>(IncrementalRows.java:55)
          at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1570)
          at org.apache.hive.beeline.Commands.execute(Commands.java:745)
          at org.apache.hive.beeline.Commands.sql(Commands.java:657)
          at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:794)
          at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:661)
          at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:366)
          at org.apache.hive.beeline.BeeLine.main(BeeLine.java:349)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

          The error was gone when we reverted back to " "--incremental=false".

          BTW: we are using Hive 0.12.

          Show
          Xingwen Zhang added a comment - We would like to report that "--incremental=true" in Beeline is not compatible with Binary columns (at least for queries like "select * from some_table_with_a_binary_column limit 10;"). The error trace we got was: Error: Invalid column type: -2 (state=,code=0) java.sql.SQLException: Invalid column type: -2 at org.apache.hive.jdbc.JdbcColumn.columnDisplaySize(JdbcColumn.java:94) at org.apache.hive.jdbc.HiveResultSetMetaData.getColumnDisplaySize(HiveResultSetMetaData.java:60) at org.apache.hive.beeline.IncrementalRows.<init>(IncrementalRows.java:55) at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1570) at org.apache.hive.beeline.Commands.execute(Commands.java:745) at org.apache.hive.beeline.Commands.sql(Commands.java:657) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:794) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:661) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:366) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:349) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) The error was gone when we reverted back to " "--incremental=false". BTW: we are using Hive 0.12.
          Hide
          Thejas M Nair added a comment -

          On the column truncation issue, I think we should not truncate columns by default. What do you think?

          I agree, truncation is bad. That is the opinion I express in HIVE-6928 also.

          Show
          Thejas M Nair added a comment - On the column truncation issue, I think we should not truncate columns by default. What do you think? I agree, truncation is bad. That is the opinion I express in HIVE-6928 also.
          Hide
          Vaibhav Gumashta added a comment -

          Thejas M Nair Sure, let me look at that. If that is the case, I think we should have it consistent for both incremental/non-incremental fetches.

          On the column truncation issue, I think we should not truncate columns by default. What do you think?

          Show
          Vaibhav Gumashta added a comment - Thejas M Nair Sure, let me look at that. If that is the case, I think we should have it consistent for both incremental/non-incremental fetches. On the column truncation issue, I think we should not truncate columns by default. What do you think?
          Hide
          Thejas M Nair added a comment -

          I am more concerned about the column width used because the table output format tends to truncate results . See HIVE-6928.

          Show
          Thejas M Nair added a comment - I am more concerned about the column width used because the table output format tends to truncate results . See HIVE-6928 .
          Hide
          Thejas M Nair added a comment -

          Vaibhav Gumashta It looks like the way the column sizes are estimated is different in case of incremental and buffered mode. In case of incremental mode, it seems to look at the ResultSetMetadata.getColumnDisplaySize to estimate how much width is to be given for each column. Can you check how different the formatting is with these two modes ?

          Show
          Thejas M Nair added a comment - Vaibhav Gumashta It looks like the way the column sizes are estimated is different in case of incremental and buffered mode. In case of incremental mode, it seems to look at the ResultSetMetadata.getColumnDisplaySize to estimate how much width is to be given for each column. Can you check how different the formatting is with these two modes ?
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12650083/HIVE-7224.1.patch

          ERROR: -1 due to 6 failed/errored test(s), 5535 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty
          org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.templeton.tool.TestTempletonUtils.testPropertiesParsing
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/453/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/453/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-453/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12650083

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12650083/HIVE-7224.1.patch ERROR: -1 due to 6 failed/errored test(s), 5535 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmpty org.apache.hive.beeline.TestBeeLineWithArgs.testNullEmptyCmdArg org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.templeton.tool.TestTempletonUtils.testPropertiesParsing Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/453/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/453/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-453/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12650083
          Hide
          Lefty Leverenz added a comment -

          [~vaibhavgumashta], thanks for documenting "--incremental" in the wiki. We can add the default with version information after this jira commits.

          Do you have time to deal with other Beeline doc issues that I raised in a comment on HIVE-6173?

          Show
          Lefty Leverenz added a comment - [~vaibhavgumashta] , thanks for documenting "--incremental" in the wiki. We can add the default with version information after this jira commits. Do you have time to deal with other Beeline doc issues that I raised in a comment on HIVE-6173 ? HiveServer2 Clients: Beeline Command Options (--incremental) HIVE-6173 comment: Beeline doc issues
          Hide
          Vaibhav Gumashta added a comment -
          Show
          Vaibhav Gumashta added a comment - cc Xuefu Zhang Thejas M Nair

            People

            • Assignee:
              Vaibhav Gumashta
              Reporter:
              Vaibhav Gumashta
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development