Hive
  1. Hive
  2. HIVE-6173

Beeline doesn't accept --hiveconf option as Hive CLI does

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11.0, 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: CLI
    • Labels:
    • Release Note:
      This (--hiveconf option) needs documentation if not being documented yet.

      Description

      > beeline -u jdbc:hive2:// --hiveconf a=b
      Usage: java org.apache.hive.cli.beeline.BeeLine 
      

      Since Beeline is replacing Hive CLI, it should support this command line option as well.

      1. HIVE-6173.patch
        6 kB
        Xuefu Zhang
      2. HIVE-6173.1.patch
        9 kB
        Xuefu Zhang
      3. HIVE-6173.2.patch
        8 kB
        Xuefu Zhang

        Issue Links

          Activity

          Hide
          Julian Hyde added a comment -

          In https://github.com/julianhyde/sqlline/issues/32, I have extended the output '!help set' to include all options, including showElapsedTime, maxHeight, maxWidth, timeout, trimScripts.

          Those changes could be merged into Hive's fork of sqlline.

          Show
          Julian Hyde added a comment - In https://github.com/julianhyde/sqlline/issues/32 , I have extended the output '!help set' to include all options, including showElapsedTime, maxHeight, maxWidth, timeout, trimScripts. Those changes could be merged into Hive's fork of sqlline.
          Lefty Leverenz made changes -
          Comment [
          Oops, my posting to the SQLLine mailing list didn't go through. So I'll try again after my request to join the list is approved. ]
          Hide
          Lefty Leverenz added a comment -

          I posted a question on the SQLLine mailing list (sqlline-dev@googlegroups.com) asking for help with these doc issues.

          By the way, four of the undocumented properties are in SqlLineOpts.java, but allowMultiLineCommand is only in BeeLineOpts.java.

          Julian Hyde, do you want to chime in?

          Show
          Lefty Leverenz added a comment - I posted a question on the SQLLine mailing list (sqlline-dev@googlegroups.com) asking for help with these doc issues. By the way, four of the undocumented properties are in SqlLineOpts.java, but allowMultiLineCommand is only in BeeLineOpts.java. Julian Hyde , do you want to chime in?
          Hide
          Xuefu Zhang added a comment -

          Aha. I see. These undocumented properties, such as maxHeight and trimScripts, are not internal, but unknown to majority of users (probably due to lack of documentation). While it's nice to have them documented, but doing so requires work load from the community. Not just write a few works about them, but also ensure they are doing what they are supposed to do. Here I give some descriptions w/o the merit of guaranty:

          1. showElapsedTime – whether to log elapsed time at command prompt. Default true.
          2. maxHeight, maxWidth – maximum height/width of the output. Default, the height/width of the terminal.
          3. timeout – unused
          4. trimScripts – whether to trim leading/trailing spaces/tabs in the script. Default true.
          5. allowMultiLineCommand – whether to allow multi-line commands. Default true.

          Show
          Xuefu Zhang added a comment - Aha. I see. These undocumented properties, such as maxHeight and trimScripts, are not internal, but unknown to majority of users (probably due to lack of documentation). While it's nice to have them documented, but doing so requires work load from the community. Not just write a few works about them, but also ensure they are doing what they are supposed to do. Here I give some descriptions w/o the merit of guaranty: 1. showElapsedTime – whether to log elapsed time at command prompt. Default true. 2. maxHeight, maxWidth – maximum height/width of the output. Default, the height/width of the terminal. 3. timeout – unused 4. trimScripts – whether to trim leading/trailing spaces/tabs in the script. Default true. 5. allowMultiLineCommand – whether to allow multi-line commands. Default true.
          Hide
          Lefty Leverenz added a comment -

          Thanks Xuefu Zhang, I'll update the wiki.

          For #7, replies A and B aren't alternatives. (B just points out that those options aren't documented anywhere, afaik. Maybe they don't need documentation.)

          Show
          Lefty Leverenz added a comment - Thanks Xuefu Zhang , I'll update the wiki. For #7, replies A and B aren't alternatives. (B just points out that those options aren't documented anywhere, afaik. Maybe they don't need documentation.)
          Hide
          Xuefu Zhang added a comment -

          Re #2: autocommit is still not applicable, even if Hive's update/delete.

          Re #4: "default" means "no format", which means values are read as strings.

          Re #7: Reply A seems better.

          Show
          Xuefu Zhang added a comment - Re #2: autocommit is still not applicable, even if Hive's update/delete. Re #4: "default" means "no format", which means values are read as strings. Re #7: Reply A seems better.
          Hide
          Lefty Leverenz added a comment -

          Doc: Strictly speaking, the TODOC13 label should be removed because --hiveconf has been documented. The release note should also be revised.

          But TODOC13 is still here for some followup issues which really belong in a separate jira. Maybe we can wrap them up here, though.

          > 2. Is the autoCommit default false as shown in BeeLineOpts.java or true as shown in the SQLLine doc?
          Hive doesn't support transaction or rollback concept. (It always auto-commit upon success.) As to this prop, it doesn't really matter as far as Hive is concerned. (Beeline is able to connect to other databases, in which case this may makes a difference.)

          Now that Hive has transactions, is Beeline's autoCommit useful? If not, should it be removed from the wiki or explained away?

          > 4. What is the default for numberFormat? BeeLineOpts.java says "default" but I don't know what that means.
          could you check what SQLLine doc says?

          The SQLLine doc doesn't cover numberFormat. Does it use the default for Java's DecimalFormat class? (See links below.)

          > 7. Should any of these options be documented? <see list below>
          SQLLine has a lot of properties, a little overwhelming. I'd think we should probably document only that's closely tied with Hive and refer user to SQLLine doc for others.

          Reply A: Then maybe some options should be removed from the wiki. At present only one option (historyfile) is in the SQLLine doc but not in the Beeline doc. Rather than direct readers to SQLLine for a single option, I'm inclined to add historyfile to the wiki.
          Reply B: The options listed in point 7 aren't covered by the SQLLine doc. They're in the code in BeeLineOpts but not in BeeLine.properties: showElapsedTime, maxHeight, timeout, trimScripts, allowMultiLineCommand. (Are they internal only?)

          Reference links:

          Show
          Lefty Leverenz added a comment - Doc: Strictly speaking, the TODOC13 label should be removed because --hiveconf has been documented. The release note should also be revised. But TODOC13 is still here for some followup issues which really belong in a separate jira. Maybe we can wrap them up here, though. > 2. Is the autoCommit default false as shown in BeeLineOpts.java or true as shown in the SQLLine doc? Hive doesn't support transaction or rollback concept. (It always auto-commit upon success.) As to this prop, it doesn't really matter as far as Hive is concerned. (Beeline is able to connect to other databases, in which case this may makes a difference.) Now that Hive has transactions, is Beeline's autoCommit useful? If not, should it be removed from the wiki or explained away? > 4. What is the default for numberFormat? BeeLineOpts.java says "default" but I don't know what that means. could you check what SQLLine doc says? The SQLLine doc doesn't cover numberFormat. Does it use the default for Java's DecimalFormat class? (See links below.) > 7. Should any of these options be documented? <see list below> SQLLine has a lot of properties, a little overwhelming. I'd think we should probably document only that's closely tied with Hive and refer user to SQLLine doc for others. Reply A: Then maybe some options should be removed from the wiki. At present only one option (historyfile) is in the SQLLine doc but not in the Beeline doc. Rather than direct readers to SQLLine for a single option, I'm inclined to add historyfile to the wiki. Reply B: The options listed in point 7 aren't covered by the SQLLine doc. They're in the code in BeeLineOpts but not in BeeLine.properties: showElapsedTime, maxHeight, timeout, trimScripts, allowMultiLineCommand. (Are they internal only?) Reference links: Beeline Command Options SQLLine documentation DecimalFormat class DecimalFormat constructor : "To obtain standard formats for a given locale, use the factory methods on NumberFormat such as getNumberInstance. These factories will return the most appropriate sub-class of NumberFormat for a given locale."
          Hide
          Xuefu Zhang added a comment -

          Sorry for missing the above message.

          1. Does the query string for -e take single quotes or double quotes?

          either works.

          2. Is the autoCommit default false as shown in BeeLineOpts.java or true as shown in the SQLLine doc?

          Hive doesn't support transaction or rollback concept. (It always auto-commit upon success.) As to this prop, it doesn't really matter as far as Hive is concerned. (Beeline is able to connect to other databases, in which case this may makes a difference.)

          3. What are the possible values & default value for isolation?

          I think you can give only one key-value per --hiveconf option.

          4. What is the default for numberFormat? BeeLineOpts.java says "default" but I don't know what that means.

          could you check what SQLLine doc says?

          5. Is the description of the default for maxWidth okay? (Modified from SQLLine doc.)

          seems fine to me.

          6. Does showNestedErrs have a JIRA ticket? (It isn't in the SQLLine doc.)

          not as I know of.

          7. Should any of these options be documented?

          --incremental (in BeeLineOpts & SQLLine doc, but not in BeeLine.properties)
          --showElapsedTime (in BeeLineOpts, not in BeeLine.properties nor SQLLine doc)
          --maxHeight (ditto)
          --timeout (ditto)
          --trimScripts (ditto)
          --allowMultiLineCommand (ditto)

          SQLLine has a lot of properties, a little overwhelming. I'd think we should probably document only that's closely tied with Hive and refer user to SQLLine doc for others.

          Show
          Xuefu Zhang added a comment - Sorry for missing the above message. 1. Does the query string for -e take single quotes or double quotes? either works. 2. Is the autoCommit default false as shown in BeeLineOpts.java or true as shown in the SQLLine doc? Hive doesn't support transaction or rollback concept. (It always auto-commit upon success.) As to this prop, it doesn't really matter as far as Hive is concerned. (Beeline is able to connect to other databases, in which case this may makes a difference.) 3. What are the possible values & default value for isolation? I think you can give only one key-value per --hiveconf option. 4. What is the default for numberFormat? BeeLineOpts.java says "default" but I don't know what that means. could you check what SQLLine doc says? 5. Is the description of the default for maxWidth okay? (Modified from SQLLine doc.) seems fine to me. 6. Does showNestedErrs have a JIRA ticket? (It isn't in the SQLLine doc.) not as I know of. 7. Should any of these options be documented? --incremental (in BeeLineOpts & SQLLine doc, but not in BeeLine.properties) --showElapsedTime (in BeeLineOpts, not in BeeLine.properties nor SQLLine doc) --maxHeight (ditto) --timeout (ditto) --trimScripts (ditto) --allowMultiLineCommand (ditto) SQLLine has a lot of properties, a little overwhelming. I'd think we should probably document only that's closely tied with Hive and refer user to SQLLine doc for others.
          Lefty Leverenz made changes -
          Labels TODOC13
          Hide
          Lefty Leverenz added a comment -

          Xuefu Zhang or any Beeline expert, would you please review the Beeline Command Options wikidoc (including --hiveconf for this jira) and answer at least some of the questions in my previous comment?

          Show
          Lefty Leverenz added a comment - Xuefu Zhang or any Beeline expert, would you please review the Beeline Command Options wikidoc (including --hiveconf for this jira) and answer at least some of the questions in my previous comment? Beeline Command Options
          Xuefu Zhang made changes -
          Link This issue is related to HIVE-6652 [ HIVE-6652 ]
          Hide
          Lefty Leverenz added a comment -

          I documented Beeline's command line options in the HiveServer2 Clients doc:

          This needs careful review and revision! My sources were BeeLine.properties, BeeLineOpts.java, the SQLLine doc, some Hive JIRAS, a bit of Java documentation, and common sense.

          Questions:

          1. Does the query string for -e take single quotes or double quotes?
          2. Is the autoCommit default false as shown in BeeLineOpts.java or true as shown in the SQLLine doc?
          3. What are the possible values & default value for isolation?
          4. What is the default for numberFormat? BeeLineOpts.java says "default" but I don't know what that means.
          5. Is the description of the default for maxWidth okay? (Modified from SQLLine doc.)
          6. Does showNestedErrs have a JIRA ticket? (It isn't in the SQLLine doc.)
          7. Should any of these options be documented?

          • --incremental (in BeeLineOpts & SQLLine doc, but not in BeeLine.properties)
          • --showElapsedTime (in BeeLineOpts, not in BeeLine.properties nor SQLLine doc)
          • --maxHeight (ditto)
          • --timeout (ditto)
          • --trimScripts (ditto)
          • --allowMultiLineCommand (ditto)

          Please fix the doc or tell me what to fix.

          Show
          Lefty Leverenz added a comment - I documented Beeline's command line options in the HiveServer2 Clients doc: Beeline Command Options This needs careful review and revision! My sources were BeeLine.properties, BeeLineOpts.java, the SQLLine doc, some Hive JIRAS, a bit of Java documentation, and common sense. Questions: 1. Does the query string for -e take single quotes or double quotes? 2. Is the autoCommit default false as shown in BeeLineOpts.java or true as shown in the SQLLine doc? 3. What are the possible values & default value for isolation? 4. What is the default for numberFormat? BeeLineOpts.java says "default" but I don't know what that means. 5. Is the description of the default for maxWidth okay? (Modified from SQLLine doc.) 6. Does showNestedErrs have a JIRA ticket? (It isn't in the SQLLine doc.) 7. Should any of these options be documented? --incremental (in BeeLineOpts & SQLLine doc, but not in BeeLine.properties) --showElapsedTime (in BeeLineOpts, not in BeeLine.properties nor SQLLine doc) --maxHeight (ditto) --timeout (ditto) --trimScripts (ditto) --allowMultiLineCommand (ditto) Please fix the doc or tell me what to fix.
          Xuefu Zhang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Release Note This (--hiveconf option) needs documentation if not being documented yet.
          Fix Version/s 0.13.0 [ 12324986 ]
          Resolution Fixed [ 1 ]
          Hide
          Xuefu Zhang added a comment -

          Patch committed to trunk. Thanks to Prasad for the review.

          Show
          Xuefu Zhang added a comment - Patch committed to trunk. Thanks to Prasad for the review.
          Hide
          Xuefu Zhang added a comment -

          I don't think the above failures has anything to do with this patch.

          Show
          Xuefu Zhang added a comment - I don't think the above failures has anything to do with this patch.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12626201/HIVE-6173.2.patch

          ERROR: -1 due to 2 failed/errored test(s), 4981 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1129/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1129/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12626201

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626201/HIVE-6173.2.patch ERROR: -1 due to 2 failed/errored test(s), 4981 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1129/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1129/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12626201
          Hide
          Prasad Mujumdar added a comment -

          +1
          Looks fine to me. Let's wait for the pre-commit test run.

          Show
          Prasad Mujumdar added a comment - +1 Looks fine to me. Let's wait for the pre-commit test run.
          Xuefu Zhang made changes -
          Attachment HIVE-6173.2.patch [ 12626201 ]
          Hide
          Xuefu Zhang added a comment -

          Patch #2 updated with the changes in HIVE-6217 removed.

          Show
          Xuefu Zhang added a comment - Patch #2 updated with the changes in HIVE-6217 removed.
          Xuefu Zhang made changes -
          Link This issue is required by HIVE-6340 [ HIVE-6340 ]
          Xuefu Zhang made changes -
          Link This issue is related to HIVE-6340 [ HIVE-6340 ]
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12625655/HIVE-6173.1.patch

          SUCCESS: +1 4973 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1108/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1108/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12625655

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12625655/HIVE-6173.1.patch SUCCESS: +1 4973 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1108/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1108/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12625655
          Xuefu Zhang made changes -
          Attachment HIVE-6173.1.patch [ 12625655 ]
          Hide
          Xuefu Zhang added a comment -

          Patch #1 includes patch for HIVE-6217.

          Show
          Xuefu Zhang added a comment - Patch #1 includes patch for HIVE-6217 .
          Xuefu Zhang made changes -
          Link This issue requires HIVE-6217 [ HIVE-6217 ]
          Hide
          Prasad Mujumdar added a comment -

          Hive supports a config parameter hive.conf.restricted.list for that purpose. It's a comma separated list of configs that can't be changed by users. Currently its empty by default, just restrict.list itself is implicitly added to it.

          Show
          Prasad Mujumdar added a comment - Hive supports a config parameter hive.conf.restricted.list for that purpose. It's a comma separated list of configs that can't be changed by users. Currently its empty by default, just restrict.list itself is implicitly added to it.
          Hide
          Navis added a comment -

          HiveServer can set hive.conf.restricted.list to prohibit overriding critical configurations. Currently empty for default but it should include some security/metastore related configs, imho.

          Show
          Navis added a comment - HiveServer can set hive.conf.restricted.list to prohibit overriding critical configurations. Currently empty for default but it should include some security/metastore related configs, imho.
          Hide
          Alan Gates added a comment -

          One question here is what options users should be able to set. Certain options set in the server shouldn't be changeable by users, such as the authorization provider. Is any work being done here to consider which values the user should and shouldn't be able to set?

          Show
          Alan Gates added a comment - One question here is what options users should be able to set. Certain options set in the server shouldn't be changeable by users, such as the authorization provider. Is any work being done here to consider which values the user should and shouldn't be able to set?
          Hide
          Prasad Mujumdar added a comment -

          Xuefu Zhang Thanks for the putting the patch out. I left some comments on the RB.

          Show
          Prasad Mujumdar added a comment - Xuefu Zhang Thanks for the putting the patch out. I left some comments on the RB.
          Hide
          Xuefu Zhang added a comment -
          Show
          Xuefu Zhang added a comment - RB: https://reviews.apache.org/r/16910/
          Xuefu Zhang made changes -
          Remote Link This issue links to "https://reviews.apache.org/r/16910/ (Web Link)" [ 13710 ]
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12622731/HIVE-6173.patch

          SUCCESS: +1 4925 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/896/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/896/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12622731

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12622731/HIVE-6173.patch SUCCESS: +1 4925 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/896/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/896/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12622731
          Xuefu Zhang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Xuefu Zhang made changes -
          Attachment HIVE-6173.patch [ 12622731 ]
          Hide
          Xuefu Zhang added a comment -

          [~vaibhavgumashta] Thanks for clarification. I guess the root cause of these issues are due to what's stated here. However, HIVE-5677 seems not realizing that, and the patch doesn't seem addressing the issue.

          Let me know if you'd like to work on this issue either via this JIRA or others.

          Show
          Xuefu Zhang added a comment - [~vaibhavgumashta] Thanks for clarification. I guess the root cause of these issues are due to what's stated here. However, HIVE-5677 seems not realizing that, and the patch doesn't seem addressing the issue. Let me know if you'd like to work on this issue either via this JIRA or others.
          Hide
          Vaibhav Gumashta added a comment -

          Xuefu Zhang Actually I had created HIVE-6142 to look into the hiveconf issue itself. But since that was marked duplicate of HIVE-5677, I closed it (I notice HIVE-5677 is a little different).

          Show
          Vaibhav Gumashta added a comment - Xuefu Zhang Actually I had created HIVE-6142 to look into the hiveconf issue itself. But since that was marked duplicate of HIVE-5677 , I closed it (I notice HIVE-5677 is a little different).
          Hide
          Xuefu Zhang added a comment -

          [~vaibhavgumashta] I assume that your duplication remark doesn't apply here.

          Show
          Xuefu Zhang added a comment - [~vaibhavgumashta] I assume that your duplication remark doesn't apply here.
          Vaibhav Gumashta made changes -
          Field Original Value New Value
          Link This issue duplicates HIVE-5677 [ HIVE-5677 ]
          Hide
          Vaibhav Gumashta added a comment -

          Infact HIVE-6142 is duplicate of: HIVE-5677, closed that.

          Show
          Vaibhav Gumashta added a comment - Infact HIVE-6142 is duplicate of: HIVE-5677 , closed that.
          Hide
          Vaibhav Gumashta added a comment -

          Duplicate of HIVE-6142

          Show
          Vaibhav Gumashta added a comment - Duplicate of HIVE-6142
          Xuefu Zhang created issue -

            People

            • Assignee:
              Xuefu Zhang
              Reporter:
              Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development