Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7175

Provide password file option to beeline

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 1.2.0
    • Component/s: CLI, Clients
    • Labels:
    • Release Note:
      Added an --password-file (or, -w) option to BeeLine CLI, to read a password from a permission-protected file instead of supplying it in plaintext form as part of the command (-p).

      Description

      For people connecting to Hive Server 2 with LDAP authentication enabled, in order to batch run commands, we currently have to provide the password openly in the command line. They could use some expect scripting, but I think a valid improvement would be to provide a password file option similar to other CLI commands in hadoop (e.g. sqoop) to be more secure.

      1. HIVE-7175.2.patch
        5 kB
        Vaibhav Gumashta
      2. HIVE-7175.1.patch
        5 kB
        Vaibhav Gumashta
      3. HIVE-7175.branch-13.patch
        2 kB
        Vaibhav Gumashta
      4. HIVE-7175.1.patch
        5 kB
        Vaibhav Gumashta
      5. HIVE-7175.patch
        3 kB
        Dr. Wendell Urth

        Issue Links

          Activity

          Show
          thejas Thejas M Nair added a comment - Documented in the wiki page mentioned by Lefty Leverenz - https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=30758725&selectedPageVersions=138&selectedPageVersions=136
          Hide
          sushanth Sushanth Sowmyan added a comment -

          This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          sushanth Sushanth Sowmyan added a comment - This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
          Hide
          leftylev Lefty Leverenz added a comment -

          Doc note: This should be documented in the Beeline section of HiveServer2 Clients, with version information for 1.2.0 and a link to this issue for the 0.13 patch.

          Show
          leftylev Lefty Leverenz added a comment - Doc note: This should be documented in the Beeline section of HiveServer2 Clients, with version information for 1.2.0 and a link to this issue for the 0.13 patch. HiveServer2 Clients – Beeline Command Options
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Patch committed to trunk. Thanks Dr. Wendell Urth for the patch, thanks Larry McCay, Robert Justice and Thejas M Nair for reviewing.

          Show
          vgumashta Vaibhav Gumashta added a comment - Patch committed to trunk. Thanks Dr. Wendell Urth for the patch, thanks Larry McCay , Robert Justice and Thejas M Nair for reviewing.
          Hide
          rjustice Robert Justice added a comment -

          Thanks for helping on this improvement to beeline: Dr. Wendell Urth, Vaibhav Gumashta, and Larry McCay

          Show
          rjustice Robert Justice added a comment - Thanks for helping on this improvement to beeline: Dr. Wendell Urth, Vaibhav Gumashta, and Larry McCay
          Hide
          hiveqa Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12696544/HIVE-7175.2.patch

          SUCCESS: +1 7476 tests passed

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2666/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2666/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2666/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12696544 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12696544/HIVE-7175.2.patch SUCCESS: +1 7476 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2666/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2666/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2666/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12696544 - PreCommit-HIVE-TRUNK-Build
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12696265/HIVE-7175.1.patch

          ERROR: -1 due to 2 failed/errored test(s), 7423 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
          org.apache.hive.beeline.TestBeelineArgParsing.testPasswordFileArgs
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2644/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2644/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2644/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12696265 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12696265/HIVE-7175.1.patch ERROR: -1 due to 2 failed/errored test(s), 7423 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23 org.apache.hive.beeline.TestBeelineArgParsing.testPasswordFileArgs Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2644/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2644/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2644/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12696265 - PreCommit-HIVE-TRUNK-Build
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Reuploading for precommit run.

          Show
          vgumashta Vaibhav Gumashta added a comment - Reuploading for precommit run.
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Patch for 0.13.0 in case anyone needs it.

          Show
          vgumashta Vaibhav Gumashta added a comment - Patch for 0.13.0 in case anyone needs it.
          Hide
          thejas Thejas M Nair added a comment -
          Show
          thejas Thejas M Nair added a comment - +1 Thanks Dr. Wendell Urth and Vaibhav Gumashta !
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Patch based on trunk.

          cc Thejas M Nair

          Show
          vgumashta Vaibhav Gumashta added a comment - Patch based on trunk. cc Thejas M Nair
          Hide
          vgumashta Vaibhav Gumashta added a comment -

          Dr. Wendell Urth Thanks for the work so far. Do you plan to incorporate the follow-up feedback? Let me know if you don't have free cycles - I can pitch in.

          Show
          vgumashta Vaibhav Gumashta added a comment - Dr. Wendell Urth Thanks for the work so far. Do you plan to incorporate the follow-up feedback? Let me know if you don't have free cycles - I can pitch in.
          Hide
          lmccay Larry McCay added a comment -

          I just realized that this is the users' LDAP password.
          It would be unfortunate to have to leave this laying around in various places unless absolutely necessary.

          Does the beeline CLI currently allow for using the java Console to collect the password from the user?

          I understand that for scripting type purposes we may need another collection mechanism but for usecases with a user and console available the users' passwords should not be persisted outside of the directory itself when it can be avoided.

          For cases where it can not be avoided the side file approach is certainly better than on the command line itself in terms of visibility.

          Show
          lmccay Larry McCay added a comment - I just realized that this is the users' LDAP password. It would be unfortunate to have to leave this laying around in various places unless absolutely necessary. Does the beeline CLI currently allow for using the java Console to collect the password from the user? I understand that for scripting type purposes we may need another collection mechanism but for usecases with a user and console available the users' passwords should not be persisted outside of the directory itself when it can be avoided. For cases where it can not be avoided the side file approach is certainly better than on the command line itself in terms of visibility.
          Hide
          brocknoland Brock Noland added a comment -

          Also could you add a nice message given an exception in obtainPasswordFromFile.

          Show
          brocknoland Brock Noland added a comment - Also could you add a nice message given an exception in obtainPasswordFromFile.
          Hide
          xuefuz Xuefu Zhang added a comment -

          [~dr.wendell.urth] Thanks for working on this. Could you please provide a review board entry for the patch?

          Show
          xuefuz Xuefu Zhang added a comment - [~dr.wendell.urth] Thanks for working on this. Could you please provide a review board entry for the patch?
          Hide
          wendell.urth Dr. Wendell Urth added a comment -

          Hi Hive QA, none of the failed tests appear related to the small additive change specific to BeeLine done here. These tests appear to be generally failing on trunk, and are not caused by this patch. Let me know if I am wrong.

          Show
          wendell.urth Dr. Wendell Urth added a comment - Hi Hive QA , none of the failed tests appear related to the small additive change specific to BeeLine done here. These tests appear to be generally failing on trunk, and are not caused by this patch. Let me know if I am wrong.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12648646/HIVE-7175.patch

          ERROR: -1 due to 7 failed/errored test(s), 5511 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
          org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/399/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/399/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-399/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 7 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12648646

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12648646/HIVE-7175.patch ERROR: -1 due to 7 failed/errored test(s), 5511 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/399/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/399/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-399/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12648646
          Hide
          wendell.urth Dr. Wendell Urth added a comment -

          I've added a patch that provides this ability akin to Sqoop's mechanism (minus the encrypted/obfuscated file loader options, as those could be better handled by Larry's proposal).

          This would be useful in the immediate future, until what Larry proposes can be compatibly added to Hive in future upon completion upstream.

          Please review.

          Show
          wendell.urth Dr. Wendell Urth added a comment - I've added a patch that provides this ability akin to Sqoop's mechanism (minus the encrypted/obfuscated file loader options, as those could be better handled by Larry's proposal). This would be useful in the immediate future, until what Larry proposes can be compatibly added to Hive in future upon completion upstream. Please review.
          Hide
          lmccay Larry McCay added a comment -

          These should be seen as complementary issues.

          Show
          lmccay Larry McCay added a comment - These should be seen as complementary issues.
          Hide
          rjustice Robert Justice added a comment -

          Great idea Larry McCay. I'm open for whatever solution is the most secure for users. Some users will require the password file to be encrypted or in a 3rd party store, based on security policies and this would be a solution to that.

          Show
          rjustice Robert Justice added a comment - Great idea Larry McCay . I'm open for whatever solution is the most secure for users. Some users will require the password file to be encrypted or in a 3rd party store, based on security policies and this would be a solution to that.
          Hide
          lmccay Larry McCay added a comment -

          Hi Robert Justice - we may want to consider the use of the CredentialProvider API that will be committed soon.
          See HADOOP-10607. This isn't mutually exclusive with the password file approach as there are plans to fallback to existing password files in certain components. However, the abstraction of the API is best realized through the new Configuration.getPassword(String name) method. This will allow you to ask for a configuration item that you know is a password and it will check for an aliased credential based on the name through the CredentialProvider API. If the name is not resolved into a credential from a provider then it falls back to the config file.

          The extra hop of the separate file isn't a problem but it isn't encapsulated by the getPassword method going into Configuration.

          Just something to keep in mind.

          Show
          lmccay Larry McCay added a comment - Hi Robert Justice - we may want to consider the use of the CredentialProvider API that will be committed soon. See HADOOP-10607 . This isn't mutually exclusive with the password file approach as there are plans to fallback to existing password files in certain components. However, the abstraction of the API is best realized through the new Configuration.getPassword(String name) method. This will allow you to ask for a configuration item that you know is a password and it will check for an aliased credential based on the name through the CredentialProvider API. If the name is not resolved into a credential from a provider then it falls back to the config file. The extra hop of the separate file isn't a problem but it isn't encapsulated by the getPassword method going into Configuration. Just something to keep in mind.

            People

            • Assignee:
              wendell.urth Dr. Wendell Urth
              Reporter:
              rjustice Robert Justice
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development