Solr
  1. Solr
  2. SOLR-4392

DIH - Need to externalize or encrypt username/password stored within data-config.xml

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0, 4.1
    • Fix Version/s: 5.2, 6.0
    • Labels:
      None

      Description

      Today, the connection (database or otherwise) credentials is wide open in data-config.xml. Not really an issue until someone sends out the config file outside of the server.

      We should look into externalizing the database lookup or providing a way to encrypt the username and password.

      The needs are:

      1/ Some projects want to enable multi-tenancy where data for each core is situated in different database servers w/ their own credentials. We need a way to expose hooks that will allow implementations to be plugged in. It can be done though the "type" attribute on the dataSource, but providing a factory might work better.

      2/ Most orgs are very protective of their credentials and weary of plain-text settings.

      <dataSource name="jdbc" driver="oracle.jdbc.driver.OracleDriver" url="jdbc:oracle:thin:@//hostname:port/SID" user="db_username" 
      <!-- This database password is encrypted using AES using the command. pwd.txt contains the actual DB password -->
      <!-- openssl enc -aes-128-cbc -a -salt -in pwd.txt -->
      password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o=" 
      <!-- Password to decrypt is stored in this file-->
      encryptKeyFile="/location/of/encryptionkey"
      />
      
      1. SOLR-4392.patch
        8 kB
        Noble Paul
      2. SOLR-4392.patch
        7 kB
        Senthuran Sivananthan

        Issue Links

          Activity

          Hide
          Senthuran Sivananthan added a comment -

          Initial patch that adds a jdbc data source factory.

          We now just need to implement the various encryption algorithms.

          I'll work on a quick DES and AES impls, next.

          Show
          Senthuran Sivananthan added a comment - Initial patch that adds a jdbc data source factory. We now just need to implement the various encryption algorithms. I'll work on a quick DES and AES impls, next.
          Hide
          Ken Geis added a comment -

          I think that the ability to use a JNDI data source makes this a non-issue.

          Show
          Ken Geis added a comment - I think that the ability to use a JNDI data source makes this a non-issue.
          Hide
          Noble Paul added a comment -

          What is stopping the user from extending JdbcDataSource and use it directly and plug in a way to read your passwords

          Show
          Noble Paul added a comment - What is stopping the user from extending JdbcDataSource and use it directly and plug in a way to read your passwords
          Hide
          Noble Paul added a comment -

          Ideally I would like it as follows

          <dataSource type="JdbcDataSource" user="x" passwordEncrpt="MIIBOgIBAAJBALsT+DLgE4qGfYc3K7JRnbPS2dlpRvC6v8j" encryptkey="/location/to/my/key"/>
          

          instead of giving too many options I would like to just use AES256 for encryption . And you can store the encryption key somewhere in the file systrem

          Show
          Noble Paul added a comment - Ideally I would like it as follows <dataSource type= "JdbcDataSource" user= "x" passwordEncrpt= "MIIBOgIBAAJBALsT+DLgE4qGfYc3K7JRnbPS2dlpRvC6v8j" encryptkey= "/location/to/my/key" /> instead of giving too many options I would like to just use AES256 for encryption . And you can store the encryption key somewhere in the file systrem
          Hide
          Noble Paul added a comment -

          Another solution is to do a two pass request. Using a PKI solution

          • Send a GET request to DIH and it would give back a public key of a key pair it generated internally and stored in-memory
          • send the password, encrypted with the public key as a param .

          This assumes that , the Solr server did not get restarted in between

          Show
          Noble Paul added a comment - Another solution is to do a two pass request. Using a PKI solution Send a GET request to DIH and it would give back a public key of a key pair it generated internally and stored in-memory send the password, encrypted with the public key as a param . This assumes that , the Solr server did not get restarted in between
          Hide
          Noble Paul added a comment -

          Use aes-128 to encrypt password

          Show
          Noble Paul added a comment - Use aes-128 to encrypt password
          Hide
          ASF subversion and git services added a comment -

          Commit 1678195 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1678195 ]

          SOLR-4392: Make it possible to specify AES encrypted password in dataconfig.xml

          Show
          ASF subversion and git services added a comment - Commit 1678195 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1678195 ] SOLR-4392 : Make it possible to specify AES encrypted password in dataconfig.xml
          Hide
          ASF subversion and git services added a comment -

          Commit 1678242 from hossman@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1678242 ]

          SOLR-4392: revert r1678195 which breaks forbidden-api

          Show
          ASF subversion and git services added a comment - Commit 1678242 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1678242 ] SOLR-4392 : revert r1678195 which breaks forbidden-api
          Hide
          ASF subversion and git services added a comment -

          Commit 1678250 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1678250 ]

          SOLR-4392: Make it possible to specify AES encrypted password in dataconfig.xml

          Show
          ASF subversion and git services added a comment - Commit 1678250 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1678250 ] SOLR-4392 : Make it possible to specify AES encrypted password in dataconfig.xml
          Hide
          ASF subversion and git services added a comment -

          Commit 1678342 from Noble Paul in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1678342 ]

          SOLR-4392: Make it possible to specify AES encrypted password in dataconfig.xml

          Show
          ASF subversion and git services added a comment - Commit 1678342 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1678342 ] SOLR-4392 : Make it possible to specify AES encrypted password in dataconfig.xml
          Hide
          Anshum Gupta added a comment -

          Bulk close for 5.2.0.

          Show
          Anshum Gupta added a comment - Bulk close for 5.2.0.
          Hide
          Shawn Heisey added a comment -

          There is some minimal documentation here in the issue for how to modify the DIH config, but no information in the reference guide at all that I can find. The information here does not explain how to properly encrypt the password or create the encryptKeyFile. The example encryptKeyFile location is absolute, so it is not clear what happens with a relative path.

          Show
          Shawn Heisey added a comment - There is some minimal documentation here in the issue for how to modify the DIH config, but no information in the reference guide at all that I can find. The information here does not explain how to properly encrypt the password or create the encryptKeyFile. The example encryptKeyFile location is absolute, so it is not clear what happens with a relative path.
          Hide
          Shawn Heisey added a comment -

          I've been trying everything I can think of and what I can find on the Internet about encrypting strings with aes256, and nothing has worked so far. Should I open a new issue?

          Show
          Shawn Heisey added a comment - I've been trying everything I can think of and what I can find on the Internet about encrypting strings with aes256, and nothing has worked so far. Should I open a new issue?
          Hide
          Shawn Heisey added a comment -

          Finally, success, using aes-128-cbc, and most critically, the key file must NOT have eol (LF in Linux).

          Show
          Shawn Heisey added a comment - Finally, success, using aes-128-cbc, and most critically, the key file must NOT have eol (LF in Linux).
          Hide
          Shawn Heisey added a comment -

          If you create a key file with vi from scratch, you will get the EOL character, and that will break the encrypted password. Would it be difficult to get this working with a standard text file, where the line might be terminated with either LF or CRLF?

          Show
          Shawn Heisey added a comment - If you create a key file with vi from scratch, you will get the EOL character, and that will break the encrypted password. Would it be difficult to get this working with a standard text file, where the line might be terminated with either LF or CRLF?
          Hide
          Noble Paul added a comment -

          we can fix that. Just open a new ticket

          Show
          Noble Paul added a comment - we can fix that. Just open a new ticket
          Hide
          Pedro Mendes added a comment -

          Hi,

          Besides the documentation here in the issue for how to modify the DIH config, where can I find additional information on how to encrypt the password or create the encryptKeyFile?

          Does anybody have a working example?

          I'd be very gratefull, thanks

          Show
          Pedro Mendes added a comment - Hi, Besides the documentation here in the issue for how to modify the DIH config, where can I find additional information on how to encrypt the password or create the encryptKeyFile? Does anybody have a working example? I'd be very gratefull, thanks
          Hide
          Alessandro Benedetti added a comment -

          +1 , is it possible to add the documentation related in the solr wiki ?

          Show
          Alessandro Benedetti added a comment - +1 , is it possible to add the documentation related in the solr wiki ?
          Hide
          Noble Paul added a comment -

          Will do

          Show
          Noble Paul added a comment - Will do
          Hide
          Alessandro Benedetti added a comment -

          Thanks Paul, anyway , it is clear how it works.
          It is easy to configure it, but would be great to have it in the official documentation only to officially show this feature ( in this way will be easier for a user to understand that the password externalisation is something possible, without even accessing lira).

          Thanks !

          Show
          Alessandro Benedetti added a comment - Thanks Paul, anyway , it is clear how it works. It is easy to configure it, but would be great to have it in the official documentation only to officially show this feature ( in this way will be easier for a user to understand that the password externalisation is something possible, without even accessing lira). Thanks !
          Hide
          Aniket Khare added a comment -

          Could you please share the working example or the documentation for the data config encryption?

          Show
          Aniket Khare added a comment - Could you please share the working example or the documentation for the data config encryption?
          Hide
          Jamie Jackson added a comment -

          As far as I know, the only place where encrypted passwords are documented is in https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler, under the "Configuring the DIH Configuration File", in a comment of a sample XML file:

          <!--
          Alternately the password can be encrypted as follows. This is the value obtained as a result of the command
          openssl enc -aes-128-cbc -a -salt -in pwd.txt
          password="U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o=" 
          WHen the password is encrypted, you must provide an extra attribute
          encryptKeyFile="/location/of/encryptionkey"
          This file should a text file with a single line containing the encrypt/decrypt password
          -->
          

          Unfortunately, I can't get it to work.

          Show
          Jamie Jackson added a comment - As far as I know, the only place where encrypted passwords are documented is in https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler , under the "Configuring the DIH Configuration File", in a comment of a sample XML file: <!-- Alternately the password can be encrypted as follows. This is the value obtained as a result of the command openssl enc -aes-128-cbc -a -salt -in pwd.txt password= "U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o=" WHen the password is encrypted, you must provide an extra attribute encryptKeyFile= "/location/of/encryptionkey" This file should a text file with a single line containing the encrypt/decrypt password --> Unfortunately, I can't get it to work .

            People

            • Assignee:
              Noble Paul
              Reporter:
              Senthuran Sivananthan
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development