Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9725

Allow Variables for All Data Import Handler Data Source Configuration Values

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 5.5.3
    • Fix Version/s: 6.4, 7.0
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:

      Description

      I need to be able to use a variable for a password when also using encryptKeyFile.

      For instance:

      	<dataSource
      		driver="${custom.dataimporter.datasource.driver}"
      		url="${custom.dataimporter.datasource.url}"
      		user="${custom.dataimporter.datasource.user}"
      		password="${custom.dataimporter.datasource.password}"
      		encryptKeyFile="/opt/solr/credentials/encrypt.key"
      	/>
      

      Because I need to change certain variables based on the environment. I'd start like this:

           -a
            -Dcustom.dataimporter.datasource.driver=org.mariadb.jdbc.Driver
            -Dcustom.dataimporter.datasource.url=jdbc:mysql://local.mysite.com:3306/mysite
            -Dcustom.dataimporter.datasource.user=root
            -Dcustom.dataimporter.datasource.password=U2FsdGVkX1/dqwTb8RBfFq82SM37DkDRGeWMOndftHY=
      

      If I hardcode the password, it works; if I use a variable reference, it doesn't.

      As far as I know this pull request was submitted to address this issue, but it didn't come with a Jira ticket or a full explanation.

      Also, note that I'm not using a variable for the value of encryptKeyFile, because it's not possible in 5.x, though it seems to be fixed in 6.1. Presumably, the above patch would encompass encryptKeyFile's value, as well.

      1. SOLR-9725.patch
        6 kB
        Mikhail Khludnev

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jamiejackson commented on the issue:

          https://github.com/apache/lucene-solr/pull/46

          Please see https://issues.apache.org/jira/browse/SOLR-9725

          Show
          githubbot ASF GitHub Bot added a comment - Github user jamiejackson commented on the issue: https://github.com/apache/lucene-solr/pull/46 Please see https://issues.apache.org/jira/browse/SOLR-9725
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Hello,

          can we have a test for it?

          Show
          mkhludnev Mikhail Khludnev added a comment - Hello, can we have a test for it?
          Hide
          jamiejackson Jamie Jackson added a comment -

          Probably not from me, unless someone's prepared to do a lot of hand-holding. At a quick glance, it doesn't look too easy (for me, anyway) to add this test to TestSqlEntityProcessor.

          Show
          jamiejackson Jamie Jackson added a comment - Probably not from me, unless someone's prepared to do a lot of hand-holding. At a quick glance, it doesn't look too easy (for me , anyway) to add this test to TestSqlEntityProcessor .
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          The beautiful approach for testing was established at SOLR-8601. Why don't approach it here again?

          Show
          mkhludnev Mikhail Khludnev added a comment - The beautiful approach for testing was established at SOLR-8601 . Why don't approach it here again?
          Hide
          jamiejackson Jamie Jackson added a comment -

          That looks pretty digestible. I'll take a crack at it if I get a chance.

          Show
          jamiejackson Jamie Jackson added a comment - That looks pretty digestible. I'll take a crack at it if I get a chance.
          Hide
          jamiejackson Jamie Jackson added a comment - - edited

          I tried for a couple hours to get the project set up and to simply run a test suite, and I wasn't getting substantive answers on IRC. I don't have the time to spin my wheels just to get the project set up.

          Show
          jamiejackson Jamie Jackson added a comment - - edited I tried for a couple hours to get the project set up and to simply run a test suite , and I wasn't getting substantive answers on IRC. I don't have the time to spin my wheels just to get the project set up.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          That's sad. If you have a spare minute, would you mind to launch it with -verbose -debug ?

          Show
          mkhludnev Mikhail Khludnev added a comment - That's sad. If you have a spare minute, would you mind to launch it with -verbose -debug ?
          Hide
          jamiejackson Jamie Jackson added a comment -

          The command hangs here.

          Show
          jamiejackson Jamie Jackson added a comment - The command hangs here .
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jamiejackson opened a pull request:

          https://github.com/apache/lucene-solr/pull/112

          Allow an encrypted password to be used in a variable.

          https://issues.apache.org/jira/browse/SOLR-9725

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jamiejackson/lucene-solr allDihVars

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/lucene-solr/pull/112.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #112


          commit b8ff9a9e101e016d771b112a16575486a6cb0de2
          Author: Jamie Jackson <jamiejaxon@gmail.com>
          Date: 2016-11-10T22:19:31Z

          Allow an encrypted password to be used in a variable.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jamiejackson opened a pull request: https://github.com/apache/lucene-solr/pull/112 Allow an encrypted password to be used in a variable. https://issues.apache.org/jira/browse/SOLR-9725 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jamiejackson/lucene-solr allDihVars Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #112 commit b8ff9a9e101e016d771b112a16575486a6cb0de2 Author: Jamie Jackson <jamiejaxon@gmail.com> Date: 2016-11-10T22:19:31Z Allow an encrypted password to be used in a variable.
          Hide
          jamiejackson Jamie Jackson added a comment -

          I gave up on Windows; I can't get tests to run there.

          I tried on Linux and was able to run tests. This is what the output looks like on Linux. After the highlighted line is where it had been hanging in Windows.

          Show
          jamiejackson Jamie Jackson added a comment - I gave up on Windows; I can't get tests to run there. I tried on Linux and was able to run tests. This is what the output looks like on Linux. After the highlighted line is where it had been hanging in Windows.
          Hide
          jamiejackson Jamie Jackson added a comment - - edited

          I barely know what I'm doing here, but it's a combination of this and this.

          Notes:

          • I haven't actually tested it in situ (variables actually in a DIH config file, running a core), since I don't have time today to figure out building and configuring.
          • It targets branch_5_5, so I'm not sure the best way to get it up to master. Separate pull request?
          • testRetrieveFromJndiWithCredentialsEncryptedAndResolved() replaces testRetrieveFromJndiWithCredentialsWithEncryptedPwd(), as I didn't think the old name made as much sense anymore.
          • Master has testRetrieveFromJndiWithCredentialsWithEncryptedAndResolvedPwd() (note the trailing pwd). IMO, my new test will replace this one.
          Show
          jamiejackson Jamie Jackson added a comment - - edited I barely know what I'm doing here, but it's a combination of this and this . Notes: I haven't actually tested it in situ (variables actually in a DIH config file, running a core), since I don't have time today to figure out building and configuring. It targets branch_5_5 , so I'm not sure the best way to get it up to master. Separate pull request? testRetrieveFromJndiWithCredentialsEncryptedAndResolved() replaces testRetrieveFromJndiWithCredentialsWithEncryptedPwd() , as I didn't think the old name made as much sense anymore. Master has testRetrieveFromJndiWithCredentialsWithEncryptedAndResolvedPwd() (note the trailing pwd ). IMO, my new test will replace this one.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Ok. Here we go. The earlier patch didn't lay smoothly. Bill Bell, Jamie Jackson, Kristine Jetzke, please have a look to SOLR-9725.patch. I see that properties are passed to resolveVariables(), but I'm not sure whether or not they are populated from the system's ones.

          Show
          mkhludnev Mikhail Khludnev added a comment - Ok. Here we go. The earlier patch didn't lay smoothly. Bill Bell , Jamie Jackson , Kristine Jetzke , please have a look to SOLR-9725.patch . I see that properties are passed to resolveVariables(), but I'm not sure whether or not they are populated from the system's ones.
          Hide
          tinexw Kristine Jetzke added a comment -

          Mikhail Khludnev Do you mean with "lay smoothly" that the patch cannot be applied or that there is a problem with what it's doing (Sorry, I'm not a native speaker...) If it's the former: The patch only works against master, not against the 5x branch because only master contains SOLR-8610. What is the desired fix version anyway?

          Show
          tinexw Kristine Jetzke added a comment - Mikhail Khludnev Do you mean with "lay smoothly" that the patch cannot be applied or that there is a problem with what it's doing (Sorry, I'm not a native speaker...) If it's the former: The patch only works against master, not against the 5x branch because only master contains SOLR-8610 . What is the desired fix version anyway?
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I had difficulties in applying the patch proposed earlier, after reading your comment I might conclude that it was caused by the fact that it was generated on 5x. The rule of thumb is to commit to master and 6x by default. There will be probably no 5.x releases anyway. Are you ok with SOLR-9725.patch for master and 6x. Once again, 5x makes no sense to bother at all.

          Show
          mkhludnev Mikhail Khludnev added a comment - I had difficulties in applying the patch proposed earlier, after reading your comment I might conclude that it was caused by the fact that it was generated on 5x. The rule of thumb is to commit to master and 6x by default. There will be probably no 5.x releases anyway. Are you ok with SOLR-9725.patch for master and 6x. Once again, 5x makes no sense to bother at all.
          Hide
          tinexw Kristine Jetzke added a comment -

          Ah, ok, sounds good. I didn't check the actual content of the patch/didn't test the behavior though.

          Show
          tinexw Kristine Jetzke added a comment - Ah, ok, sounds good. I didn't check the actual content of the patch/didn't test the behavior though.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5c8a70fb57fd877e336c3d90a98ae10c5a6906ae in lucene-solr's branch refs/heads/master from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5c8a70f ]

          SOLR-9725: substitute properties in JdbcDataSource configuration

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5c8a70fb57fd877e336c3d90a98ae10c5a6906ae in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5c8a70f ] SOLR-9725 : substitute properties in JdbcDataSource configuration
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 5cda93afb2bb886780f1c68a8d0a2eaad42febca in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5cda93a ]

          SOLR-9725: substitute properties in JdbcDataSource configuration

          Show
          jira-bot ASF subversion and git services added a comment - Commit 5cda93afb2bb886780f1c68a8d0a2eaad42febca in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5cda93a ] SOLR-9725 : substitute properties in JdbcDataSource configuration
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I wonder why it hasn't been closed after 6.4 release..

          Show
          mkhludnev Mikhail Khludnev added a comment - I wonder why it hasn't been closed after 6.4 release..

            People

            • Assignee:
              mkhludnev Mikhail Khludnev
              Reporter:
              jamiejackson Jamie Jackson
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development