Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9997

Enable configuring SolrHttpClientBuilder via java system property

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.3
    • Fix Version/s: 6.5, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Currently SolrHttpClientBuilder needs to be configured via invoking HttpClientUtil#setHttpClientBuilder(...) API. On the other hand SolrCLI attempts to support configuring SolrHttpClientBuilder via Java system property. https://github.com/apache/lucene-solr/blob/9f58b6cd177f72b226c83adbb965cfe08d61d2fb/solr/core/src/java/org/apache/solr/util/SolrCLI.java#L265

      But after changes for SOLR-4509, this is no longer working. This is because we need to configure HttpClientBuilderFactory which can provide appropriate SolrHttpClientBuilder instance (e.g. Krb5HttpClientBuilder). I verified that SolrCLI does not work in a kerberos enabled cluster. During the testing I also found that SolrCLI is hardcoded to use basic authentication,

      https://github.com/apache/lucene-solr/blob/9f58b6cd177f72b226c83adbb965cfe08d61d2fb/solr/core/src/java/org/apache/solr/util/SolrCLI.java#L156

      This jira is to add support for configuring HttpClientBuilderFactory as a java system property so that SolrCLI as well as other Solr clients can also benefit this. Also we should provide a HttpClientBuilderFactory which support configuring preemptive basic authentication so that we can remove the hardcoded basic auth usage in SolrCLI (and enable it work with kerberos).

      1. SOLR-9997_6x.patch
        25 kB
        Hrishikesh Gadre

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user hgadre opened a pull request:

          https://github.com/apache/lucene-solr/pull/140

          SOLR-9997 Enable configuring SolrHttpClientBuilder via java system …

          …property

          • Added support to configure SolrHttpClientBuilder via java system property
          • Added a concrete implementation of HttpClientBuilderFactory which supports
            preemptive basic authentication.
          • Removed the hardcoded basic auth support from SolrCLI (and verified that
            it works with basic auth plugin as well as with kerberos).
          • Added support to log HTTP request/response for debugging authentication issues
            in HadoopAuthPlugin.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/hgadre/lucene-solr SOLR-9997_fix

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/lucene-solr/pull/140.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #140


          commit 852668b0c7977aea9d73541e7d36e74e30165079
          Author: Hrishikesh Gadre <hgadre@cloudera.com>
          Date: 2017-01-18T20:31:20Z

          SOLR-9997 Enable configuring SolrHttpClientBuilder via java system property

          • Added support to configure SolrHttpClientBuilder via java system property
          • Added a concrete implementation of HttpClientBuilderFactory which supports
            preemptive basic authentication.
          • Removed the hardcoded basic auth support from SolrCLI (and verified that
            it works with basic auth plugin as well as with kerberos).
          • Added support to log HTTP request/response for debugging authentication issues
            in HadoopAuthPlugin.

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user hgadre opened a pull request: https://github.com/apache/lucene-solr/pull/140 SOLR-9997 Enable configuring SolrHttpClientBuilder via java system … …property Added support to configure SolrHttpClientBuilder via java system property Added a concrete implementation of HttpClientBuilderFactory which supports preemptive basic authentication. Removed the hardcoded basic auth support from SolrCLI (and verified that it works with basic auth plugin as well as with kerberos). Added support to log HTTP request/response for debugging authentication issues in HadoopAuthPlugin. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hgadre/lucene-solr SOLR-9997 _fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/140.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #140 commit 852668b0c7977aea9d73541e7d36e74e30165079 Author: Hrishikesh Gadre <hgadre@cloudera.com> Date: 2017-01-18T20:31:20Z SOLR-9997 Enable configuring SolrHttpClientBuilder via java system property Added support to configure SolrHttpClientBuilder via java system property Added a concrete implementation of HttpClientBuilderFactory which supports preemptive basic authentication. Removed the hardcoded basic auth support from SolrCLI (and verified that it works with basic auth plugin as well as with kerberos). Added support to log HTTP request/response for debugging authentication issues in HadoopAuthPlugin.
          Hide
          hgadre Hrishikesh Gadre added a comment - - edited

          I am running into a weird issue while running precommit,

          BUILD FAILED
          /Users/hgadre/git-repo/upstream/lucene-solr/build.xml:117: The following error occurred while executing this line:
          /Users/hgadre/git-repo/upstream/lucene-solr/solr/build.xml:349: The following error occurred while executing this line:
          /Users/hgadre/git-repo/upstream/lucene-solr/lucene/tools/custom-tasks.xml:62: JAR resource does not exist: core/test-lib/mockito-core-1.9.5.jar

          I don't know the cause yet. Will investigate tomorrow. Have anyone seen this yet?

          Show
          hgadre Hrishikesh Gadre added a comment - - edited I am running into a weird issue while running precommit, BUILD FAILED /Users/hgadre/git-repo/upstream/lucene-solr/build.xml:117: The following error occurred while executing this line: /Users/hgadre/git-repo/upstream/lucene-solr/solr/build.xml:349: The following error occurred while executing this line: /Users/hgadre/git-repo/upstream/lucene-solr/lucene/tools/custom-tasks.xml:62: JAR resource does not exist: core/test-lib/mockito-core-1.9.5.jar I don't know the cause yet. Will investigate tomorrow. Have anyone seen this yet?
          Hide
          hgadre Hrishikesh Gadre added a comment - - edited

          Ishan Chattopadhyaya Jan Høydahl can you please review the patch?

          Show
          hgadre Hrishikesh Gadre added a comment - - edited Ishan Chattopadhyaya Jan Høydahl can you please review the patch?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user janhoy commented on a diff in the pull request:

          https://github.com/apache/lucene-solr/pull/140#discussion_r96840773

          — Diff: solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java —
          @@ -111,28 +112,46 @@
          // cannot be established within x ms. with a
          // java.net.SocketTimeoutException: Connection timed out
          public static final String PROP_CONNECTION_TIMEOUT = "connTimeout";

          • +
            + /**
            + * A Java system property to select the

            {@linkplain HttpClientBuilderFactory}

            used for
            + * configuring the

            {@linkplain HttpClientBuilder}

            instance by default.
            + */
            + public static final String SYS_PROP_HTTP_CLIENT_BUILDER_FACTORY = "solr.httpclient.builder.factory";
            +
            static final DefaultHttpRequestRetryHandler NO_RETRY = new DefaultHttpRequestRetryHandler(
            0, false);

          private static volatile SolrHttpClientBuilder httpClientBuilder;

          • +
            private static SolrHttpClientContextBuilder httpClientRequestContextBuilder = new SolrHttpClientContextBuilder();

          • +
            + private static volatile SchemaRegistryProvider schemaRegistryProvider;
            + private static volatile String cookiePolicy;
            + private static final List<HttpRequestInterceptor> interceptors = Collections.synchronizedList(new ArrayList<HttpRequestInterceptor>());
            +
            +
            static {
            resetHttpClientBuilder();
            +
            + // Configure the HttpClientBuilder if user has specified the factory type.
            + String factoryClassName = System.getProperty(SYS_PROP_HTTP_CLIENT_BUILDER_FACTORY);
            + if (factoryClassName != null) {
            + logger.info ("Using " + factoryClassName);

              • End diff –

          use logger.debug

          Show
          githubbot ASF GitHub Bot added a comment - Github user janhoy commented on a diff in the pull request: https://github.com/apache/lucene-solr/pull/140#discussion_r96840773 — Diff: solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java — @@ -111,28 +112,46 @@ // cannot be established within x ms. with a // java.net.SocketTimeoutException: Connection timed out public static final String PROP_CONNECTION_TIMEOUT = "connTimeout"; + + /** + * A Java system property to select the {@linkplain HttpClientBuilderFactory} used for + * configuring the {@linkplain HttpClientBuilder} instance by default. + */ + public static final String SYS_PROP_HTTP_CLIENT_BUILDER_FACTORY = "solr.httpclient.builder.factory"; + static final DefaultHttpRequestRetryHandler NO_RETRY = new DefaultHttpRequestRetryHandler( 0, false); private static volatile SolrHttpClientBuilder httpClientBuilder; + private static SolrHttpClientContextBuilder httpClientRequestContextBuilder = new SolrHttpClientContextBuilder(); + + private static volatile SchemaRegistryProvider schemaRegistryProvider; + private static volatile String cookiePolicy; + private static final List<HttpRequestInterceptor> interceptors = Collections.synchronizedList(new ArrayList<HttpRequestInterceptor>()); + + static { resetHttpClientBuilder(); + + // Configure the HttpClientBuilder if user has specified the factory type. + String factoryClassName = System.getProperty(SYS_PROP_HTTP_CLIENT_BUILDER_FACTORY); + if (factoryClassName != null) { + logger.info ("Using " + factoryClassName); End diff – use logger.debug
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hgadre commented on the issue:

          https://github.com/apache/lucene-solr/pull/140

          @janhoy Thanks for the review. We will need to update ref guide with instructions for configuring SolrCLI against the secure SOLR cluster. e.g. with the basic auth plugin, following env variable needs to be configured,

          export SOLR_AUTHENTICATION_CLIENT_BUILDER="org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory"

          We can make this nicer e.g. by adding a command-line param such as --use-basic-auth. This way the configuration of actual factory type can be hidden inside the script.

          With the latest changes, the pre-commit is also passing now.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hgadre commented on the issue: https://github.com/apache/lucene-solr/pull/140 @janhoy Thanks for the review. We will need to update ref guide with instructions for configuring SolrCLI against the secure SOLR cluster. e.g. with the basic auth plugin, following env variable needs to be configured, export SOLR_AUTHENTICATION_CLIENT_BUILDER="org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" We can make this nicer e.g. by adding a command-line param such as --use-basic-auth. This way the configuration of actual factory type can be hidden inside the script. With the latest changes, the pre-commit is also passing now.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user janhoy commented on the issue:

          https://github.com/apache/lucene-solr/pull/140

          I like the --use-basic-auth convenience, or should we defailt to basic if no builder is specified (like it kind of is today)?

          Show
          githubbot ASF GitHub Bot added a comment - Github user janhoy commented on the issue: https://github.com/apache/lucene-solr/pull/140 I like the --use-basic-auth convenience, or should we defailt to basic if no builder is specified (like it kind of is today)?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hgadre commented on the issue:

          https://github.com/apache/lucene-solr/pull/140

          @janhoy I think adding --use-basic-auth would be future proof as we add newer authentication mechanisms. We can add --use-kerberos <path_to_jaas_conf> as well for completeness. Let me take a look and update the patch accordingly.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hgadre commented on the issue: https://github.com/apache/lucene-solr/pull/140 @janhoy I think adding --use-basic-auth would be future proof as we add newer authentication mechanisms. We can add --use-kerberos <path_to_jaas_conf> as well for completeness. Let me take a look and update the patch accordingly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hgadre commented on the issue:

          https://github.com/apache/lucene-solr/pull/140

          @janhoy I found it difficult to add command-line parameters without extensive refactoring of scripts. Hence as an intermediate solution - I have defined an environment variable SOLR_AUTH_TYPE which can be configured with the scheme to be used (e.g. basic, kerberos etc). The script takes care of configuring the appropriate Solr HttpClientBuilderFactory. Please take a look and let me know your feedback.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hgadre commented on the issue: https://github.com/apache/lucene-solr/pull/140 @janhoy I found it difficult to add command-line parameters without extensive refactoring of scripts. Hence as an intermediate solution - I have defined an environment variable SOLR_AUTH_TYPE which can be configured with the scheme to be used (e.g. basic, kerberos etc). The script takes care of configuring the appropriate Solr HttpClientBuilderFactory. Please take a look and let me know your feedback.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user janhoy commented on the issue:

          https://github.com/apache/lucene-solr/pull/140

          I like it even better Let's add a comment to solr.in.sh that you can *either* specify `SOLR_AUTHENTICATION_CLIENT_BUILDER` or `SOLR_AUTH_TYPE`, and perhaps give a reference to the RefGuide chapter with full docs? Should we also try to detect if both BUILDER and AUTH_TYPE is specified and bail out?

          Show
          githubbot ASF GitHub Bot added a comment - Github user janhoy commented on the issue: https://github.com/apache/lucene-solr/pull/140 I like it even better Let's add a comment to solr.in.sh that you can * either * specify `SOLR_AUTHENTICATION_CLIENT_BUILDER` or `SOLR_AUTH_TYPE`, and perhaps give a reference to the RefGuide chapter with full docs? Should we also try to detect if both BUILDER and AUTH_TYPE is specified and bail out?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hgadre commented on the issue:

          https://github.com/apache/lucene-solr/pull/140

          @janhoy Done! Please take a look and let me know your feedback.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hgadre commented on the issue: https://github.com/apache/lucene-solr/pull/140 @janhoy Done! Please take a look and let me know your feedback.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hgadre commented on the issue:

          https://github.com/apache/lucene-solr/pull/140

          @janhoy Now that we have proper support for configuring basic auth credentials, we should also consider deprecating following logic,

          https://github.com/apache/lucene-solr/blob/1b80691f28b045c7a8d9552f3c63f7bafdf52d48/solr/solrj/src/java/org/apache/solr/client/solrj/SolrRequest.java#L50

          Show
          githubbot ASF GitHub Bot added a comment - Github user hgadre commented on the issue: https://github.com/apache/lucene-solr/pull/140 @janhoy Now that we have proper support for configuring basic auth credentials, we should also consider deprecating following logic, https://github.com/apache/lucene-solr/blob/1b80691f28b045c7a8d9552f3c63f7bafdf52d48/solr/solrj/src/java/org/apache/solr/client/solrj/SolrRequest.java#L50
          Hide
          hgadre Hrishikesh Gadre added a comment -

          Jan Høydahl Could you please review the latest patch whenever you get a chance?

          Show
          hgadre Hrishikesh Gadre added a comment - Jan Høydahl Could you please review the latest patch whenever you get a chance?
          Hide
          janhoy Jan Høydahl added a comment -

          You mean the latest version of the PR? Looks fine to me, but I have not tested it yet.

          Show
          janhoy Jan Høydahl added a comment - You mean the latest version of the PR? Looks fine to me, but I have not tested it yet.
          Hide
          hgadre Hrishikesh Gadre added a comment -

          Jan Høydahl Thanks for the review. I have already tested it on MacOS and Windows. Please let me know if anything required from my side to move this forward.

          Show
          hgadre Hrishikesh Gadre added a comment - Jan Høydahl Thanks for the review. I have already tested it on MacOS and Windows. Please let me know if anything required from my side to move this forward.
          Hide
          janhoy Jan Høydahl added a comment -

          I cannot look at it this week or the next, so anyone else feel free to grab it

          Show
          janhoy Jan Høydahl added a comment - I cannot look at it this week or the next, so anyone else feel free to grab it
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Is this just for 7x or 6x too?

          Show
          markrmiller@gmail.com Mark Miller added a comment - Is this just for 7x or 6x too?
          Hide
          hgadre Hrishikesh Gadre added a comment -

          Mark Miller This patch will work only for 7x. Let me upload a patch for 6x.

          Show
          hgadre Hrishikesh Gadre added a comment - Mark Miller This patch will work only for 7x. Let me upload a patch for 6x.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          Yeah, I know it's quite different on 6x vs 7x, I was just asking intention so I can place it correctly in CHANGES.

          Show
          markrmiller@gmail.com Mark Miller added a comment - Yeah, I know it's quite different on 6x vs 7x, I was just asking intention so I can place it correctly in CHANGES.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e1a577645756addc8ed060024e4af7ccd15c2321 in lucene-solr's branch refs/heads/master from markrmiller
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e1a5776 ]

          SOLR-9997: Enable configuring SolrHttpClientBuilder via java system property.

          Show
          jira-bot ASF subversion and git services added a comment - Commit e1a577645756addc8ed060024e4af7ccd15c2321 in lucene-solr's branch refs/heads/master from markrmiller [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e1a5776 ] SOLR-9997 : Enable configuring SolrHttpClientBuilder via java system property.
          Hide
          hgadre Hrishikesh Gadre added a comment -

          Mark Miller Here is the patch for branch_6x. Please take a look and let me know if anything needed from my side.

          Show
          hgadre Hrishikesh Gadre added a comment - Mark Miller Here is the patch for branch_6x. Please take a look and let me know if anything needed from my side.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a986368fd0670840177a8c19fb15dcd1f0e69797 in lucene-solr's branch refs/heads/branch_6x from markrmiller
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a986368 ]

          SOLR-9997: Enable configuring SolrHttpClientBuilder via java system property.

          Show
          jira-bot ASF subversion and git services added a comment - Commit a986368fd0670840177a8c19fb15dcd1f0e69797 in lucene-solr's branch refs/heads/branch_6x from markrmiller [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a986368 ] SOLR-9997 : Enable configuring SolrHttpClientBuilder via java system property.

            People

            • Assignee:
              markrmiller@gmail.com Mark Miller
              Reporter:
              hgadre Hrishikesh Gadre
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development