Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10718

Configuring Basic auth prevents adding a collection

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.5, 6.5.1
    • Fix Version/s: 6.6
    • Component/s: Server
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      Configure Basic auth according to documentation
      Add basic auth params
      SOLR_AUTH_TYPE="basic"
      SOLR_AUTHENTICATION_OPTS="-Dbasicauth=solr:SolrRocks"

      Try to add a collection
      Receive a timeout and error in the logs

      java.lang.IllegalArgumentException: Credentials may not be null
              at org.apache.http.util.Args.notNull(Args.java:54)
              at org.apache.http.auth.AuthState.update(AuthState.java:113)
              at org.apache.solr.client.solrj.impl.PreemptiveAuth.process(PreemptiveAuth.java:56)
              at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
              at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:166)
              at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:485)
              at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
              at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
              at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
              at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
              at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
      
      1. repro.sh
        1 kB
        Jason Gerlowski
      2. SOLR-10718.patch
        0.7 kB
        Hrishikesh Gadre
      3. SOLR-10718.patch
        0.8 kB
        Ishan Chattopadhyaya

        Issue Links

          Activity

          Hide
          gerlowskija Jason Gerlowski added a comment -

          Attaching a script to reproduce the issue on Linux boxes.

          Can verify that the issue does exist on branch_6_5. Cannot reproduce on master though; likely already fixed.

          Show
          gerlowskija Jason Gerlowski added a comment - Attaching a script to reproduce the issue on Linux boxes. Can verify that the issue does exist on branch_6_5. Cannot reproduce on master though; likely already fixed.
          Hide
          janhoy Jan Høydahl added a comment -

          I manage to reproduce. And it works in master branch. However, the mechanism for building HttpClient changed a lot between 6.x and 7.x (PreemptiveBasicAuthConfigurer vs PreemptiveBasicAuthClientBuilderFactory etc), so I suppose there is something here that has not been tested properly.

          It works fine for the initial call but fails when it is the Overseer that issues the collection creation from the queue.

          Show
          janhoy Jan Høydahl added a comment - I manage to reproduce. And it works in master branch. However, the mechanism for building HttpClient changed a lot between 6.x and 7.x (PreemptiveBasicAuthConfigurer vs PreemptiveBasicAuthClientBuilderFactory etc), so I suppose there is something here that has not been tested properly. It works fine for the initial call but fails when it is the Overseer that issues the collection creation from the queue.
          Hide
          hgadre Hrishikesh Gadre added a comment -

          Jan Høydahl I contributed the relevant changes recently (SOLR-9997). Let me take a look and get back to you.

          Show
          hgadre Hrishikesh Gadre added a comment - Jan Høydahl I contributed the relevant changes recently ( SOLR-9997 ). Let me take a look and get back to you.
          Hide
          hgadre Hrishikesh Gadre added a comment -

          Jan Høydahl While I don't (yet) know the exact implementation issue, the root-cause is setting these two parameters while starting the Solr server.

          grep -q "^SOLR_AUTH_TYPE=\"basic\"" bin/solr.in.sh
          if [ $? != 0 ]; then
          	echo 'SOLR_AUTH_TYPE="basic"' >> bin/solr.in.sh
          	echo 'SOLR_AUTHENTICATION_OPTS="-Dbasicauth=solr:SolrRocks"' >> bin/solr.in.sh
          fi
          

          Note that these parameters are required only on the client side. If you don't add these parameters to solr.in.sh before server startup, then it works as expected. For testing purpose, I just exported these environment variables on the command-line (instead of adding them to solr.in.sh).

          May be we should just ensure that these environment variables are not considered during Solr server startup?

          Show
          hgadre Hrishikesh Gadre added a comment - Jan Høydahl While I don't (yet) know the exact implementation issue, the root-cause is setting these two parameters while starting the Solr server. grep -q "^SOLR_AUTH_TYPE=\"basic\"" bin/solr.in.sh if [ $? != 0 ]; then echo 'SOLR_AUTH_TYPE="basic"' >> bin/solr.in.sh echo 'SOLR_AUTHENTICATION_OPTS="-Dbasicauth=solr:SolrRocks"' >> bin/solr.in.sh fi Note that these parameters are required only on the client side. If you don't add these parameters to solr.in.sh before server startup, then it works as expected. For testing purpose, I just exported these environment variables on the command-line (instead of adding them to solr.in.sh). May be we should just ensure that these environment variables are not considered during Solr server startup?
          Hide
          janhoy Jan Høydahl added a comment -

          Yea, I can confirm it is only a problem when Solr is started with these options. I think that the first CREATE hop from client to Solr is authenticated ok, but once the overseer picks the job from the queue, things are messed up. Wonder if this will also be the case for other overseer operations?

          I'm not sure that avoiding these options during Solr startup will work, since it may break kerberos auth, see SOLR-10646.

          Note: I reproduced this using the new bin/solr auth feature in SOLR-8440 as well:

          bin/solr start -c
          bin/solr auth enable -credentials solr:rocks
          bin/solr stop
          bin/solr start -c
          bin/solr create -c foo
          *BOOM*
          

          I believe we must find a way to fix this where it crashes, else people who start using SOLR-8440 will start complaining once they restart their Solr instance...

          Show
          janhoy Jan Høydahl added a comment - Yea, I can confirm it is only a problem when Solr is started with these options. I think that the first CREATE hop from client to Solr is authenticated ok, but once the overseer picks the job from the queue, things are messed up. Wonder if this will also be the case for other overseer operations? I'm not sure that avoiding these options during Solr startup will work, since it may break kerberos auth, see SOLR-10646 . Note: I reproduced this using the new bin/solr auth feature in SOLR-8440 as well: bin/solr start -c bin/solr auth enable -credentials solr:rocks bin/solr stop bin/solr start -c bin/solr create -c foo *BOOM* I believe we must find a way to fix this where it crashes, else people who start using SOLR-8440 will start complaining once they restart their Solr instance...
          Hide
          ichattopadhyaya Ishan Chattopadhyaya added a comment - - edited

          All the stack traces seem to originate from SolrCLI (i.e. this seems unrelated to internode communication within the Solr server), so this is not a very serious problem. I am still not sure which differences between master and branch_6x caused this to have happened.

          The attached patch makes it work as expected. Jan Høydahl, Hrishikesh Gadre, can you please give the patch a test?

          Show
          ichattopadhyaya Ishan Chattopadhyaya added a comment - - edited All the stack traces seem to originate from SolrCLI (i.e. this seems unrelated to internode communication within the Solr server), so this is not a very serious problem. I am still not sure which differences between master and branch_6x caused this to have happened. The attached patch makes it work as expected. Jan Høydahl , Hrishikesh Gadre , can you please give the patch a test?
          Hide
          ichattopadhyaya Ishan Chattopadhyaya added a comment - - edited

          I'm not sure that avoiding these options during Solr startup will work, since it may break kerberos auth,

          Agreed. It can break kerberos plugin configuration.

          Show
          ichattopadhyaya Ishan Chattopadhyaya added a comment - - edited I'm not sure that avoiding these options during Solr startup will work, since it may break kerberos auth, Agreed. It can break kerberos plugin configuration.
          Hide
          janhoy Jan Høydahl added a comment -

          I think I also reproduced the situation through a HTTP call directly to admin/collection/, not using bin/solr.
          Looking at your patch, that was along the lines I was thinking. But why on earth are creds==null here when -Dbasicauth exists as a global System property?

          Show
          janhoy Jan Høydahl added a comment - I think I also reproduced the situation through a HTTP call directly to admin/collection/, not using bin/solr. Looking at your patch, that was along the lines I was thinking. But why on earth are creds==null  here when -Dbasicauth exists as a global System property?
          Hide
          ichattopadhyaya Ishan Chattopadhyaya added a comment -

          Seems like most requests that pass through that part of the code have valid credentials set in them. However, on just one occasion, I am observing, the request doesn't have valid credentials. I'm unable to figure out why that could be. Ignoring that request, as I have done there in the patch, seems to work fine.

          My suggestion is to commit the patch, and release the 6.6 and follow up with a better fix (if any exists) in a 6.6.1 later.

          Show
          ichattopadhyaya Ishan Chattopadhyaya added a comment - Seems like most requests that pass through that part of the code have valid credentials set in them. However, on just one occasion, I am observing, the request doesn't have valid credentials. I'm unable to figure out why that could be. Ignoring that request, as I have done there in the patch, seems to work fine. My suggestion is to commit the patch, and release the 6.6 and follow up with a better fix (if any exists) in a 6.6.1 later.
          Hide
          hgadre Hrishikesh Gadre added a comment - - edited

          Jan Høydahl Ishan Chattopadhyaya I found the issue and the fix is attached.

          Here is the summary of the problem,

          • Since we are using embedded ZK, the security.json needs to be uploaded after starting Solr server.
          • But since the basic authentication is configured during the server startup (via SOLR_AUTH_TYPE env variable), the default HTTP client in HttpShardHandler is configured with PreemptiveAuth request interceptor.
          • When we upload security.json file, we invoke HttpShardHandlerFactory#reconfigureHttpClient(...) API to configure PKI authentication scheme. In this process, HttpClientUtil#setBasicAuth(...) API is invoked.
          • In the setBasicAuth(...) method we are cleaning only the credentials but not the PreemptiveAuth request interceptor. Hence when this HTTP client is used subsequently, we observe IllegalArgumentException since PreemptiveAuth request interceptor requires non-null credentials.

          So the fix in this case is to remove PreemptiveAuth request interceptor when basic auth is not to be used.

          Show
          hgadre Hrishikesh Gadre added a comment - - edited Jan Høydahl Ishan Chattopadhyaya I found the issue and the fix is attached. Here is the summary of the problem, Since we are using embedded ZK, the security.json needs to be uploaded after starting Solr server. But since the basic authentication is configured during the server startup (via SOLR_AUTH_TYPE env variable), the default HTTP client in HttpShardHandler is configured with PreemptiveAuth request interceptor. When we upload security.json file, we invoke HttpShardHandlerFactory#reconfigureHttpClient(...) API to configure PKI authentication scheme. In this process, HttpClientUtil#setBasicAuth(...) API is invoked. In the setBasicAuth(...) method we are cleaning only the credentials but not the PreemptiveAuth request interceptor. Hence when this HTTP client is used subsequently, we observe IllegalArgumentException since PreemptiveAuth request interceptor requires non-null credentials. So the fix in this case is to remove PreemptiveAuth request interceptor when basic auth is not to be used.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit aec5d68a068aedbeec91a9d218a688acde1cacd0 in lucene-solr's branch refs/heads/branch_6_6 from Ishan Chattopadhyaya
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=aec5d68 ]

          SOLR-10718: Not using pre-emptive authentication interceptor when not using BasicAuth

          Show
          jira-bot ASF subversion and git services added a comment - Commit aec5d68a068aedbeec91a9d218a688acde1cacd0 in lucene-solr's branch refs/heads/branch_6_6 from Ishan Chattopadhyaya [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=aec5d68 ] SOLR-10718 : Not using pre-emptive authentication interceptor when not using BasicAuth
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 91316cc8037712d81a623078e2e75664a4e03eb4 in lucene-solr's branch refs/heads/branch_6x from Ishan Chattopadhyaya
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=91316cc ]

          SOLR-10718: Not using pre-emptive authentication interceptor when not using BasicAuth

          Show
          jira-bot ASF subversion and git services added a comment - Commit 91316cc8037712d81a623078e2e75664a4e03eb4 in lucene-solr's branch refs/heads/branch_6x from Ishan Chattopadhyaya [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=91316cc ] SOLR-10718 : Not using pre-emptive authentication interceptor when not using BasicAuth
          Hide
          ichattopadhyaya Ishan Chattopadhyaya added a comment -

          I tested the fix, and it works. Thanks everyone!

          Show
          ichattopadhyaya Ishan Chattopadhyaya added a comment - I tested the fix, and it works. Thanks everyone!
          Hide
          janhoy Jan Høydahl added a comment -

          Good research Hrishikesh Gadre. Feels good to nail that one.

          How could we improve our test suite to get better overall Auth coverage? Is it time to randomize BasicAuth true/false for all or majority of Cloud tests?

          Show
          janhoy Jan Høydahl added a comment - Good research Hrishikesh Gadre . Feels good to nail that one. How could we improve our test suite to get better overall Auth coverage? Is it time to randomize BasicAuth true/false for all or majority of Cloud tests?

            People

            • Assignee:
              ichattopadhyaya Ishan Chattopadhyaya
              Reporter:
              safeldm Shawn Feldman
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development