Solr
  1. Solr
  2. SOLR-8355

RuleBasedAuthenticationPlugin doesn't work with update permission enabled

    Details

      Description

      Here are the steps that recreate this issue. I tried this on Solr 5.4 and I had the following stack trace when I issued an ADDREPLICA. This seems pretty similar to what we saw on SOLR-8326 so it might be just something we missed but we should make sure that we ship 5.4 with this fixed.

      #Upload Security Conf
      server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd putfile /security.json ~/security.json

      #Start Solr
      bin/solr start -e cloud -z localhost:2181

      #Collection Admin Edit Command:
      curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-permission" : {"name":"collection-admin-edit", "role":"admin"}}'

      #Read User and permission:
      curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-permission" : {"name":"read", "role":"read"}}'

      curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-permission" : {"name":"update", "role":"update"]}}'

      #Add Users
      #Read User
      curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication -H 'Content-type:application/json' -d '{"set-user" : {"solrread":"solrRocks"}}'

      #Update user
      curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication -H 'Content-type:application/json' -d '{"set-user" : {"solrupdate":"solrRocks"}}'

      #Set user roles
      curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-user-role" : {"solrupdate":["read","update"]}}'

      #Read User
      curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization -H 'Content-type:application/json' -d '{"set-user-role" : {"solrread":["read"]}}'

      #Create collection
      curl --user solr:SolrRocks 'http://localhost:8983/solr/admin/collections?action=CREATE&name=a&numShards=1&replicationFactor=1&collection.configName=gettingstarted&wt=json'

      #Add Replica
      curl --user solr:SolrRocks 'http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=a&shard=shard1&wt=json'

      Exception log:

      INFO - 2015-12-01 04:57:47.022; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery.
      INFO - 2015-12-01 04:57:47.023; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from http://172.20.10.4:7574/solr/a_shard1_replica1/.
      ERROR - 2015-12-01 04:57:47.027; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.common.SolrException; Error while trying to recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.20.10.4:7574/solr/a_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html>
      <head>
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
      <title>Error 401 Unauthorized request, Response code: 401</title>
      </head>
      <body><h2>HTTP ERROR 401</h2>
      <p>Problem accessing /solr/a_shard1_replica1/update. Reason:
      <pre> Unauthorized request, Response code: 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

      </body>
      </html>

      at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
      at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
      at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
      at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
      at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:167)
      at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:205)
      at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:145)
      at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:436)
      at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:225)

      INFO - 2015-12-01 04:57:47.028; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping buffered updates FSUpdateLog

      {state=BUFFERING, tlog=null}

      ERROR - 2015-12-01 04:57:47.028; [c:a s:shard1 r:core_node2 x:a_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (4)

        Activity

        Hide
        Noble Paul added a comment -

        Nailed the issue, RecoveryStrategy creates its own httpclient. So it does not send the PKI Authentication Headers. So, the request is not authorized at the leader.

        Show
        Noble Paul added a comment - Nailed the issue, RecoveryStrategy creates its own httpclient. So it does not send the PKI Authentication Headers. So, the request is not authorized at the leader.
        Hide
        Ishan Chattopadhyaya added a comment -

        Nailed the issue, RecoveryStrategy creates its own httpclient.

        The RecoveryStrategy.commitOnLeader() creates a HttpSolrClient, which in turn creates its own HttpClient using HttpClientUtil.createClient(). If this is a problem, are all places where a HttpSolrClient is created for intershard communication affected by this problem?

        For example, LeaderInitiatedRecoveryThread.sendRecoveryCommandWithRetry() also creates its own HttpSolrClient. Other places where HttpSolrClient is created are: SnitchContext, OCMH, OverseerAutoReplicaFailoverThread, SyncStrategy, CollectionsHandler etc.

        Do you think we should do something with the HttpClientUtil.createClient() to return a HttpClient which passes on the PKI headers?

        Show
        Ishan Chattopadhyaya added a comment - Nailed the issue, RecoveryStrategy creates its own httpclient. The RecoveryStrategy.commitOnLeader() creates a HttpSolrClient, which in turn creates its own HttpClient using HttpClientUtil.createClient(). If this is a problem, are all places where a HttpSolrClient is created for intershard communication affected by this problem? For example, LeaderInitiatedRecoveryThread.sendRecoveryCommandWithRetry() also creates its own HttpSolrClient. Other places where HttpSolrClient is created are: SnitchContext, OCMH, OverseerAutoReplicaFailoverThread, SyncStrategy, CollectionsHandler etc. Do you think we should do something with the HttpClientUtil.createClient() to return a HttpClient which passes on the PKI headers?
        Hide
        Noble Paul added a comment -

        Yes Ishan Chattopadhyaya I think all our inter node authentications will fail if we don't have a standard mechanism for creating/using httpclient

        Show
        Noble Paul added a comment - Yes Ishan Chattopadhyaya I think all our inter node authentications will fail if we don't have a standard mechanism for creating/using httpclient
        Hide
        Noble Paul added a comment -

        Actually , it fails only if we create our own thread. Some places do a new Thread() and in those places the appropriate thread local values are not set

        Show
        Noble Paul added a comment - Actually , it fails only if we create our own thread. Some places do a new Thread() and in those places the appropriate thread local values are not set
        Hide
        Noble Paul added a comment -

        I have made the recovery thread to run in an executor pool thread

        Folks please review if it can cause any problem

        Show
        Noble Paul added a comment - I have made the recovery thread to run in an executor pool thread Folks please review if it can cause any problem
        Hide
        Ishan Chattopadhyaya added a comment -

        Looks good to me!
        I think we need to do the same for CoreAdminHandler.handleRequestRecoveryAction(), where a new Thread is created and start() is called on it directly.

        Show
        Ishan Chattopadhyaya added a comment - Looks good to me! I think we need to do the same for CoreAdminHandler.handleRequestRecoveryAction(), where a new Thread is created and start() is called on it directly.
        Hide
        Ishan Chattopadhyaya added a comment - - edited

        Maybe also for:
        OCMH.processRoleCommand() (at the end of the method)
        Overseer.start() has start() called on many threads.
        There may be other places. Do you think we should forbid the Thread.start(), and instead enforce the use of threadpools, so as to ensure secure internode communication is never broken? Or does it sound too extreme?

        Show
        Ishan Chattopadhyaya added a comment - - edited Maybe also for: OCMH.processRoleCommand() (at the end of the method) Overseer.start() has start() called on many threads. There may be other places. Do you think we should forbid the Thread.start(), and instead enforce the use of threadpools, so as to ensure secure internode communication is never broken? Or does it sound too extreme?
        Hide
        ASF subversion and git services added a comment -

        Commit 1717493 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1717493 ]

        SOLR-8355: update permissions were failing node recovery

        Show
        ASF subversion and git services added a comment - Commit 1717493 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1717493 ] SOLR-8355 : update permissions were failing node recovery
        Hide
        ASF subversion and git services added a comment -

        Commit 1717494 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1717494 ]

        SOLR-8355: update permissions were failing node recovery

        Show
        ASF subversion and git services added a comment - Commit 1717494 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1717494 ] SOLR-8355 : update permissions were failing node recovery
        Hide
        ASF subversion and git services added a comment -

        Commit 1717495 from Noble Paul in branch 'dev/branches/lucene_solr_5_4'
        [ https://svn.apache.org/r1717495 ]

        SOLR-8355: update permissions were failing node recovery

        Show
        ASF subversion and git services added a comment - Commit 1717495 from Noble Paul in branch 'dev/branches/lucene_solr_5_4' [ https://svn.apache.org/r1717495 ] SOLR-8355 : update permissions were failing node recovery
        Hide
        Anshum Gupta added a comment -

        Reopening to backport to 5.3.2

        Show
        Anshum Gupta added a comment - Reopening to backport to 5.3.2
        Hide
        ASF subversion and git services added a comment -

        Commit 1722075 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_3'
        [ https://svn.apache.org/r1722075 ]

        SOLR-8355: fix for update permissions causing node recovery failures(backport from branch_5x for 5.3.2 release)

        Show
        ASF subversion and git services added a comment - Commit 1722075 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_3' [ https://svn.apache.org/r1722075 ] SOLR-8355 : fix for update permissions causing node recovery failures(backport from branch_5x for 5.3.2 release)
        Hide
        ASF subversion and git services added a comment -

        Commit 1722076 from Anshum Gupta in branch 'dev/trunk'
        [ https://svn.apache.org/r1722076 ]

        SOLR-8355: Add change log entry to 5.3.2 section on trunk

        Show
        ASF subversion and git services added a comment - Commit 1722076 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1722076 ] SOLR-8355 : Add change log entry to 5.3.2 section on trunk
        Hide
        ASF subversion and git services added a comment -

        Commit 1722077 from Anshum Gupta in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1722077 ]

        SOLR-8355: Add change log entry to 5.3.2 section (merge from trunk)

        Show
        ASF subversion and git services added a comment - Commit 1722077 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1722077 ] SOLR-8355 : Add change log entry to 5.3.2 section (merge from trunk)
        Hide
        ASF subversion and git services added a comment -

        Commit 1724183 from Adrien Grand in branch 'dev/branches/lucene_solr_5_4'
        [ https://svn.apache.org/r1724183 ]

        SOLR-8355: Add change log entry to 5.3.2 section.

        Show
        ASF subversion and git services added a comment - Commit 1724183 from Adrien Grand in branch 'dev/branches/lucene_solr_5_4' [ https://svn.apache.org/r1724183 ] SOLR-8355 : Add change log entry to 5.3.2 section.
        Hide
        Varun Thacker added a comment -

        Adding BasicAuth tag

        Show
        Varun Thacker added a comment - Adding BasicAuth tag

          People

          • Assignee:
            Anshum Gupta
            Reporter:
            Anshum Gupta
          • Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development