Solr
  1. Solr
  2. SOLR-5676

SolrCloud updates rejected if talking to secure ZooKeeper

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.6.1
    • Fix Version/s: 4.7, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      When using secure zookeeper and a valid jaas configuration, SolrCloud will reject updates with a 503 error.

      The problem is that in this case ZooKeeper sends states to the watchers like "SaslAuthenticated", but the ConnectionManager treats any state it doesn't know about as a disconnect. Then, whenever a request comes in, SolrCloud will think it can't talk to ZooKeeper and reject the request.

      These are the valid states that watchers can see with the current ZooKeeper version (3.4.5):
      Disconnected
      SyncConnected
      AuthFailed
      ConnectedReadOnly
      SaslAuthenticated
      Expired

      ConnectionManager currently does not handle:
      SaslAuthenticated
      ConnectedReadOnly
      AuthFailed

      From my tests, it seems like the correct thing to do is just ignore these states:
      1) SaslAuthenticated - nothing has gone wrong here, so no need to disconnected
      2) ConnectedReadOnly - the client would only see this state if they specifically specified a read only client, so this is expected
      3) AuthFailed - this one is a little tricky. If you try to authentication but fail (say you have an invalid JAAS conf), you may get AuthFailed, but depending on the ZK settings, you may still be able to communicate with ZooKeeper (i.e. see this in the log: Will continue connection to Zookeeper server \
      without SASL authentication, if Zookeeper server allows it.)

      So, it seems correct to ignore these states, possibly warning on AuthFailed, because that isn't expected in proper operation.

      1. SOLR-5676.patch
        0.7 kB
        Gregory Chanan

        Activity

        Hide
        Gregory Chanan added a comment -

        Here's a patch that does the above, warning if the client receives AuthFailed.

        Show
        Gregory Chanan added a comment - Here's a patch that does the above, warning if the client receives AuthFailed.
        Hide
        ASF subversion and git services added a comment -

        Commit 1563189 from Mark Miller in branch 'dev/trunk'
        [ https://svn.apache.org/r1563189 ]

        SOLR-5676: SolrCloud updates rejected if talking to secure ZooKeeper.

        Show
        ASF subversion and git services added a comment - Commit 1563189 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1563189 ] SOLR-5676 : SolrCloud updates rejected if talking to secure ZooKeeper.
        Hide
        ASF subversion and git services added a comment -

        Commit 1563192 from Mark Miller in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1563192 ]

        SOLR-5676: SolrCloud updates rejected if talking to secure ZooKeeper.

        Show
        ASF subversion and git services added a comment - Commit 1563192 from Mark Miller in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1563192 ] SOLR-5676 : SolrCloud updates rejected if talking to secure ZooKeeper.
        Hide
        Mark Miller added a comment -

        Thanks Greg! Very simple fix, so I just put it in, we we should also really consider adding a test against a secure zk to ensure proper support in the future.

        Show
        Mark Miller added a comment - Thanks Greg! Very simple fix, so I just put it in, we we should also really consider adding a test against a secure zk to ensure proper support in the future.
        Hide
        Gregory Chanan added a comment -

        Agree on the test, I'll look into some other projects (hbase?) and see if they have unit tests for secure ZK.

        Show
        Gregory Chanan added a comment - Agree on the test, I'll look into some other projects (hbase?) and see if they have unit tests for secure ZK.
        Hide
        Mark Miller added a comment -

        Thanks Greg! I'll make a new JIRA issue for a secure test.

        Show
        Mark Miller added a comment - Thanks Greg! I'll make a new JIRA issue for a secure test.

          People

          • Assignee:
            Mark Miller
            Reporter:
            Gregory Chanan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development