Solr
  1. Solr
  2. SOLR-8373

KerberosPlugin: Using multiple nodes on same machine leads clients to fetch TGT for every request

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3.2, 5.4.1, 5.5, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Kerberized solr nodes accept negotiate/spnego/kerberos requests and processes them. It also passes back to the client a cookie called "hadoop.auth" (which is currently unused, but will eventually be used for delegation tokens).

      If two or more nodes are on the same machine, they all send out the cookie which have the same domain (hostname) and same path, but different cookie values.

      Upon receipt at the client, if a cookie is rejected (which in this case will be), the client gets a​​ TGT from the KDC. This is causing the heavy traffic at the KDC, plus intermittent "Request is a replay" (which indicates race condition at KDC while handing out the TGT for the same principal). I think having a (well configured) ticket cache is a potential solution, but having cookies get rejected is bad enough.

      1. SOLR-8373.patch
        19 kB
        Noble Paul
      2. SOLR-8373.patch
        23 kB
        Ishan Chattopadhyaya
      3. SOLR-8373.patch
        15 kB
        Ishan Chattopadhyaya
      4. SOLR-8373.patch
        5 kB
        Ishan Chattopadhyaya
      5. SOLR-8373.patch
        1 kB
        Ishan Chattopadhyaya

        Activity

        Hide
        Ishan Chattopadhyaya added a comment -

        I'm testing this patch that lets the clients ignore the cookies when talking to the kerberized Solr nodes.

        Show
        Ishan Chattopadhyaya added a comment - I'm testing this patch that lets the clients ignore the cookies when talking to the kerberized Solr nodes.
        Hide
        Ishan Chattopadhyaya added a comment -

        It seems if ticket caching (credentials cache) isn't set up properly, ignoring cookies always (as in this patch) will have the client fetch the TGT from the KDC again.

        Since, fetching the ticket from the KDC (or even the ticket cache) and sending again and again isn't ideal, I am now looking to have a modified cookie spec implemented within the realms of HttpClient (which SolrJ depends on), which will restrict the cookies by host and port, since the standard cookie RFCs and the browsers are okay to share cookies for the same host across different applications running on different ports. This will allow multiple solr nodes on the same host to work properly without the clients going to the KDC (or even ticket cache) for the tickets.

        I shall post a patch for this approach in a while.

        Show
        Ishan Chattopadhyaya added a comment - It seems if ticket caching (credentials cache) isn't set up properly, ignoring cookies always (as in this patch) will have the client fetch the TGT from the KDC again. Since, fetching the ticket from the KDC (or even the ticket cache) and sending again and again isn't ideal, I am now looking to have a modified cookie spec implemented within the realms of HttpClient (which SolrJ depends on), which will restrict the cookies by host and port , since the standard cookie RFCs and the browsers are okay to share cookies for the same host across different applications running on different ports. This will allow multiple solr nodes on the same host to work properly without the clients going to the KDC (or even ticket cache) for the tickets. I shall post a patch for this approach in a while.
        Hide
        Ishan Chattopadhyaya added a comment -

        Here's a patch I'm working on.

        As of this patch, to invoke this, Solr nodes (that are on a shared hosting, so to speak) need to start Solr using the port number as part of the cookie domain:

        bin/solr -c -p 8983 -Dsolr.kerberos.cookie.domain=hostname:8983
        (This, obviously, cannot go into the solr.in.sh, and hence needs to be removed from there).

        Looking to see if there's something better that can be done to pass the port number to the kerberos authentication plugin.

        Show
        Ishan Chattopadhyaya added a comment - Here's a patch I'm working on. As of this patch, to invoke this, Solr nodes (that are on a shared hosting, so to speak) need to start Solr using the port number as part of the cookie domain: bin/solr -c -p 8983 -Dsolr.kerberos.cookie.domain=hostname:8983 (This, obviously, cannot go into the solr.in.sh, and hence needs to be removed from there). Looking to see if there's something better that can be done to pass the port number to the kerberos authentication plugin.
        Hide
        Ishan Chattopadhyaya added a comment - - edited

        Updated patch.

        1. All authentication plugins are CoreContainer aware. This was needed for letting the plugin know the port number on which Solr was started.
        2. Introduces a new startup parameter, solr.kerberos.cookie.portaware=true. When using SolrCloud, and this parameter is true, the cookies use both the host and the port to identify the domain. This should be enabled only on hosts where more than one solr node needs to be setup. This can go in the bin/solr.in.sh.
        Show
        Ishan Chattopadhyaya added a comment - - edited Updated patch. All authentication plugins are CoreContainer aware. This was needed for letting the plugin know the port number on which Solr was started. Introduces a new startup parameter, solr.kerberos.cookie.portaware=true . When using SolrCloud, and this parameter is true, the cookies use both the host and the port to identify the domain. This should be enabled only on hosts where more than one solr node needs to be setup. This can go in the bin/solr.in.sh .
        Hide
        Ishan Chattopadhyaya added a comment -

        Added test cases, minor refactoring here and there. I did some end to end testing and the changes look good to me thus far.

        Was just wondering if the change to have all authentication and authorization plugins now accept a CoreContainer warrants a separate issue?

        Show
        Ishan Chattopadhyaya added a comment - Added test cases, minor refactoring here and there. I did some end to end testing and the changes look good to me thus far. Was just wondering if the change to have all authentication and authorization plugins now accept a CoreContainer warrants a separate issue?
        Hide
        Noble Paul added a comment - - edited

        smaller patch eliminating the backward incompatible change to the init()

        Show
        Noble Paul added a comment - - edited smaller patch eliminating the backward incompatible change to the init()
        Hide
        ASF subversion and git services added a comment -

        Commit 1718852 from Noble Paul in branch 'dev/trunk'
        [ https://svn.apache.org/r1718852 ]

        SOLR-8373: KerberosPlugin: Using multiple nodes on same machine leads clients to
        fetch TGT for every request

        Show
        ASF subversion and git services added a comment - Commit 1718852 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1718852 ] SOLR-8373 : KerberosPlugin: Using multiple nodes on same machine leads clients to fetch TGT for every request
        Hide
        ASF subversion and git services added a comment -

        Commit 1718854 from Noble Paul in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1718854 ]

        SOLR-8373: KerberosPlugin: Using multiple nodes on same machine leads clients to
        fetch TGT for every request

        Show
        ASF subversion and git services added a comment - Commit 1718854 from Noble Paul in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1718854 ] SOLR-8373 : KerberosPlugin: Using multiple nodes on same machine leads clients to fetch TGT for every request
        Hide
        Ishan Chattopadhyaya added a comment -

        Reopening for backporting to 5.3.2. Noble Paul, can you please commit the backport? Thanks.

        Show
        Ishan Chattopadhyaya added a comment - Reopening for backporting to 5.3.2. Noble Paul , can you please commit the backport? Thanks.
        Hide
        ASF subversion and git services added a comment -

        Commit 1721679 from Noble Paul in branch 'dev/branches/lucene_solr_5_3'
        [ https://svn.apache.org/r1721679 ]

        SOLR-8373: KerberosPlugin: Using multiple nodes on same machine leads clients to
        fetch TGT for every request

        Show
        ASF subversion and git services added a comment - Commit 1721679 from Noble Paul in branch 'dev/branches/lucene_solr_5_3' [ https://svn.apache.org/r1721679 ] SOLR-8373 : KerberosPlugin: Using multiple nodes on same machine leads clients to fetch TGT for every request
        Hide
        Ishan Chattopadhyaya added a comment -

        Thanks Noble. Merry Christmas!

        Show
        Ishan Chattopadhyaya added a comment - Thanks Noble. Merry Christmas!
        Hide
        ASF subversion and git services added a comment -

        Commit 1722060 from Anshum Gupta in branch 'dev/trunk'
        [ https://svn.apache.org/r1722060 ]

        SOLR-8373: Add change log entry to 5.3.2 section on trunk

        Show
        ASF subversion and git services added a comment - Commit 1722060 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1722060 ] SOLR-8373 : Add change log entry to 5.3.2 section on trunk
        Hide
        ASF subversion and git services added a comment -

        Commit 1722061 from Anshum Gupta in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1722061 ]

        SOLR-8373: Add change log entry to 5.3.2 section (merge from trunk)

        Show
        ASF subversion and git services added a comment - Commit 1722061 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1722061 ] SOLR-8373 : Add change log entry to 5.3.2 section (merge from trunk)
        Hide
        ASF subversion and git services added a comment -

        Commit 1724179 from Adrien Grand in branch 'dev/branches/lucene_solr_5_4'
        [ https://svn.apache.org/r1724179 ]

        SOLR-8373: KerberosPlugin: Using multiple nodes on same machine leads clients to fetch TGT for every request

        Show
        ASF subversion and git services added a comment - Commit 1724179 from Adrien Grand in branch 'dev/branches/lucene_solr_5_4' [ https://svn.apache.org/r1724179 ] SOLR-8373 : KerberosPlugin: Using multiple nodes on same machine leads clients to fetch TGT for every request
        Hide
        ASF subversion and git services added a comment -

        Commit 9ef144ddefe21f30c1c9ebd5246e7e03387488e1 in lucene-solr's branch refs/heads/branch_5_4 from Adrien Grand
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9ef144d ]

        SOLR-8460, SOLR-8373, SOLR-8422, SOLR-7462, SOLR-8470: Add CHANGES entries for 5.4.1.

        git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_4@1724198 13f79535-47bb-0310-9956-ffa450edef68

        Show
        ASF subversion and git services added a comment - Commit 9ef144ddefe21f30c1c9ebd5246e7e03387488e1 in lucene-solr's branch refs/heads/branch_5_4 from Adrien Grand [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9ef144d ] SOLR-8460 , SOLR-8373 , SOLR-8422 , SOLR-7462 , SOLR-8470 : Add CHANGES entries for 5.4.1. git-svn-id: https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_4@1724198 13f79535-47bb-0310-9956-ffa450edef68

          People

          • Assignee:
            Noble Paul
            Reporter:
            Ishan Chattopadhyaya
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development