Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.2
    • Component/s: security
    • Labels:
      None

      Description

      SOLR-7274 introduces a pluggable authentication framework. This issue provides a Kerberos plugin implementation.

      1. hoss_trunk_r1681791_TEST-org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.xml
        21 kB
        Hoss Man
      2. hoss_trunk_r1681791_tests-failures.txt
        55 kB
        Hoss Man
      3. SOLR-7468.patch
        3 kB
        Ishan Chattopadhyaya
      4. SOLR-7468.patch
        37 kB
        Anshum Gupta
      5. SOLR-7468.patch
        36 kB
        Ishan Chattopadhyaya
      6. SOLR-7468.patch
        36 kB
        Anshum Gupta
      7. SOLR-7468.patch
        35 kB
        Ishan Chattopadhyaya
      8. SOLR-7468.patch
        30 kB
        Ishan Chattopadhyaya
      9. SOLR-7468.patch
        30 kB
        Ishan Chattopadhyaya
      10. SOLR-7468.patch
        30 kB
        Ishan Chattopadhyaya
      11. SOLR-7468.patch
        30 kB
        Ishan Chattopadhyaya
      12. SOLR-7468.patch
        29 kB
        Ishan Chattopadhyaya
      13. SOLR-7468.patch
        36 kB
        Anshum Gupta
      14. SOLR-7468.patch
        29 kB
        Anshum Gupta
      15. SOLR-7468.patch
        21 kB
        Ishan Chattopadhyaya
      16. SOLR-7468.patch
        34 kB
        Anshum Gupta
      17. SOLR-7468.patch
        34 kB
        Ishan Chattopadhyaya
      18. SOLR-7468.patch
        35 kB
        Ishan Chattopadhyaya
      19. SOLR-7468.patch
        13 kB
        Ishan Chattopadhyaya
      20. SOLR-7468.patch
        16 kB
        Ishan Chattopadhyaya
      21. SOLR-7468-alt-test.patch
        16 kB
        Ishan Chattopadhyaya
      22. SOLR-7468-alt-test.patch
        11 kB
        Ishan Chattopadhyaya
      23. SOLR-7468-alt-test.patch
        11 kB
        Anshum Gupta

        Issue Links

          Activity

          Hide
          Ishan Chattopadhyaya added a comment -

          This patch should be applied after SOLR-7274 patch is applied/committed. Some details and discussion regarding the kerberos plugin is in SOLR-7274.

          Show
          Ishan Chattopadhyaya added a comment - This patch should be applied after SOLR-7274 patch is applied/committed. Some details and discussion regarding the kerberos plugin is in SOLR-7274 .
          Hide
          Ishan Chattopadhyaya added a comment -

          Updating the patch.

          Now contains:

          • A test based on MiniSolrCloudCluster with kerberos, using hadoop-minikdc. It is sometimes fails due to minikdc, but with a real external KDC, always passes. Need to look into why.
          • Parameter names changed slightly.
          • Adding a @lucene.experimental annotation to some of the classes.

          Here is an example start command:

          bin/solr -c -a "-Djava.security.auth.login.config=/home/ishan/jaas-client.conf -Dsolr.kerberos.jaas.appname=SolrClient -Dcookie.domain=192.168.122.1 -Dkerberos.principal=HTTP/192.168.122.1@EXAMPLE.COM -Dkerberos.keytab=/tmp/solr.keytab -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin"
          

          This starts the solr service with kerberos plugin. This can also be specified at ZK (see SOLR-7275 for /security.json format). The jaas-client.conf is specified at SOLR-7274.

          TODO:

          • SolrCLI changes
          • Look into why the test fails sometimes
          Show
          Ishan Chattopadhyaya added a comment - Updating the patch. Now contains: A test based on MiniSolrCloudCluster with kerberos, using hadoop-minikdc. It is sometimes fails due to minikdc, but with a real external KDC, always passes. Need to look into why. Parameter names changed slightly. Adding a @lucene.experimental annotation to some of the classes. Here is an example start command: bin/solr -c -a "-Djava.security.auth.login.config=/home/ishan/jaas-client.conf -Dsolr.kerberos.jaas.appname=SolrClient -Dcookie.domain=192.168.122.1 -Dkerberos.principal=HTTP/192.168.122.1@EXAMPLE.COM -Dkerberos.keytab=/tmp/solr.keytab -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin" This starts the solr service with kerberos plugin. This can also be specified at ZK (see SOLR-7275 for /security.json format). The jaas-client.conf is specified at SOLR-7274 . TODO: SolrCLI changes Look into why the test fails sometimes
          Hide
          Anshum Gupta added a comment -

          Here's some feedback:

          1. Can we avoid the addition of the extra Servlet Filter (KerberosFilter) ? Now that SDF is essentially a wrapper, perhaps we could reuse the wrapper.
          2. If we do #1, we also wouldn't need the change/hack in the JettySolrRunner. Also, no change would be needed in MiniSolrCloudCluster.
          3. Minor but important, I noticed a lot of unused imports, you should clean those up.
          Show
          Anshum Gupta added a comment - Here's some feedback: Can we avoid the addition of the extra Servlet Filter (KerberosFilter) ? Now that SDF is essentially a wrapper, perhaps we could reuse the wrapper. If we do #1, we also wouldn't need the change/hack in the JettySolrRunner. Also, no change would be needed in MiniSolrCloudCluster. Minor but important, I noticed a lot of unused imports, you should clean those up.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updated the patch. Removing unused imports.

          Show
          Ishan Chattopadhyaya added a comment - Updated the patch. Removing unused imports.
          Hide
          Anshum Gupta added a comment -

          Updated patch with the following changes:

          1. Considering TestMiniCloudClusterKerberos was duplicating code from TestMiniCloudCluster, I changed the scope of a few things in the TestMiniCloudCluster and got the former to extend it. This test should ideally just set up kdc and then piggy back on the other test.
          2. Unset authenticationPlugin system prop instead of authcPlugin in the test.
          3. Cleaned up code.

          P.S: The TestMiniCloudClusterKerberos still doesn't pass. I'm not clear what's going on at this point but I see a lot of the following errors logged suggesting something about Zk and SASL:

            [junit4]   2> 53100 T314 oazs.ZooKeeperServer.processSasl WARN Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
             [junit4]   2> 53101 T314 oazs.ZooKeeperServer.processSasl WARN Closing client connection due to SASL authentication failure.
             [junit4]   2> 53101 T314 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception:  java.nio.channels.CancelledKeyException
          

          It'd be great to have suggestions on this.

          Show
          Anshum Gupta added a comment - Updated patch with the following changes: Considering TestMiniCloudClusterKerberos was duplicating code from TestMiniCloudCluster, I changed the scope of a few things in the TestMiniCloudCluster and got the former to extend it. This test should ideally just set up kdc and then piggy back on the other test. Unset authenticationPlugin system prop instead of authcPlugin in the test. Cleaned up code. P.S: The TestMiniCloudClusterKerberos still doesn't pass. I'm not clear what's going on at this point but I see a lot of the following errors logged suggesting something about Zk and SASL: [junit4] 2> 53100 T314 oazs.ZooKeeperServer.processSasl WARN Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] [junit4] 2> 53101 T314 oazs.ZooKeeperServer.processSasl WARN Closing client connection due to SASL authentication failure. [junit4] 2> 53101 T314 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException It'd be great to have suggestions on this.
          Hide
          Ishan Chattopadhyaya added a comment - - edited

          +1 to extending the test. I had kept it copied because I was desperately trying out changes to see if the test passes consistently, but to no avail.

          I finally found out that the test fails due to lack of a ticket cache in minikdc, and due to which all 5 servers on the node were competing to get the same service ticket from the minikdc for every request. hadoop-minikdc depends on ApacheDS, and currently using 2.0.0-M15, but support for ticket cache is available only after M16. HADOOP-9893.

          I suggest we mark the test as @AwaitsFix("HADOOP-9893") and revisit it later. Meanwhile, we can write a simple test for kerberos that tests a single end-point authentication.

          Show
          Ishan Chattopadhyaya added a comment - - edited +1 to extending the test. I had kept it copied because I was desperately trying out changes to see if the test passes consistently, but to no avail. I finally found out that the test fails due to lack of a ticket cache in minikdc, and due to which all 5 servers on the node were competing to get the same service ticket from the minikdc for every request. hadoop-minikdc depends on ApacheDS, and currently using 2.0.0-M15, but support for ticket cache is available only after M16. HADOOP-9893 . I suggest we mark the test as @AwaitsFix(" HADOOP-9893 ") and revisit it later. Meanwhile, we can write a simple test for kerberos that tests a single end-point authentication.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updated patch.

          • Removed some of test framework changes from here, included in SOLR-7274.
          • Added @AwaitsFix to TestSolrMiniCloudClusterKerberos tests, bugUrl as HADOOP-9893. The test passes fine with external KDC since that supports ticket caches, but fails with minikdc.

          TODO:

          • A simpler test that uses only one Solr node (so as to avoid HADOOP-9893).
          • [Nice to have] Figure out a way to change the plugin to support folding in of AuthenticationLayerFilter into SDF.
          Show
          Ishan Chattopadhyaya added a comment - Updated patch. Removed some of test framework changes from here, included in SOLR-7274 . Added @AwaitsFix to TestSolrMiniCloudClusterKerberos tests, bugUrl as HADOOP-9893 . The test passes fine with external KDC since that supports ticket caches, but fails with minikdc. TODO: A simpler test that uses only one Solr node (so as to avoid HADOOP-9893 ). [Nice to have] Figure out a way to change the plugin to support folding in of AuthenticationLayerFilter into SDF.
          Hide
          Anshum Gupta added a comment -

          Folding in of the AuthenticationLayerFilter is not really a blocker from where I see it. Specially considering that it's an implementation detail and wouldn't relate to changing the public API.

          Show
          Anshum Gupta added a comment - Folding in of the AuthenticationLayerFilter is not really a blocker from where I see it. Specially considering that it's an implementation detail and wouldn't relate to changing the public API.
          Hide
          Anshum Gupta added a comment - - edited

          Added a test to test Kerberized Solr.

          Show
          Anshum Gupta added a comment - - edited Added a test to test Kerberized Solr.
          Hide
          Anshum Gupta added a comment -

          Updated patch with a fix for TestMiniSolrCloudCluster.

          Show
          Anshum Gupta added a comment - Updated patch with a fix for TestMiniSolrCloudCluster.
          Hide
          Anshum Gupta added a comment -

          TestMiniSolrCloudClusterKerberos needs to enable authentication via zk, else I see the system prop stepping on other tests which shouldn't be aware of authc. You might need to change MiniSolrCloudCluster to be able to do that as right now, it wouldn't let you set zk data before the jettys come up.
          Other than that, things work for me.

          Show
          Anshum Gupta added a comment - TestMiniSolrCloudClusterKerberos needs to enable authentication via zk, else I see the system prop stepping on other tests which shouldn't be aware of authc. You might need to change MiniSolrCloudCluster to be able to do that as right now, it wouldn't let you set zk data before the jettys come up. Other than that, things work for me.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updating the patch; just moving a few changes from here to SOLR-7274.

          Show
          Ishan Chattopadhyaya added a comment - Updating the patch; just moving a few changes from here to SOLR-7274 .
          Hide
          Ishan Chattopadhyaya added a comment -

          Updating the patch. Some minor changes to the jaas configuration for tests, SSL properties for the tests, got rid of the minikdc-krb5.conf (from previous patch) etc.

          Show
          Ishan Chattopadhyaya added a comment - Updating the patch. Some minor changes to the jaas configuration for tests, SSL properties for the tests, got rid of the minikdc-krb5.conf (from previous patch) etc.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updated patch based on some changes for SOLR-7274 (mainly AuthenticationPlugin is now an abstract class instead of interface).

          Show
          Ishan Chattopadhyaya added a comment - Updated patch based on some changes for SOLR-7274 (mainly AuthenticationPlugin is now an abstract class instead of interface).
          Hide
          Ishan Chattopadhyaya added a comment -

          Fixing a packaging issue; the NoContext class wasn't available for SolrJ's packaging.

          Show
          Ishan Chattopadhyaya added a comment - Fixing a packaging issue; the NoContext class wasn't available for SolrJ's packaging.
          Hide
          Anshum Gupta added a comment -

          I've changed the TestSolrCloudWithKerberos to only test a 1 node - 1 shard setup as anything more hits the minikdc issue (HADOOP-9893), leading to test failure.
          I've also added a test to index and query on the newly created collection in the Kerberized environment.

          Show
          Anshum Gupta added a comment - I've changed the TestSolrCloudWithKerberos to only test a 1 node - 1 shard setup as anything more hits the minikdc issue ( HADOOP-9893 ), leading to test failure. I've also added a test to index and query on the newly created collection in the Kerberized environment.
          Hide
          Anshum Gupta added a comment -

          I've tested this manually, will spend another day doing so and all tests pass.
          If there are no objections, I'd like to commit this soon.

          Also, if someone has a strong reason to put this in contrib vs main code base, it'd be good to know.

          Show
          Anshum Gupta added a comment - I've tested this manually, will spend another day doing so and all tests pass. If there are no objections, I'd like to commit this soon. Also, if someone has a strong reason to put this in contrib vs main code base, it'd be good to know.
          Hide
          Ishan Chattopadhyaya added a comment -

          Been testing this for 1-2 days, there were minor hiccups, but all with SOLR-7274 and not this one. Updating the patch to rename the system properties needed with a "solr.kerberos" prefix.

          Show
          Ishan Chattopadhyaya added a comment - Been testing this for 1-2 days, there were minor hiccups, but all with SOLR-7274 and not this one. Updating the patch to rename the system properties needed with a "solr.kerberos" prefix.
          Hide
          Anshum Gupta added a comment -

          Thanks Ishan. I've been testing this too.
          The .solr prefix makes sense.

          I'll commit this after running the tests one last time.

          Show
          Anshum Gupta added a comment - Thanks Ishan. I've been testing this too. The .solr prefix makes sense. I'll commit this after running the tests one last time.
          Hide
          Anshum Gupta added a comment -

          A few changes.
          TestSolrCloudWithKerberos fails when running the entire test suite but passes everytime when running alone, even with the same seed + locale + ... combination. Here's the seed and other info to re-run:

          ant test -Dtestcase=TestSolrCloudWithKerberos -Dtests.method=testKerberizedSolr -Dtests.seed=BAEC87E7FCC3630 -Dtests.slow=true -Dtests.locale=is -Dtests.timezone=America/St_Johns -Dtests.asserts=true -Dtests.file.encoding=UTF-8

          I'm assuming some other test is tripping something here. Looking at how to fix this.

          Show
          Anshum Gupta added a comment - A few changes. TestSolrCloudWithKerberos fails when running the entire test suite but passes everytime when running alone, even with the same seed + locale + ... combination. Here's the seed and other info to re-run: ant test -Dtestcase=TestSolrCloudWithKerberos -Dtests.method=testKerberizedSolr -Dtests.seed=BAEC87E7FCC3630 -Dtests.slow=true -Dtests.locale=is -Dtests.timezone=America/St_Johns -Dtests.asserts=true -Dtests.file.encoding=UTF-8 I'm assuming some other test is tripping something here. Looking at how to fix this.
          Hide
          Anshum Gupta added a comment -

          Both me and Ishan have spent more than some time trying to fix this. The test passes by itself (with the same seed). We also tried comparing the System props that are set at the start of this test in both the cases and there wasn't anything that stood out other than -flush being adding during the standalone run to the sun.java.command property. I intend to add a @SuppressSSL to this test now and document it.

          Show
          Anshum Gupta added a comment - Both me and Ishan have spent more than some time trying to fix this. The test passes by itself (with the same seed). We also tried comparing the System props that are set at the start of this test in both the cases and there wasn't anything that stood out other than -flush being adding during the standalone run to the sun.java.command property. I intend to add a @SuppressSSL to this test now and document it.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681001 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1681001 ]

          SOLR-7468: Kerberos plugin for authentication framework. This will enable using Kerberos for authentication in Solr.

          Show
          ASF subversion and git services added a comment - Commit 1681001 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1681001 ] SOLR-7468 : Kerberos plugin for authentication framework. This will enable using Kerberos for authentication in Solr.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681009 from Anshum Gupta in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1681009 ]

          SOLR-7468: Kerberos plugin for authentication framework. This will enable using Kerberos for authentication in Solr.(merge from trunk)

          Show
          ASF subversion and git services added a comment - Commit 1681009 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1681009 ] SOLR-7468 : Kerberos plugin for authentication framework. This will enable using Kerberos for authentication in Solr.(merge from trunk)
          Hide
          Anshum Gupta added a comment -

          Thanks Ishan Chattopadhyaya, Greg Chanan, and Noble.

          Show
          Anshum Gupta added a comment - Thanks Ishan Chattopadhyaya , Greg Chanan, and Noble.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681198 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1681198 ]

          SOLR-7468: Fix the Kerberos test to use a reconfigured client always.

          Show
          ASF subversion and git services added a comment - Commit 1681198 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1681198 ] SOLR-7468 : Fix the Kerberos test to use a reconfigured client always.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681220 from Anshum Gupta in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1681220 ]

          SOLR-7468: Fix the Kerberos test to use a reconfigured client always.(merge from trunk)

          Show
          ASF subversion and git services added a comment - Commit 1681220 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1681220 ] SOLR-7468 : Fix the Kerberos test to use a reconfigured client always.(merge from trunk)
          Hide
          ASF subversion and git services added a comment -

          Commit 1681226 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_2'
          [ https://svn.apache.org/r1681226 ]

          SOLR-7468: Fix the Kerberos test to use a reconfigured client always.(merge from branch_5x)

          Show
          ASF subversion and git services added a comment - Commit 1681226 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_2' [ https://svn.apache.org/r1681226 ] SOLR-7468 : Fix the Kerberos test to use a reconfigured client always.(merge from branch_5x)
          Hide
          ASF subversion and git services added a comment -

          Commit 1681413 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1681413 ]

          SOLR-7468: Close the cloud client created for test in a finally block.

          Show
          ASF subversion and git services added a comment - Commit 1681413 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1681413 ] SOLR-7468 : Close the cloud client created for test in a finally block.
          Hide
          Ishan Chattopadhyaya added a comment -

          New patch, attempting to debug/fix the test failure.
          1. Starts just 1 jetty, so as not to hit HADOOP-9893.
          2. Better error reporting for missing kerberos params.
          3. Enabled debug logging of kerberos connections (solr.jaas.debug) during the tests.

          Show
          Ishan Chattopadhyaya added a comment - New patch, attempting to debug/fix the test failure. 1. Starts just 1 jetty, so as not to hit HADOOP-9893 . 2. Better error reporting for missing kerberos params. 3. Enabled debug logging of kerberos connections (solr.jaas.debug) during the tests.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681597 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1681597 ]

          SOLR-7468: Enabling debug logging for kerberos connections during tests and trying to fix # of jettys (shards)

          Show
          ASF subversion and git services added a comment - Commit 1681597 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1681597 ] SOLR-7468 : Enabling debug logging for kerberos connections during tests and trying to fix # of jettys (shards)
          Hide
          Anshum Gupta added a comment -

          Here's a patch that ignores the current test and adds a new test.
          Considering that the setup mechanism here is different, this test should run cleanly.

          It might however still hit the HADOOP issue, but I want to try this approach.
          I'll run this test locally for a few hours before committing.

          Show
          Anshum Gupta added a comment - Here's a patch that ignores the current test and adds a new test. Considering that the setup mechanism here is different, this test should run cleanly. It might however still hit the HADOOP issue, but I want to try this approach. I'll run this test locally for a few hours before committing.
          Hide
          Ishan Chattopadhyaya added a comment -

          It seems, sometimes, hadoop-auth is unable to apply the DEFAULT name rule to principals. Possible reason could be that it cannot determine minikdc's default realm. Due to this, there are 500 errors in the tests.

          I've explicitly added a name rule which should suffice for the tests. Also, enabling back the original test (which Anshum Gupta disabled in previous patch), so it can be tested with this name rules fix.

          Show
          Ishan Chattopadhyaya added a comment - It seems, sometimes, hadoop-auth is unable to apply the DEFAULT name rule to principals. Possible reason could be that it cannot determine minikdc's default realm. Due to this, there are 500 errors in the tests. I've explicitly added a name rule which should suffice for the tests. Also, enabling back the original test (which Anshum Gupta disabled in previous patch), so it can be tested with this name rules fix.
          Hide
          Anshum Gupta added a comment -

          The test suite + independent test passed for me with the fix. Since I figured that it may have been passing for me due to my settings in the /etc/krb5.conf, I've moved that file and I'm running the tests without that file.

          Show
          Anshum Gupta added a comment - The test suite + independent test passed for me with the fix. Since I figured that it may have been passing for me due to my settings in the /etc/krb5.conf, I've moved that file and I'm running the tests without that file.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updating the patch. Now client and server use different principals (solr@EXAMPLE.COM and HTTP/127.0.0.1@EXAMPLE.COM).

          I'm running this patch at my jenkins:
          1. Jenkins: http://162.244.24.210:8080/job/Anshum-Solr-7468/
          Source: https://github.com/anshumg/lucene-solr/tree/SOLR-7468
          2. Another version of this patch with external KDC instead of minikdc:
          Jenkins: http://162.244.24.210:8080/job/Anshum-Solr-7468-With-External-KDC/
          Source: https://github.com/anshumg/lucene-solr/tree/SOLR-7468-with-external-kdc

          So far both look to be passing at the moment, but I'll give it a few more runs before confirming. If (2) passes consistently, we can infer that there is some problem with minikdc that is causing the failures.

          Gregory Chanan, do you have any experience testing the kerberos intergration of cloudera's solr with minikdc vs. external kdc? Do you see any problem with the SOLR-7468's plugin/test code? Looking forward to your valuable inputs.

          Show
          Ishan Chattopadhyaya added a comment - Updating the patch. Now client and server use different principals (solr@EXAMPLE.COM and HTTP/127.0.0.1@EXAMPLE.COM). I'm running this patch at my jenkins: 1. Jenkins: http://162.244.24.210:8080/job/Anshum-Solr-7468/ Source: https://github.com/anshumg/lucene-solr/tree/SOLR-7468 2. Another version of this patch with external KDC instead of minikdc: Jenkins: http://162.244.24.210:8080/job/Anshum-Solr-7468-With-External-KDC/ Source: https://github.com/anshumg/lucene-solr/tree/SOLR-7468-with-external-kdc So far both look to be passing at the moment, but I'll give it a few more runs before confirming. If (2) passes consistently, we can infer that there is some problem with minikdc that is causing the failures. Gregory Chanan , do you have any experience testing the kerberos intergration of cloudera's solr with minikdc vs. external kdc? Do you see any problem with the SOLR-7468 's plugin/test code? Looking forward to your valuable inputs.
          Hide
          Anshum Gupta added a comment -
          1. The test error doesn't seem to happen when using the external KDC.
          2. The internal KDC hit the error until it was patched (with the most recent patch)
          3. The internal KDC could now also be passing as we have a krb5.conf file on the same machine (as there's an external KDC running on that machine).
          4. I cleaned up Kerberos related stuff from my machine (moved the krb5.conf and even did a kdestroy) and ran the tests locally. They ran without hitting 500 for me which I was seeing before applying the patch.

          I'll commit this patch to trunk after this one run that I'm doing so I can also get the Solr Jenkins to test it out.

          Show
          Anshum Gupta added a comment - The test error doesn't seem to happen when using the external KDC. The internal KDC hit the error until it was patched (with the most recent patch) The internal KDC could now also be passing as we have a krb5.conf file on the same machine (as there's an external KDC running on that machine). I cleaned up Kerberos related stuff from my machine (moved the krb5.conf and even did a kdestroy) and ran the tests locally. They ran without hitting 500 for me which I was seeing before applying the patch. I'll commit this patch to trunk after this one run that I'm doing so I can also get the Solr Jenkins to test it out.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681778 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1681778 ]

          SOLR-7468: Added an alt. test, change for client and server to use different principals, and explicit addition of name.rules for test

          Show
          ASF subversion and git services added a comment - Commit 1681778 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1681778 ] SOLR-7468 : Added an alt. test, change for client and server to use different principals, and explicit addition of name.rules for test
          Hide
          ASF subversion and git services added a comment -

          Commit 1681792 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1681792 ]

          SOLR-7468: Ignoring the older test as the new one tests exactly the same thing but with less moving parts and bootstrapping.

          Show
          ASF subversion and git services added a comment - Commit 1681792 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1681792 ] SOLR-7468 : Ignoring the older test as the new one tests exactly the same thing but with less moving parts and bootstrapping.
          Hide
          Gregory Chanan added a comment -

          Gregory Chanan, do you have any experience testing the kerberos intergration of cloudera's solr with minikdc vs. external kdc? Do you see any problem with the SOLR-7468's plugin/test code? Looking forward to your valuable inputs.

          For solr proper, we use external kdc. The only thing we use the MiniKDC for is SOLR-6915. I should have time to look at this tomorrow.

          Show
          Gregory Chanan added a comment - Gregory Chanan, do you have any experience testing the kerberos intergration of cloudera's solr with minikdc vs. external kdc? Do you see any problem with the SOLR-7468 's plugin/test code? Looking forward to your valuable inputs. For solr proper, we use external kdc. The only thing we use the MiniKDC for is SOLR-6915 . I should have time to look at this tomorrow.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681826 from Anshum Gupta in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1681826 ]

          SOLR-7468: Merging commits to fix test issue from trunk. Commits merged: r1681413 r1681597 r1681778 r1681792

          Show
          ASF subversion and git services added a comment - Commit 1681826 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1681826 ] SOLR-7468 : Merging commits to fix test issue from trunk. Commits merged: r1681413 r1681597 r1681778 r1681792
          Hide
          Hoss Man added a comment -

          I've got a reproduce line for TestSolrCloudWithKerberosAlt that fails reliably for me on trunk as of r1681791...

          ant test  -Dtestcase=TestSolrCloudWithKerberosAlt -Dtests.method=testBasics -Dtests.seed=781DFD20AEC5E01F -Dtests.slow=true -Dtests.locale=th_TH -Dtests.timezone=America/Argentina/San_Juan -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
          

          note the frequently problematic th_TH locale

          Show
          Hoss Man added a comment - I've got a reproduce line for TestSolrCloudWithKerberosAlt that fails reliably for me on trunk as of r1681791... ant test -Dtestcase=TestSolrCloudWithKerberosAlt -Dtests.method=testBasics -Dtests.seed=781DFD20AEC5E01F -Dtests.slow=true -Dtests.locale=th_TH -Dtests.timezone=America/Argentina/San_Juan -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 note the frequently problematic th_TH locale
          Hide
          Hoss Man added a comment -

          This seems to fail regardless of seed...

          ant test  -Dtestcase=TestSolrCloudWithKerberosAlt -Dtests.slow=true -Dtests.locale=th_TH -Dtests.asserts=true
          

          ...smells like some third party lib has a thai lowercasing/uppercasing bug in it. the question is does this affect the code? (ie: the broken library is in solr and we need a good error check in solr itself w/a clean errir) or is the bug in a library we only use for testing (in which case we just need a test assumes)

          Show
          Hoss Man added a comment - This seems to fail regardless of seed... ant test -Dtestcase=TestSolrCloudWithKerberosAlt -Dtests.slow=true -Dtests.locale=th_TH -Dtests.asserts=true ...smells like some third party lib has a thai lowercasing/uppercasing bug in it. the question is does this affect the code? (ie: the broken library is in solr and we need a good error check in solr itself w/a clean errir) or is the bug in a library we only use for testing (in which case we just need a test assumes)
          Hide
          Gregory Chanan added a comment - - edited

          related to SOLR-7183?

          Show
          Gregory Chanan added a comment - - edited related to SOLR-7183 ?
          Hide
          Ishan Chattopadhyaya added a comment - - edited

          Thanks Hoss Man. It looks like it is due to broken minikdc's dates for thai locale. I opened a new issue SOLR-7598 to fix this, but just only now realized that SOLR-7183 already existed (apologies!). I've tested this locale with an external KDC just now, and there seems no issue with the code. It is just the test setup (minikdc) that is affected.

          Show
          Ishan Chattopadhyaya added a comment - - edited Thanks Hoss Man . It looks like it is due to broken minikdc's dates for thai locale. I opened a new issue SOLR-7598 to fix this, but just only now realized that SOLR-7183 already existed (apologies!). I've tested this locale with an external KDC just now, and there seems no issue with the code. It is just the test setup (minikdc) that is affected.
          Hide
          ASF subversion and git services added a comment -

          Commit 1681926 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_2'
          [ https://svn.apache.org/r1681926 ]

          SOLR-7468: Merging commits to fix test issue from trunk. Commits merged: r1681413 r1681597 r1681778 r1681792

          Show
          ASF subversion and git services added a comment - Commit 1681926 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_2' [ https://svn.apache.org/r1681926 ] SOLR-7468 : Merging commits to fix test issue from trunk. Commits merged: r1681413 r1681597 r1681778 r1681792
          Hide
          Gregory Chanan added a comment - - edited

          Sorry for the delay, I took a look at this. Some notes below:

          1) Great work Ishan Chattopadhyaya! So glad to see this in Apache Solr.
          2) The KerberosFilter should either check that kerberos is actually enabled (via "type") or be a private nested class of the KerberosPlugin, to ensure it is only used with Kerberos. That can be handled as a separate jira.
          3) I'm a little concerned with the "NoContext" code in KerberosPlugin moving forward (I understand this is more a generic auth question than kerberos specific). For example, in the latest version of the filter we are using at Cloudera, we play around with the ServletContext in order to pass information around (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106). Is there any way we can get the actual ServletContext in a plugin? Again, this doesn't need to change right now.
          4)

          /**
          43	 * Test 5 nodes Solr cluster with Kerberos plugin enabled.
          44	 * This test is Ignored right now as Mini KDC has a known bug that
          45	 * doesn't allow us to run multiple nodes on the same host.
          46	 * https://issues.apache.org/jira/browse/HADOOP-9893
          47	 */
          

          This description is a little confusing – it sounds like you can't run multiple MiniKDC nodes on one host, but you typically woldn't want to do that so I doubt that is the issue. What exactly is the issue?
          5)

          String jaas = "Client {\n"
          102	        + " com.sun.security.auth.module.Krb5LoginModule required\n"
          103	        + " useKeyTab=true\n"
          104	        + " keyTab=\""+keytabFile.getAbsolutePath()+"\"\n"
          105	        + " storeKey=true\n"
          106	        + " useTicketCache=false\n"
          107	        + " doNotPrompt=true\n"
          108	        + " debug=true\n"
          109	        + " principal=\""+principal+"\";\n" 
          110	        + "};\n"
          111	        + "Server {\n"
          112	        + " com.sun.security.auth.module.Krb5LoginModule required\n"
          113	        + " useKeyTab=true\n"
          114	        + " keyTab=\""+keytabFile.getAbsolutePath()+"\"\n"
          115	        + " storeKey=true\n"
          116	        + " doNotPrompt=true\n"
          117	        + " useTicketCache=false\n"
          118	        + " debug=true\n"
          119	        + " principal=\""+zkServerPrincipal+"\";\n" 
          120	        + "};\n";
          

          It would be nice if we could just create a jaas configuration and pass it to the client, like we do in SOLR-6915. Again, nothing that needs to change now, but having the jaas configuration management in one place (the KerberosTestUtil) is ideal, because that code is known to be fragile, i.e. different JVMs require different parameters, capitalization, etc. If we have that sort of code around in different tests we won't be able to handle that.
          6)

          httpClient.addRequestInterceptor(bufferedEntityInterceptor);

          I think I mentioned this in a previous JIRA, but it would be nice to do some more investigation to figure out if we can avoid this. The hadoop auth filter has some code where you can use a cookie to avoid re-doing the negotiate...obviously you'd only want to do that if ssl was enabled.

          Show
          Gregory Chanan added a comment - - edited Sorry for the delay, I took a look at this. Some notes below: 1) Great work Ishan Chattopadhyaya ! So glad to see this in Apache Solr. 2) The KerberosFilter should either check that kerberos is actually enabled (via "type") or be a private nested class of the KerberosPlugin, to ensure it is only used with Kerberos. That can be handled as a separate jira. 3) I'm a little concerned with the "NoContext" code in KerberosPlugin moving forward (I understand this is more a generic auth question than kerberos specific). For example, in the latest version of the filter we are using at Cloudera, we play around with the ServletContext in order to pass information around ( https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106 ). Is there any way we can get the actual ServletContext in a plugin? Again, this doesn't need to change right now. 4) /** 43 * Test 5 nodes Solr cluster with Kerberos plugin enabled. 44 * This test is Ignored right now as Mini KDC has a known bug that 45 * doesn't allow us to run multiple nodes on the same host. 46 * https: //issues.apache.org/jira/browse/HADOOP-9893 47 */ This description is a little confusing – it sounds like you can't run multiple MiniKDC nodes on one host, but you typically woldn't want to do that so I doubt that is the issue. What exactly is the issue? 5) String jaas = "Client {\n" 102 + " com.sun.security.auth.module.Krb5LoginModule required\n" 103 + " useKeyTab= true \n" 104 + " keyTab=\" "+keytabFile.getAbsolutePath()+" \ "\n" 105 + " storeKey= true \n" 106 + " useTicketCache= false \n" 107 + " doNotPrompt= true \n" 108 + " debug= true \n" 109 + " principal=\" "+principal+" \ ";\n" 110 + "};\n" 111 + "Server {\n" 112 + " com.sun.security.auth.module.Krb5LoginModule required\n" 113 + " useKeyTab= true \n" 114 + " keyTab=\" "+keytabFile.getAbsolutePath()+" \ "\n" 115 + " storeKey= true \n" 116 + " doNotPrompt= true \n" 117 + " useTicketCache= false \n" 118 + " debug= true \n" 119 + " principal=\" "+zkServerPrincipal+" \ ";\n" 120 + "};\n" ; It would be nice if we could just create a jaas configuration and pass it to the client, like we do in SOLR-6915 . Again, nothing that needs to change now, but having the jaas configuration management in one place (the KerberosTestUtil) is ideal, because that code is known to be fragile, i.e. different JVMs require different parameters, capitalization, etc. If we have that sort of code around in different tests we won't be able to handle that. 6) httpClient.addRequestInterceptor(bufferedEntityInterceptor); I think I mentioned this in a previous JIRA, but it would be nice to do some more investigation to figure out if we can avoid this. The hadoop auth filter has some code where you can use a cookie to avoid re-doing the negotiate...obviously you'd only want to do that if ssl was enabled.
          Hide
          Anshum Gupta added a comment -

          About #4:
          Without the ticket caching support, minikdc has issues when multiple clients try to get tickets for the same principal (from the same host).

          Show
          Anshum Gupta added a comment - About #4: Without the ticket caching support, minikdc has issues when multiple clients try to get tickets for the same principal (from the same host).
          Hide
          Anshum Gupta added a comment -

          Also, the link to your codebase for the SolrHadoopAuthenticationFilter seems internal as I can't get to it.

          Show
          Anshum Gupta added a comment - Also, the link to your codebase for the SolrHadoopAuthenticationFilter seems internal as I can't get to it.
          Hide
          Gregory Chanan added a comment -

          About #4: Without the ticket caching support, minikdc has issues when multiple clients try to get tickets for the same principal (from the same host).

          What is a client? A thread? I looked into upgrading the hadoop minikdc dependency a month or so back but a release wasn't ready. When I have some time I'll look again.

          Also, the link to your codebase for the SolrHadoopAuthenticationFilter seems internal as I can't get to it.

          Whoops my apology! I meant: https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106

          Show
          Gregory Chanan added a comment - About #4: Without the ticket caching support, minikdc has issues when multiple clients try to get tickets for the same principal (from the same host). What is a client? A thread? I looked into upgrading the hadoop minikdc dependency a month or so back but a release wasn't ready. When I have some time I'll look again. Also, the link to your codebase for the SolrHadoopAuthenticationFilter seems internal as I can't get to it. Whoops my apology! I meant: https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106
          Hide
          Anshum Gupta added a comment -

          Bulk close for 5.2.0.

          Show
          Anshum Gupta added a comment - Bulk close for 5.2.0.
          Hide
          Kai Zheng added a comment -

          Just in case it can help: Not sure Hadoop will be going to upgrade Hadoop-minikdc to use the latest ApacheDS as the project has shifted the Kerberos related effort to a standalone sub-project Apache Kerby. I just made a proposal to the Hadoop community to upgrade Hadoop-minikdc to update the relevant codes rebased on Kerby and wish it can happen. If so the desired credential cache will be available as Kerby has the nice support.

          Show
          Kai Zheng added a comment - Just in case it can help: Not sure Hadoop will be going to upgrade Hadoop-minikdc to use the latest ApacheDS as the project has shifted the Kerberos related effort to a standalone sub-project Apache Kerby . I just made a proposal to the Hadoop community to upgrade Hadoop-minikdc to update the relevant codes rebased on Kerby and wish it can happen. If so the desired credential cache will be available as Kerby has the nice support.

            People

            • Assignee:
              Anshum Gupta
              Reporter:
              Ishan Chattopadhyaya
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development