Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.2
    • Component/s: None
    • Labels:
      None

      Description

      It would be good to have Solr support different authentication protocols.
      To begin with, it'd be good to have support for kerberos and basic auth.

      Starting point for documentation: https://cwiki.apache.org/confluence/display/solr/Security

      1. SOLR-7274.patch
        3 kB
        Anshum Gupta
      2. SOLR-7274.patch
        29 kB
        Anshum Gupta
      3. SOLR-7274.patch
        31 kB
        Ishan Chattopadhyaya
      4. SOLR-7274.patch
        31 kB
        Ishan Chattopadhyaya
      5. SOLR-7274.patch
        31 kB
        Ishan Chattopadhyaya
      6. SOLR-7274.patch
        30 kB
        Ishan Chattopadhyaya
      7. SOLR-7274.patch
        31 kB
        Ishan Chattopadhyaya
      8. SOLR-7274.patch
        42 kB
        Ishan Chattopadhyaya
      9. SOLR-7274.patch
        45 kB
        Ishan Chattopadhyaya
      10. SOLR-7274.patch
        28 kB
        Ishan Chattopadhyaya
      11. SOLR-7274.patch
        16 kB
        Anshum Gupta
      12. SOLR-7274.patch
        17 kB
        Ishan Chattopadhyaya
      13. SOLR-7274.patch
        17 kB
        Ishan Chattopadhyaya
      14. SOLR-7274.patch
        17 kB
        Ishan Chattopadhyaya
      15. SOLR-7274.patch
        40 kB
        Ishan Chattopadhyaya
      16. SOLR-7274.patch
        39 kB
        Ishan Chattopadhyaya
      17. SOLR-7274.patch
        26 kB
        Ishan Chattopadhyaya
      18. SOLR-7274-reconfigure-sdf-httpclient.patch
        4 kB
        Ishan Chattopadhyaya
      19. SOLR-7274-reconfigure-sdf-httpclient.patch
        2 kB
        Ishan Chattopadhyaya
      20. SOLR-7274-reconfigure-sdf-httpclient.patch
        4 kB
        Ishan Chattopadhyaya
      21. SOLR-7274-reconfigure-sdf-httpclient.patch
        1 kB
        Anshum Gupta

        Issue Links

          Activity

          Hide
          Ishan Chattopadhyaya added a comment - - edited

          I am working on implementing pluggable authentication support, initially supporting Kerberos and Basic Auth mechanisms.

          Here's a high level design that I'm working towards:

          • An authentication layer, consisting of plugins for each of the supported mechanisms, needs to be written to be invoked before the SolrDispatchFilter.
          • The configuration as to which plugin to be used, or if at all a security mechanism is needed, could come from ZK.
          • Every plugin's configuration (e.g. a keytab file path, service principal for kerberos) could be done using System.getProperties().
          • This authentication layer should ensure that the request, which leaves this layer and gets propogated down the chain, must, at least, have a java.security.Principal object associated with the request.
          • This user principal could be used, for example, by any downstream authorization layer (SOLR-7275) to perform fine grained access control based on requests, resources etc.
          • As for inter-node requests, the interfaces should support both (a) inter-node requests authenticating using the original user principal (where possible); as well as (b) inter-node requests authenticating using a node's own service principal.

          (SOLR-4470 has some context for this with respect to basic auth.)

          Show
          Ishan Chattopadhyaya added a comment - - edited I am working on implementing pluggable authentication support, initially supporting Kerberos and Basic Auth mechanisms. Here's a high level design that I'm working towards: An authentication layer, consisting of plugins for each of the supported mechanisms, needs to be written to be invoked before the SolrDispatchFilter. The configuration as to which plugin to be used, or if at all a security mechanism is needed, could come from ZK. Every plugin's configuration (e.g. a keytab file path, service principal for kerberos) could be done using System.getProperties(). This authentication layer should ensure that the request, which leaves this layer and gets propogated down the chain, must, at least, have a java.security.Principal object associated with the request. This user principal could be used, for example, by any downstream authorization layer ( SOLR-7275 ) to perform fine grained access control based on requests, resources etc. As for inter-node requests, the interfaces should support both (a) inter-node requests authenticating using the original user principal (where possible); as well as (b) inter-node requests authenticating using a node's own service principal. ( SOLR-4470 has some context for this with respect to basic auth.)
          Hide
          Noble Paul added a comment -

          Users editing web.xml is not an option

          Show
          Noble Paul added a comment - Users editing web.xml is not an option
          Hide
          Ishan Chattopadhyaya added a comment -

          I would think this one time editing would be performed by a security conscious system administrator, not a "user" per se. However, if even that is not a good idea, then, in such a case, the configuration properties can be fetched from ZK. Though, doing that would mean we wouldn't be able to support SolrCloud.

          Show
          Ishan Chattopadhyaya added a comment - I would think this one time editing would be performed by a security conscious system administrator, not a "user" per se. However, if even that is not a good idea, then, in such a case, the configuration properties can be fetched from ZK. Though, doing that would mean we wouldn't be able to support SolrCloud.
          Hide
          Noble Paul added a comment -

          I would think this one time editing would be performed by a security conscious system administrator, not a "user" per se.

          We don't expose web.xml anymore. So it is not even something we would like to document. OTOH , A user would be hacking Solr to add a servlet filter . it could be an option for some esoteric custom authentication plugin. But , it cannot be an option that we document or recommend

          Show
          Noble Paul added a comment - I would think this one time editing would be performed by a security conscious system administrator, not a "user" per se. We don't expose web.xml anymore. So it is not even something we would like to document. OTOH , A user would be hacking Solr to add a servlet filter . it could be an option for some esoteric custom authentication plugin. But , it cannot be an option that we document or recommend
          Hide
          Anshum Gupta added a comment -

          +1 on what Noble says.

          Can we use what Cloudera does?
          Gregory Chanan, you might have something to say here.

          Ideally, we should not expose web.xml or any other jetty specific implementation detail to the end users. Perhaps configure this via environment variables?

          Show
          Anshum Gupta added a comment - +1 on what Noble says. Can we use what Cloudera does? Gregory Chanan , you might have something to say here. Ideally, we should not expose web.xml or any other jetty specific implementation detail to the end users. Perhaps configure this via environment variables?
          Hide
          Ishan Chattopadhyaya added a comment - - edited

          Sure, sounds good. We could do the configuration via environment variables. I've modified my initial comment to reflect this.

          Show
          Ishan Chattopadhyaya added a comment - - edited Sure, sounds good. We could do the configuration via environment variables. I've modified my initial comment to reflect this.
          Hide
          Noble Paul added a comment - - edited

          Let me give a thread dump of my thought process

          • We give an interface for an authentication plugin. The users can choose to use it or not use it (our basic impl must use it) . All it does is , return an instance of java.security.Principal. Solr would just set it to request.setAttribute("java.security.Principal", principalObj).
          • Solr would provide an interface the user can implement and we also give a mechanism to configure that.
          • If somebody wishes to implement this using a filter , they can still do the same without our plugin interface . Because, it just uses the servlet API. And, in that case they would NOT have an authentication plugin and Solr doesn't care . It only only cares about the request attribute. We will NOT have to support or document the filter mode. In the future, if we move away from a web container, all the plugins implemented using our API will be back compatible and we DO NOT have to offer any such guarantees to the filter based implementations
          • The authorization module would be passed the Principal and it can decide on how to authorize the Principal for the given request
          • Solr would give an API and a mechanism to configure the authorization plugin and . Solr will give a basic impl for the same .
          Show
          Noble Paul added a comment - - edited Let me give a thread dump of my thought process We give an interface for an authentication plugin. The users can choose to use it or not use it (our basic impl must use it) . All it does is , return an instance of java.security.Principal. Solr would just set it to request.setAttribute("java.security.Principal", principalObj) . Solr would provide an interface the user can implement and we also give a mechanism to configure that. If somebody wishes to implement this using a filter , they can still do the same without our plugin interface . Because, it just uses the servlet API. And, in that case they would NOT have an authentication plugin and Solr doesn't care . It only only cares about the request attribute. We will NOT have to support or document the filter mode. In the future, if we move away from a web container, all the plugins implemented using our API will be back compatible and we DO NOT have to offer any such guarantees to the filter based implementations The authorization module would be passed the Principal and it can decide on how to authorize the Principal for the given request Solr would give an API and a mechanism to configure the authorization plugin and . Solr will give a basic impl for the same .
          Hide
          Ishan Chattopadhyaya added a comment -

          Even the basic auth implementation could be a plugin, just like kerberos and others, as per the design. Even if someone has a servlet filter to be used as a authentication mechanism, it can be wrapped around into the authentication layer as a plugin (since plugins could have servlet filter semantics). I've updated the design in my initial comment to reflect this, https://issues.apache.org/jira/browse/SOLR-7274?focusedCommentId=14389336&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14389336

          Show
          Ishan Chattopadhyaya added a comment - Even the basic auth implementation could be a plugin, just like kerberos and others, as per the design. Even if someone has a servlet filter to be used as a authentication mechanism, it can be wrapped around into the authentication layer as a plugin (since plugins could have servlet filter semantics). I've updated the design in my initial comment to reflect this, https://issues.apache.org/jira/browse/SOLR-7274?focusedCommentId=14389336&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14389336
          Hide
          Gus Heck added a comment -

          Doesn't Apache Shiro do all of this and give you an ini file with which to configure everything? They also have a web-app filtering system too (also see http://shiro.apache.org/web.html#Web-EnablingandDisablingFilters)

          Show
          Gus Heck added a comment - Doesn't Apache Shiro do all of this and give you an ini file with which to configure everything? They also have a web-app filtering system too (also see http://shiro.apache.org/web.html#Web-EnablingandDisablingFilters )
          Hide
          Gregory Chanan added a comment -

          Can we use what Cloudera does? Gregory Chanan, you might have something to say here.

          Right now we edit the web.xml. Given that is going away, I don't have an objection to alternative configuration, whether in ZK, system props, some combination of those, etc. What I'm not sure about is how you will make the configuration general enough without mentioning Filters. I.e. will there be pre-approved authentication mechanisms? Will I be able to write my own?

          This discussion also seems focused on the server side. Is the client side considered outside the scope of this jira? (i'm thinking something like SOLR-6625, but SOLR-4470 is related).

          Here's a pointer to the server-side stuff we do at Cloudera. I'm eager to contribute (or help contribute) this as part of a new authentication module. I just want to make sure the pluggable authentication model is general enough for our use case.

          Our web.xml:
          https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/webapp/web/WEB-INF/web.xml
          This adds two filters: HostnameFilter and SolrHadoopAuthenticationFilter. Together these support:

          • basic auth
          • kerberos auth
          • proxy user support (like sudo, see https://hadoop.apache.org/docs/r1.2.1/Secure_Impersonation.html)
          • delegation token support (used for MR/spark related jobs: get an authentication token at the outset and use it throughout the job lifetime so you don't have to pass kerberos keytabs around the cluster)

          The Filters:
          https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/java/org/apache/solr/servlet/HostnameFilter.java
          https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java – Note this supports delegation tokens.

          Some tests around the various functional pieces:
          https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/test/org/apache/solr/servlet/SolrHadoopAuthenticationFilterTest.java
          https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/test/org/apache/solr/servlet/SolrHadoopAuthenticationFilterProxyUserTest.java
          https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/test/org/apache/solr/servlet/SolrHadoopAuthenticationFilterDelegationTokenTest.java
          https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/test/org/apache/solr/servlet/HostnameFilterTest.java

          Show
          Gregory Chanan added a comment - Can we use what Cloudera does? Gregory Chanan, you might have something to say here. Right now we edit the web.xml. Given that is going away, I don't have an objection to alternative configuration, whether in ZK, system props, some combination of those, etc. What I'm not sure about is how you will make the configuration general enough without mentioning Filters. I.e. will there be pre-approved authentication mechanisms? Will I be able to write my own? This discussion also seems focused on the server side. Is the client side considered outside the scope of this jira? (i'm thinking something like SOLR-6625 , but SOLR-4470 is related). Here's a pointer to the server-side stuff we do at Cloudera. I'm eager to contribute (or help contribute) this as part of a new authentication module. I just want to make sure the pluggable authentication model is general enough for our use case. Our web.xml: https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/webapp/web/WEB-INF/web.xml This adds two filters: HostnameFilter and SolrHadoopAuthenticationFilter. Together these support: basic auth kerberos auth proxy user support (like sudo, see https://hadoop.apache.org/docs/r1.2.1/Secure_Impersonation.html ) delegation token support (used for MR/spark related jobs: get an authentication token at the outset and use it throughout the job lifetime so you don't have to pass kerberos keytabs around the cluster) The Filters: https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/java/org/apache/solr/servlet/HostnameFilter.java https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java – Note this supports delegation tokens. Some tests around the various functional pieces: https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/test/org/apache/solr/servlet/SolrHadoopAuthenticationFilterTest.java https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/test/org/apache/solr/servlet/SolrHadoopAuthenticationFilterProxyUserTest.java https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/test/org/apache/solr/servlet/SolrHadoopAuthenticationFilterDelegationTokenTest.java https://github.com/cloudera/lucene-solr/blob/cdh5-4.4.0_5.3.2/solr/core/src/test/org/apache/solr/servlet/HostnameFilterTest.java
          Hide
          Ishan Chattopadhyaya added a comment -

          Gregory Chanan Thanks for pitching in! Cloudera's implementation was actually one of my starting points when I started looking into this.

          What I'm not sure about is how you will make the configuration general enough without mentioning Filters. I.e. will there be pre-approved authentication mechanisms? Will I be able to write my own?

          My thought was to have configurations actually mention the filters (which may deal with any authentication mechanism, not just preapproved ones), without the user having to add it to the web.xml. For instance (and this may look different in the implementation), a user could have a configuration as "HostnameFilter,SolrHadoopAuthenticationFilter" and the authentication layer (which might itself be a servlet filter) would call the doFilter() on each of the two filters.

          This discussion also seems focused on the server side. Is the client side considered outside the scope of this jira? (i'm thinking something like SOLR-6625, but SOLR-4470 is related).

          Client side configurations are in the scope of pluggable items for each authentication mechanism. My thought was that this issue (SOLR-7274) could leverage the callback "frameworks" of SOLR-6625, SOLR-4470 and focus on the pluggable aspects of the configurations for each authc mechanism.

          Show
          Ishan Chattopadhyaya added a comment - Gregory Chanan Thanks for pitching in! Cloudera's implementation was actually one of my starting points when I started looking into this. What I'm not sure about is how you will make the configuration general enough without mentioning Filters. I.e. will there be pre-approved authentication mechanisms? Will I be able to write my own? My thought was to have configurations actually mention the filters (which may deal with any authentication mechanism, not just preapproved ones), without the user having to add it to the web.xml. For instance (and this may look different in the implementation), a user could have a configuration as "HostnameFilter,SolrHadoopAuthenticationFilter" and the authentication layer (which might itself be a servlet filter) would call the doFilter() on each of the two filters. This discussion also seems focused on the server side. Is the client side considered outside the scope of this jira? (i'm thinking something like SOLR-6625 , but SOLR-4470 is related). Client side configurations are in the scope of pluggable items for each authentication mechanism. My thought was that this issue ( SOLR-7274 ) could leverage the callback "frameworks" of SOLR-6625 , SOLR-4470 and focus on the pluggable aspects of the configurations for each authc mechanism.
          Hide
          Ishan Chattopadhyaya added a comment -

          Doesn't Apache Shiro do all of this and give you an ini file with which to configure everything? They also have a web-app filtering system too (also see http://shiro.apache.org/web.html#Web-EnablingandDisablingFilters)

          I did have a look at Shiro, but my initial thought was that it might not fit our bill due to a couple of reasons:

          • Shiro doesn't have out of the box support for Kerberos
          • Shiro's commit patterns indicated that it is not a very active project at the moment, and hence I wasn't sure if having Solr depend on Shiro was a good idea.

          Maybe someone more experienced with Shiro might help me understand if this isn't true. Hadoop Common's hadoop-auth library seems easier to leverage here, esp. for Kerberos, and it is already a dependency for Solr.

          Show
          Ishan Chattopadhyaya added a comment - Doesn't Apache Shiro do all of this and give you an ini file with which to configure everything? They also have a web-app filtering system too (also see http://shiro.apache.org/web.html#Web-EnablingandDisablingFilters ) I did have a look at Shiro, but my initial thought was that it might not fit our bill due to a couple of reasons: Shiro doesn't have out of the box support for Kerberos Shiro's commit patterns indicated that it is not a very active project at the moment, and hence I wasn't sure if having Solr depend on Shiro was a good idea. Maybe someone more experienced with Shiro might help me understand if this isn't true. Hadoop Common's hadoop-auth library seems easier to leverage here, esp. for Kerberos, and it is already a dependency for Solr.
          Hide
          Ishan Chattopadhyaya added a comment -

          Added an initial patch, mainly focussing around the pluggable framework part.

          It defines an AuthenticationLayerFilter which is called just before the SolrDispatchFilter. The AuthenticationLayerFilter (during its init()) looks for a AuthenticationLayerPlugin classname from the following places, in order, (a) ZK's clusterprops.json containing an "authcPlugin" property, (b) System property "authcPlugin".

          AuthenticationLayerPlugin has the following interface:

          public interface AuthenticationLayerPlugin {
            
            public void init(FilterConfig conf);
            
            public void doAuthenticate(ServletRequest req, ServletResponse rsp,
                FilterChain chain) throws Exception;
          
            public HttpClientConfigurer getDefaultConfigurer();
          }
          

          A reference Kerberos authentication plugin, based on the hadoop-auth's AuthenticationFilter, is included in the patch.

          On the server side, the plugin contains a filter, called KerberosFilter, whose properties (e.g. keytab, service principal etc.) are populated from system properties.

          On the client side (also used for internode communication), an extended HttpClientConfigurer instance (derived from Cloudera's implementation) is used to configure the HttpClient used by HttpSolrClient. It uses a jaas config file from the system property "java.security.auth.login.config" containing client's keytab and user principal. For repeatable requests through the HttpSolrClient, I've wrapped around the request entities in BufferedHttpEntity. Currently, in this patch, no extra authentication metadata is being passed around from main request to sub request for now. Going forward, a generic callback framework might be more desireable, e.g. SOLR-6625.

          Here's how I start and test this:

          Here's my clusterprops.json in ZK, before starting solr (if one doesn't want to use ZK for this, could pass -DauthcPlugin=org.apache.solr.security.KerberosPlugin to start script):

          {"authcPlugin":"org.apache.solr.security.KerberosPlugin"}
          

          Here's my start command:

          bin/solr -c -z localhost:2181 -a "-Djava.security.auth.login.config=/home/ishan/jaas-client.conf 
          \     -Dcookie.domain=192.168.0.107 -Dkerberos.principal=HTTP/192.168.0.107@EXAMPLE.COM 
          \     -Dkerberos.keytab=/tmp/107solr.keytab"
          

          Here's my /home/ishan/jaas-client.conf file:

          SolrClient {
           com.sun.security.auth.module.Krb5LoginModule required
           useKeyTab=true
           keyTab="/tmp/107solr.keytab"
           storeKey=true
           useTicketCache=true
           debug=true
           principal="HTTP/192.168.0.107@EXAMPLE.COM"; 
          };
          

          Here's a way to test:

          kinit
          curl --negotiate -u : "http://192.168.0.107:8983/solr/"
          
          Show
          Ishan Chattopadhyaya added a comment - Added an initial patch, mainly focussing around the pluggable framework part. It defines an AuthenticationLayerFilter which is called just before the SolrDispatchFilter. The AuthenticationLayerFilter (during its init()) looks for a AuthenticationLayerPlugin classname from the following places, in order, (a) ZK's clusterprops.json containing an "authcPlugin" property, (b) System property "authcPlugin". AuthenticationLayerPlugin has the following interface: public interface AuthenticationLayerPlugin { public void init(FilterConfig conf); public void doAuthenticate(ServletRequest req, ServletResponse rsp, FilterChain chain) throws Exception; public HttpClientConfigurer getDefaultConfigurer(); } A reference Kerberos authentication plugin, based on the hadoop-auth's AuthenticationFilter, is included in the patch. On the server side, the plugin contains a filter, called KerberosFilter, whose properties (e.g. keytab, service principal etc.) are populated from system properties. On the client side (also used for internode communication), an extended HttpClientConfigurer instance (derived from Cloudera's implementation) is used to configure the HttpClient used by HttpSolrClient. It uses a jaas config file from the system property "java.security.auth.login.config" containing client's keytab and user principal. For repeatable requests through the HttpSolrClient, I've wrapped around the request entities in BufferedHttpEntity. Currently, in this patch, no extra authentication metadata is being passed around from main request to sub request for now. Going forward, a generic callback framework might be more desireable, e.g. SOLR-6625 . Here's how I start and test this: Here's my clusterprops.json in ZK, before starting solr (if one doesn't want to use ZK for this, could pass -DauthcPlugin=org.apache.solr.security.KerberosPlugin to start script): {"authcPlugin":"org.apache.solr.security.KerberosPlugin"} Here's my start command: bin/solr -c -z localhost:2181 -a "-Djava.security.auth.login.config=/home/ishan/jaas-client.conf \ -Dcookie.domain=192.168.0.107 -Dkerberos.principal=HTTP/192.168.0.107@EXAMPLE.COM \ -Dkerberos.keytab=/tmp/107solr.keytab" Here's my /home/ishan/jaas-client.conf file: SolrClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/tmp/107solr.keytab" storeKey=true useTicketCache=true debug=true principal="HTTP/192.168.0.107@EXAMPLE.COM"; }; Here's a way to test: kinit curl --negotiate -u : "http://192.168.0.107:8983/solr/"
          Hide
          Gregory Chanan added a comment -

          Some initial thoughts...

          --- solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java	(revision 1672548)
          +++ solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java	(working copy)
          @@ -215,7 +215,7 @@
                     if (urls.size() <= 1) {
                       String url = urls.get(0);
                       srsp.setShardAddress(url);
          -            try (SolrClient client = new HttpSolrClient(url, httpClient)) {
          +            try (SolrClient client = new HttpSolrClient(url)) {
                         ssr.nl = client.request(req);
                       }
                     } else {
          

          Why this change?

          2. Can we comment on the AuthenticationLayerPlugin? I don't think it's obvious what needs to be implemented

          3. params.put("token.valid", System.getProperty("kerberos.name.rules", "30"));
          This doesn't look correct

          4. Using "SolrClient" instead of whatever zookeeper uses (default "Client", see the code I linked above) for the jaas configuration app name will cause you to have to deal with two issues. I'm not against doing something different here, just pointing out that these problems need to be solved before you can make this change:
          A) kerberos tickets need to be refreshed. The zookeeper client automatically refreshes kerberos tickets, so if you just use the same configuration and app name, that's handled for you. If you want to use something different, you'll have to write code to refresh the tickets. This also means you only refresh the code when using zookeeper (i.e. SolrCloud)...you might want to pull out support for specifying the plugin via system props (so you know they have to be using solrcloud in order to read the zk props), or may want to add some error checking.
          B) assuming you want to use the same properties for talking to zookeeper as talking to other solr servers, you'll have to specify the jaas entry twice (once for SolrClient, once for Client or whatever zookeeper is using (ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY). Or you may be able to change how we handle zookeeper sasl (see SaslACLProvider, I think you'd have to write a Credentials provider?).
          As I said, I'm not against doing this, it just introduces additional issues for a first version. The code I linked above just uses whatever zookeeper users.

          5. In HttpSolrClient.java: "postOrPut.setEntity(new BufferedHttpEntity(postOrPut.getEntity()));"
          Why are we always buffering the http entities? That seems like something that should be handled by the authentication plugin, i.e. usually we don't buffer. If we are using kerberos, we set up a client configurer that is smart enough to handle the http requests for that authentication scheme (here buffering is probably fine for the initial version, there are some discussions of an optimization in SOLR-6625). See this comment in SOLR-6625 for some implementation ideas https://issues.apache.org/jira/browse/SOLR-6625?focusedCommentId=14238865&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14238865

          6. This doesn't seem to handle forwarding requests in the SolrDispatchFilter. There's more discussion in SOLR-6625, see here: https://issues.apache.org/jira/browse/SOLR-6625?focusedCommentId=14239957&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14239957
          I don't know how to solve that without implementing SOLR-6625, though.

          Show
          Gregory Chanan added a comment - Some initial thoughts... --- solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java (revision 1672548) +++ solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java (working copy) @@ -215,7 +215,7 @@ if (urls.size() <= 1) { String url = urls.get(0); srsp.setShardAddress(url); - try (SolrClient client = new HttpSolrClient(url, httpClient)) { + try (SolrClient client = new HttpSolrClient(url)) { ssr.nl = client.request(req); } } else { Why this change? 2. Can we comment on the AuthenticationLayerPlugin? I don't think it's obvious what needs to be implemented 3. params.put("token.valid", System.getProperty("kerberos.name.rules", "30")); This doesn't look correct 4. Using "SolrClient" instead of whatever zookeeper uses (default "Client", see the code I linked above) for the jaas configuration app name will cause you to have to deal with two issues. I'm not against doing something different here, just pointing out that these problems need to be solved before you can make this change: A) kerberos tickets need to be refreshed. The zookeeper client automatically refreshes kerberos tickets, so if you just use the same configuration and app name, that's handled for you. If you want to use something different, you'll have to write code to refresh the tickets. This also means you only refresh the code when using zookeeper (i.e. SolrCloud)...you might want to pull out support for specifying the plugin via system props (so you know they have to be using solrcloud in order to read the zk props), or may want to add some error checking. B) assuming you want to use the same properties for talking to zookeeper as talking to other solr servers, you'll have to specify the jaas entry twice (once for SolrClient, once for Client or whatever zookeeper is using (ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY). Or you may be able to change how we handle zookeeper sasl (see SaslACLProvider, I think you'd have to write a Credentials provider?). As I said, I'm not against doing this, it just introduces additional issues for a first version. The code I linked above just uses whatever zookeeper users. 5. In HttpSolrClient.java: "postOrPut.setEntity(new BufferedHttpEntity(postOrPut.getEntity()));" Why are we always buffering the http entities? That seems like something that should be handled by the authentication plugin, i.e. usually we don't buffer. If we are using kerberos, we set up a client configurer that is smart enough to handle the http requests for that authentication scheme (here buffering is probably fine for the initial version, there are some discussions of an optimization in SOLR-6625 ). See this comment in SOLR-6625 for some implementation ideas https://issues.apache.org/jira/browse/SOLR-6625?focusedCommentId=14238865&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14238865 6. This doesn't seem to handle forwarding requests in the SolrDispatchFilter. There's more discussion in SOLR-6625 , see here: https://issues.apache.org/jira/browse/SOLR-6625?focusedCommentId=14239957&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14239957 I don't know how to solve that without implementing SOLR-6625 , though.
          Hide
          Anshum Gupta added a comment -

          Thanks for putting up this patch Ishan.
          I just had a quick look and most of my concerns were covered by Gregory Chanan! Thanks for looking at this. It would be good to have more details about AuthenticationLayerPlugin and what needs to be implemented.

          3. params.put("token.valid", System.getProperty("kerberos.name.rules", "30"));

          This doesn't look correct

          This just seems like a typo.

          This change I assume is also just for the purpose of testing/dev, right?

          -log4j.logger.org.apache.hadoop=WARN
          +log4j.logger.org.apache.hadoop=TRACE
          

          About forwarding of requests, do you think we could borrow the code to send a repeatable request beforehand in case of POST/PUT? Or does it make sense to fix SOLR-6625 ?

          Minor but can you also fix the parameters to more understandable names instead of arg(0|1|..)?

          Most importantly, unless I'm missing out on something, are you propagating the userPrincipal out of the plugin and back to Solr?

          Show
          Anshum Gupta added a comment - Thanks for putting up this patch Ishan. I just had a quick look and most of my concerns were covered by Gregory Chanan ! Thanks for looking at this. It would be good to have more details about AuthenticationLayerPlugin and what needs to be implemented. 3. params.put("token.valid", System.getProperty("kerberos.name.rules", "30")); This doesn't look correct This just seems like a typo. This change I assume is also just for the purpose of testing/dev, right? -log4j.logger.org.apache.hadoop=WARN +log4j.logger.org.apache.hadoop=TRACE About forwarding of requests, do you think we could borrow the code to send a repeatable request beforehand in case of POST/PUT? Or does it make sense to fix SOLR-6625 ? Minor but can you also fix the parameters to more understandable names instead of arg(0|1|..)? Most importantly, unless I'm missing out on something, are you propagating the userPrincipal out of the plugin and back to Solr?
          Hide
          Ishan Chattopadhyaya added a comment - - edited

          Some initial thoughts...

          Gregory Chanan, thanks a lot for your review!

          • try (SolrClient client = new HttpSolrClient(url, httpClient)) {
            + try (SolrClient client = new HttpSolrClient(url)) {
            Why this change?

          I couldn't think of a way to configure the httpclient used by the HttpShardHandler for inter node communication. This was because by the time the plugin's init() did a HttpClientUtils.setConfigurer() with the plugin's httpclient configurer, the default httpclient used by the HttpShardHandler and its factory was already configured by the stock HttpClientConfigurer. So, to go ahead with the rest of my testing, I made had used this hack, which in effect created a new httpclient for every request (and thus used the plugin's configurer).

          Now, I've fixed this 1 by adding a reconfigureHttpClient() method for the ShardHandlerFactory.

          2. Can we comment on the AuthenticationLayerPlugin? I don't think it's obvious what needs to be implemented

          I've added some javadocs 1.

          public interface AuthenticationLayerPlugin {
          
            /**
             * This is called upon loading up of a plugin, used for setting it up.
             * @param filterConfig used for any configs passed in from the servlet filter 
             * config, or for access to the servlet context.
             * @param zkController the zk controller
             */
            public void init(FilterConfig filterConfig, ZkController zkController);
           
            /**
             * This method must authenticate the request. Upon a successful authentication, this 
             * must call the next filter in the filter chain and set the user principal of the request,
             * or else, upon an error or an authentication failure, throw an exception.
             * 
             * @param request the http request
             * @param response the http response
             * @param filterChain the servlet filter chain
             * @throws Exception any exception thrown during the authentication, e.g. 
             * PriviledgedAccessException
             */
            public void doAuthenticate(ServletRequest request, ServletResponse response,
                FilterChain filterChain) throws Exception;
          
            /**
             * 
             * @return Returns an instance of a HttpClientConfigurer to be used for configuring the
             * httpclients for use during internode communication.
             */
            public HttpClientConfigurer getDefaultConfigurer();
          }
          

          Do you think it is clearer now? Also, do you have any questions or suggestions for a change to the interface?

          3. params.put("token.valid", System.getProperty("kerberos.name.rules", "30"));
          This doesn't look correct

          Oops, fixed 1. Over zealous copy-pasting.

          5. In HttpSolrClient.java: "postOrPut.setEntity(new BufferedHttpEntity(postOrPut.getEntity()));"
          ...
          we set up a client configurer that is smart enough to handle the http requests for that authentication scheme

          Thanks for the pointer! I had put it in there, thinking of ways to fix it in a subsequent patch. The HttpRequestInterceptor seems like a much cleaner way than I had initially planned for. I've updated the patch to reflect this 1.

          6. This doesn't seem to handle forwarding requests in the SolrDispatchFilter.
          ...
          I don't know how to solve that without implementing SOLR-6625, though.

          After the change to add a reconfigureHttpClient() to ShardHandlerFactory 1, the SolrDispatchFilter's remoteQuery() now works. Do you see any problem with that in the patch 1 / have I missed something?

          4. ....
          Using "SolrClient" instead of whatever zookeeper uses (default "Client", see the code I linked above) for the jaas configuration app name will cause you to have to deal with two issues.

          The reason why I went with a different app name for ZK and solr clients in the patch was that in my dev testing environment, zk wasn't kerberized, and hence I wanted to use plain unsecured connection to zk, while using kerberos based authc for Solr. I don't know how users actually setup their deployments of ZK and Solr, so I am looking for suggestions on how to go about it. I wasn't aware of the side effect of refreshing of tickets, so that seems like a great benefit. Do you think the plugin should should support both (1. same jaas configs for zk & solr clients, and 2. different jaas configs for zk and solr clients), configureably? Of course, that would involve code for refreshing credentials, but maybe some generic hooks for this in an "pluggable authentication framework" might be handy anyway?

          [1] I'll add the patch shortly

          Show
          Ishan Chattopadhyaya added a comment - - edited Some initial thoughts... Gregory Chanan , thanks a lot for your review! try (SolrClient client = new HttpSolrClient(url, httpClient)) { + try (SolrClient client = new HttpSolrClient(url)) { Why this change? I couldn't think of a way to configure the httpclient used by the HttpShardHandler for inter node communication. This was because by the time the plugin's init() did a HttpClientUtils.setConfigurer() with the plugin's httpclient configurer, the default httpclient used by the HttpShardHandler and its factory was already configured by the stock HttpClientConfigurer. So, to go ahead with the rest of my testing , I made had used this hack, which in effect created a new httpclient for every request (and thus used the plugin's configurer). Now, I've fixed this 1 by adding a reconfigureHttpClient() method for the ShardHandlerFactory. 2. Can we comment on the AuthenticationLayerPlugin? I don't think it's obvious what needs to be implemented I've added some javadocs 1 . public interface AuthenticationLayerPlugin { /** * This is called upon loading up of a plugin, used for setting it up. * @param filterConfig used for any configs passed in from the servlet filter * config, or for access to the servlet context. * @param zkController the zk controller */ public void init(FilterConfig filterConfig, ZkController zkController); /** * This method must authenticate the request. Upon a successful authentication, this * must call the next filter in the filter chain and set the user principal of the request, * or else, upon an error or an authentication failure, throw an exception. * * @param request the http request * @param response the http response * @param filterChain the servlet filter chain * @throws Exception any exception thrown during the authentication, e.g. * PriviledgedAccessException */ public void doAuthenticate(ServletRequest request, ServletResponse response, FilterChain filterChain) throws Exception; /** * * @return Returns an instance of a HttpClientConfigurer to be used for configuring the * httpclients for use during internode communication. */ public HttpClientConfigurer getDefaultConfigurer(); } Do you think it is clearer now? Also, do you have any questions or suggestions for a change to the interface? 3. params.put("token.valid", System.getProperty("kerberos.name.rules", "30")); This doesn't look correct Oops, fixed 1 . Over zealous copy-pasting. 5. In HttpSolrClient.java: "postOrPut.setEntity(new BufferedHttpEntity(postOrPut.getEntity()));" ... we set up a client configurer that is smart enough to handle the http requests for that authentication scheme Thanks for the pointer! I had put it in there, thinking of ways to fix it in a subsequent patch. The HttpRequestInterceptor seems like a much cleaner way than I had initially planned for. I've updated the patch to reflect this 1 . 6. This doesn't seem to handle forwarding requests in the SolrDispatchFilter. ... I don't know how to solve that without implementing SOLR-6625 , though. After the change to add a reconfigureHttpClient() to ShardHandlerFactory 1 , the SolrDispatchFilter's remoteQuery() now works. Do you see any problem with that in the patch 1 / have I missed something? 4. .... Using "SolrClient" instead of whatever zookeeper uses (default "Client", see the code I linked above) for the jaas configuration app name will cause you to have to deal with two issues. The reason why I went with a different app name for ZK and solr clients in the patch was that in my dev testing environment, zk wasn't kerberized, and hence I wanted to use plain unsecured connection to zk, while using kerberos based authc for Solr. I don't know how users actually setup their deployments of ZK and Solr, so I am looking for suggestions on how to go about it. I wasn't aware of the side effect of refreshing of tickets, so that seems like a great benefit. Do you think the plugin should should support both (1. same jaas configs for zk & solr clients, and 2. different jaas configs for zk and solr clients), configureably? Of course, that would involve code for refreshing credentials, but maybe some generic hooks for this in an "pluggable authentication framework" might be handy anyway? [1] I'll add the patch shortly
          Hide
          Ishan Chattopadhyaya added a comment - - edited

          Thanks Anshum Gupta for your review!

          This change I assume is also just for the purpose of testing/dev, right?

          Oh yes, and it was very helpful. But, I've removed it from the next patch.

          About forwarding of requests, do you think we could borrow the code to send a repeatable request beforehand in case of POST/PUT? Or does it make sense to fix SOLR-6625 ?

          POST/PUT requests are implementations of HttpEntityEnclosingRequestBase, and in this patch, I've wrapped such entities inside a BufferedHttpEntity (in Krb5HttpClientConfigurer) to take care of repeatable requests.

          Most importantly, unless I'm missing out on something, are you propagating the userPrincipal out of the plugin and back to Solr?

          The plugin should set the user principal to the request so that req.getUserPrincipal() returns a javax.security.Principal object. The way this can be done is to use a HttpServletRequestWrapper in the plugin. Would you instead prefer that we use an internal solr header / parameter for passing along the user principal?

          Show
          Ishan Chattopadhyaya added a comment - - edited Thanks Anshum Gupta for your review! This change I assume is also just for the purpose of testing/dev, right? Oh yes, and it was very helpful. But, I've removed it from the next patch. About forwarding of requests, do you think we could borrow the code to send a repeatable request beforehand in case of POST/PUT? Or does it make sense to fix SOLR-6625 ? POST/PUT requests are implementations of HttpEntityEnclosingRequestBase, and in this patch, I've wrapped such entities inside a BufferedHttpEntity (in Krb5HttpClientConfigurer) to take care of repeatable requests. Most importantly, unless I'm missing out on something, are you propagating the userPrincipal out of the plugin and back to Solr? The plugin should set the user principal to the request so that req.getUserPrincipal() returns a javax.security.Principal object. The way this can be done is to use a HttpServletRequestWrapper in the plugin. Would you instead prefer that we use an internal solr header / parameter for passing along the user principal?
          Hide
          Ishan Chattopadhyaya added a comment - - edited

          Added an updated patch (still WIP). This has:

          • Using HttpRequestInterceptor to wrap up the entity into BufferedHttpEntity
          • Adding a zk controller instance to a plugin's init()
          • Adding support to reconfigure the already configured httpclients of ShardHandlerFactory using the plugin's httpclient configurer. This is for the httpclients used during internode communication.

          Still TODO:

          • Tests
          • SolrCLI, bin/solr script to use authentication
          • Refreshing kerberos credentials (for the kerberos plugin). Maybe go with cloudera's approach, to keep things simple, by using the zk jaas configs for solr clients?
          • ...
          Show
          Ishan Chattopadhyaya added a comment - - edited Added an updated patch (still WIP). This has: Using HttpRequestInterceptor to wrap up the entity into BufferedHttpEntity Adding a zk controller instance to a plugin's init() Adding support to reconfigure the already configured httpclients of ShardHandlerFactory using the plugin's httpclient configurer. This is for the httpclients used during internode communication. Still TODO: Tests SolrCLI, bin/solr script to use authentication Refreshing kerberos credentials (for the kerberos plugin). Maybe go with cloudera's approach, to keep things simple, by using the zk jaas configs for solr clients? ...
          Hide
          Anshum Gupta added a comment -

          Yet to look at the patch but you shouldn't be passing zkController instance to the init. You might want to piggy back on what I'm trying to do in SOLR-7275 i.e. read and pass the JSON/Map to the init instead of the zkController.
          I'll provide more feedback once I have a look at the patch.

          Show
          Anshum Gupta added a comment - Yet to look at the patch but you shouldn't be passing zkController instance to the init. You might want to piggy back on what I'm trying to do in SOLR-7275 i.e. read and pass the JSON/Map to the init instead of the zkController. I'll provide more feedback once I have a look at the patch.
          Hide
          Ishan Chattopadhyaya added a comment -

          Splitting out the framework (here) and the kerberos plugin (SOLR-7468). Here's a patch with just the plugin framework.

          Show
          Ishan Chattopadhyaya added a comment - Splitting out the framework (here) and the kerberos plugin ( SOLR-7468 ). Here's a patch with just the plugin framework.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updating the patch to trunk.

          I'm working on integrating the ZK configuration part of SOLR-7275. Right now, to use authc, the "authcPlugin" system property can be used. This will go away in a subsequent patch.

          Here's an example of starting Solr with SOLR-7468 also applied:

          bin/solr -c -z localhost:2181 -a "-Djava.security.auth.login.config=/home/ishan/jaas-client.conf -Dcookie.domain=192.168.0.107 -Dkerberos.principal=HTTP/192.168.0.107@EXAMPLE.COM -Dkerberos.keytab=/tmp/107solr.keytab -DauthcPlugin=org.apache.solr.security.KerberosPlugin"
          
          Show
          Ishan Chattopadhyaya added a comment - Updating the patch to trunk. I'm working on integrating the ZK configuration part of SOLR-7275 . Right now, to use authc, the "authcPlugin" system property can be used. This will go away in a subsequent patch. Here's an example of starting Solr with SOLR-7468 also applied: bin/solr -c -z localhost:2181 -a "-Djava.security.auth.login.config=/home/ishan/jaas-client.conf -Dcookie.domain=192.168.0.107 -Dkerberos.principal=HTTP/192.168.0.107@EXAMPLE.COM -Dkerberos.keytab=/tmp/107solr.keytab -DauthcPlugin=org.apache.solr.security.KerberosPlugin"
          Hide
          Don Bosco Durai added a comment -

          Ishan Chattopadhyaya, I had some issues running with the above syntax. The code is expecting that -Dsolr.hdfs.security.kerberos.keytabfile passed. I have not yet traced through the code, but maybe some other defaults are kicking in and expecting the above property.

          My working syntax is below. I think, -Dkerberos.keytab is not getting into affect.

          bin/solr -c -z localhost:2181 -a "-Djava.security.auth.login.config=$HOME/conf/jaas-client.conf -Dcookie.domain=`hostname -f` -Dkerberos.principal=HTTP/`hostname -f`@EXAMPLE.COM -Dkerberos.keytab=/keytabs/http.keytab -DauthcPlugin=org.apache.solr.security.KerberosPlugin -Dsolr.hdfs.security.kerberos.keytabfile=/keytabs/http.keytab"

          Show
          Don Bosco Durai added a comment - Ishan Chattopadhyaya , I had some issues running with the above syntax. The code is expecting that -Dsolr.hdfs.security.kerberos.keytabfile passed. I have not yet traced through the code, but maybe some other defaults are kicking in and expecting the above property. My working syntax is below. I think, -Dkerberos.keytab is not getting into affect. bin/solr -c -z localhost:2181 -a "-Djava.security.auth.login.config=$HOME/conf/jaas-client.conf -Dcookie.domain=`hostname -f` -Dkerberos.principal=HTTP/`hostname -f`@EXAMPLE.COM -Dkerberos.keytab=/keytabs/http.keytab -DauthcPlugin=org.apache.solr.security.KerberosPlugin -Dsolr.hdfs.security.kerberos.keytabfile=/keytabs/http.keytab"
          Hide
          Ishan Chattopadhyaya added a comment -

          Updating the patch, now uses ZK changes (/security.json) from SOLR-7275. Please apply SOLR-7275 patch first before this.

          Show
          Ishan Chattopadhyaya added a comment - Updating the patch, now uses ZK changes (/security.json) from SOLR-7275 . Please apply SOLR-7275 patch first before this.
          Hide
          Don Bosco Durai added a comment -

          Ishan Chattopadhyaya, thanks for uploading the updated patch. Can you give the sample json for configuring the kerberos plugin?

          Thanks

          Show
          Don Bosco Durai added a comment - Ishan Chattopadhyaya , thanks for uploading the updated patch. Can you give the sample json for configuring the kerberos plugin? Thanks
          Hide
          Ishan Chattopadhyaya added a comment -

          I've used stuff from SOLR-7275, so it should be same format as Anshum mentions here:
          https://issues.apache.org/jira/browse/SOLR-7275?focusedCommentId=14497128&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14497128

          Something like:

          {"authorization":
            {"class":"solr.SimpleSolrAuthorizationPlugin",
            "deny":["user1","user2"]
            },
            "authentication":
            {"class":"org.apache.solr.security.KerberosPlugin",
             "conf1": "val1", ...
            }
          }
          

          The kerberos plugin (SOLR-7468) doesn't require any other config than "class" at the moment. All the other config parameters are host specific and are picked up from system properties.

          Show
          Ishan Chattopadhyaya added a comment - I've used stuff from SOLR-7275 , so it should be same format as Anshum mentions here: https://issues.apache.org/jira/browse/SOLR-7275?focusedCommentId=14497128&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14497128 Something like: {"authorization": {"class":"solr.SimpleSolrAuthorizationPlugin", "deny":["user1","user2"] }, "authentication": {"class":"org.apache.solr.security.KerberosPlugin", "conf1": "val1", ... } } The kerberos plugin ( SOLR-7468 ) doesn't require any other config than "class" at the moment. All the other config parameters are host specific and are picked up from system properties.
          Hide
          Anshum Gupta added a comment -

          Thanks for the patch Ishan. Here's some feedback:

          1. You've converted protected methods to be static protected, that doesn't sound right to me. It makes more sense to merge the AuthenticationLayerFilter with SDF and not change those methods. SDF is just a wrapper now and so this should be easy to plugin. I'm not sure but it's perhaps just a matter of moving the code into SDF.
          2. There are unused imports in your code, you should clean them up.
          Show
          Anshum Gupta added a comment - Thanks for the patch Ishan. Here's some feedback: You've converted protected methods to be static protected, that doesn't sound right to me. It makes more sense to merge the AuthenticationLayerFilter with SDF and not change those methods. SDF is just a wrapper now and so this should be easy to plugin. I'm not sure but it's perhaps just a matter of moving the code into SDF. There are unused imports in your code, you should clean them up.
          Hide
          Anshum Gupta added a comment -

          Got rid of the AuthenticationLayerFilter and moved all of that into SDF. Also cleaned up unused imports.
          Another pair of eyes would be good to confirm that it's all good. Ishan Chattopadhyaya and Gregory Chanan?

          Show
          Anshum Gupta added a comment - Got rid of the AuthenticationLayerFilter and moved all of that into SDF. Also cleaned up unused imports. Another pair of eyes would be good to confirm that it's all good. Ishan Chattopadhyaya and Gregory Chanan ?
          Hide
          Ishan Chattopadhyaya added a comment - - edited

          Thanks Anshum Gupta for the patch. I think it should be possible to fold in all the changes from the extra "AuthenticationLayerFilter" to SDF. However, at this point, I am unable to get Kerberos plugin (SOLR-7468) to work with this due to the fact that the hadoop-auth's AuthenticationFilter (which the plugin is based on) does a chain.doFilter() for the next filter/servlet in the chain, and this doesn't work since SDF is the only filter in the chain. I'll try to fix that for SOLR-7468 later, and then fold in your changes.

          Meanwhile, I'm putting out a patch which doesn't contain your changes, and does the following:

          • SolrCLI, bin/solr support for authentication options. (TODO: Windows bin/solr.cmd changes)
          • Fixed a bug where the CUSC wasn't getting configured using the plugin's client configurer.

          Note: I have added code from my patch for SOLR-7545 (which was a blocker for this issue), which I'll remove from this patch as soon as it gets committed.

          Show
          Ishan Chattopadhyaya added a comment - - edited Thanks Anshum Gupta for the patch. I think it should be possible to fold in all the changes from the extra "AuthenticationLayerFilter" to SDF. However, at this point, I am unable to get Kerberos plugin ( SOLR-7468 ) to work with this due to the fact that the hadoop-auth's AuthenticationFilter (which the plugin is based on) does a chain.doFilter() for the next filter/servlet in the chain, and this doesn't work since SDF is the only filter in the chain. I'll try to fix that for SOLR-7468 later, and then fold in your changes. Meanwhile, I'm putting out a patch which doesn't contain your changes, and does the following: SolrCLI, bin/solr support for authentication options. (TODO: Windows bin/solr.cmd changes) Fixed a bug where the CUSC wasn't getting configured using the plugin's client configurer. Note: I have added code from my patch for SOLR-7545 (which was a blocker for this issue), which I'll remove from this patch as soon as it gets committed.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updated patch.

          • Adding a test, TestAuthenticationFramework.
          • Included changes in latest patch for SOLR-7545 (without the bin/solr.cmd change). Once SOLR-7545 is committed, some of bin/solr changes should be not necessary.

          TODO:
          bin/solr.cmd changes for Windows.

          Show
          Ishan Chattopadhyaya added a comment - Updated patch. Adding a test, TestAuthenticationFramework. Included changes in latest patch for SOLR-7545 (without the bin/solr.cmd change). Once SOLR-7545 is committed, some of bin/solr changes should be not necessary. TODO: bin/solr.cmd changes for Windows.
          Hide
          Noble Paul added a comment -

          From the discussions and looking at the code , I believe it's totally possible to make this work without the extra filter. Committing it without that change will be lame.
          Please rename the classes to drop the "Layer" from class names

          Show
          Noble Paul added a comment - From the discussions and looking at the code , I believe it's totally possible to make this work without the extra filter. Committing it without that change will be lame. Please rename the classes to drop the "Layer" from class names
          Hide
          Ishan Chattopadhyaya added a comment -

          Updating the previous patch, now with some changes made in SOLR-7468 and with changes committed in SOLR-7545.

          Noble Paul, I'll look at folding the extra filter (AuthenticationLayerFilter) into SDF next. Thanks for your review.

          Show
          Ishan Chattopadhyaya added a comment - Updating the previous patch, now with some changes made in SOLR-7468 and with changes committed in SOLR-7545 . Noble Paul , I'll look at folding the extra filter (AuthenticationLayerFilter) into SDF next. Thanks for your review.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updating the patch. Now got rid of the extra filter and folded in all the initialization code into the core container.

          Show
          Ishan Chattopadhyaya added a comment - Updating the patch. Now got rid of the extra filter and folded in all the initialization code into the core container.
          Hide
          Ishan Chattopadhyaya added a comment -

          Updated patch after some cleanup. Also put this up at https://reviews.apache.org/r/34400 for any line-by-line review comments.

          Show
          Ishan Chattopadhyaya added a comment - Updated patch after some cleanup. Also put this up at https://reviews.apache.org/r/34400 for any line-by-line review comments.
          Hide
          Noble Paul added a comment -

          it's possible to move the excludedPatterns check before authc , right ?

          Show
          Noble Paul added a comment - it's possible to move the excludedPatterns check before authc , right ?
          Hide
          Anshum Gupta added a comment -

          A bit of cleanup and I think this is good to go in. I'll commit this after running the tests one last time.

          Show
          Anshum Gupta added a comment - A bit of cleanup and I think this is good to go in. I'll commit this after running the tests one last time.
          Hide
          ASF subversion and git services added a comment -

          Commit 1680391 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1680391 ]

          SOLR-7274: Pluggable authentication module in Solr. This defines an interface and a mechanism to create, load, and use an Authentication plugin.

          Show
          ASF subversion and git services added a comment - Commit 1680391 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1680391 ] SOLR-7274 : Pluggable authentication module in Solr. This defines an interface and a mechanism to create, load, and use an Authentication plugin.
          Hide
          ASF subversion and git services added a comment -

          Commit 1680415 from Anshum Gupta in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1680415 ]

          SOLR-7274: Pluggable authentication module in Solr. This defines an interface and a mechanism to create, load, and use an Authentication plugin.(merge from trunk)

          Show
          ASF subversion and git services added a comment - Commit 1680415 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1680415 ] SOLR-7274 : Pluggable authentication module in Solr. This defines an interface and a mechanism to create, load, and use an Authentication plugin.(merge from trunk)
          Hide
          Anshum Gupta added a comment -

          Thanks Ishan, this is good.
          Also, thanks Noble, and Greg.

          Show
          Anshum Gupta added a comment - Thanks Ishan, this is good. Also, thanks Noble, and Greg.
          Hide
          Anshum Gupta added a comment -

          We also need to reconfigure the HttpClient in SDF after the coreContainer (authc plugin init) have happened.

          Show
          Anshum Gupta added a comment - We also need to reconfigure the HttpClient in SDF after the coreContainer (authc plugin init) have happened.
          Hide
          Anshum Gupta added a comment - - edited

          ignore this patch! I'm getting an NPE with this on the admin UI (with the SOLR-7468 patch). fixing this. It might be something with the Kerberos patch too.

          Show
          Anshum Gupta added a comment - - edited ignore this patch! I'm getting an NPE with this on the admin UI (with the SOLR-7468 patch). fixing this. It might be something with the Kerberos patch too.
          Hide
          Anshum Gupta added a comment -

          One more fix.

          Show
          Anshum Gupta added a comment - One more fix.
          Hide
          Ishan Chattopadhyaya added a comment -

          The HttpSolrCall's remoteQuery() was blindly copying the headers from original request to the forwarded request, including the "Host" header, which should be avoided for SPNego to work.

          Show
          Ishan Chattopadhyaya added a comment - The HttpSolrCall's remoteQuery() was blindly copying the headers from original request to the forwarded request, including the "Host" header, which should be avoided for SPNego to work.
          Hide
          Ishan Chattopadhyaya added a comment -

          And adding more offending headers ("Authorization" and "Accept", along with "Host") that were forwarded, which were screwing up SPNego for forwarded requests. Updated the patch.

          Show
          Ishan Chattopadhyaya added a comment - And adding more offending headers ("Authorization" and "Accept", along with "Host") that were forwarded, which were screwing up SPNego for forwarded requests. Updated the patch.
          Hide
          ASF subversion and git services added a comment -

          Commit 1680931 from Anshum Gupta in branch 'dev/trunk'
          [ https://svn.apache.org/r1680931 ]

          SOLR-7274: Stop forwarding Authorization, Host, and Accept headers for SPNego to work. Also fixed an exception forwarding from SDF and reconfigure SDF's httpClient after authentication has been initialized.

          Show
          ASF subversion and git services added a comment - Commit 1680931 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1680931 ] SOLR-7274 : Stop forwarding Authorization, Host, and Accept headers for SPNego to work. Also fixed an exception forwarding from SDF and reconfigure SDF's httpClient after authentication has been initialized.
          Hide
          ASF subversion and git services added a comment -

          Commit 1680940 from Anshum Gupta in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1680940 ]

          SOLR-7274: Stop forwarding Authorization, Host, and Accept headers for SPNego to work. Also fixed an exception forwarding from SDF and reconfigure SDF's httpClient after authentication has been initialized.(merge from trunk)

          Show
          ASF subversion and git services added a comment - Commit 1680940 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1680940 ] SOLR-7274 : Stop forwarding Authorization, Host, and Accept headers for SPNego to work. Also fixed an exception forwarding from SDF and reconfigure SDF's httpClient after authentication has been initialized.(merge from trunk)
          Hide
          ASF subversion and git services added a comment -

          Commit 1684268 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_2'
          [ https://svn.apache.org/r1684268 ]

          SOLR-7623: backporting for Solr 5.2.1 (commit was actually a part of SOLR-7274)

          Show
          ASF subversion and git services added a comment - Commit 1684268 from Anshum Gupta in branch 'dev/branches/lucene_solr_5_2' [ https://svn.apache.org/r1684268 ] SOLR-7623 : backporting for Solr 5.2.1 (commit was actually a part of SOLR-7274 )
          Hide
          ASF subversion and git services added a comment -

          Commit 1685289 from Ramkumar Aiyengar in branch 'dev/trunk'
          [ https://svn.apache.org/r1685289 ]

          SOLR-7274: Removed a few eager string constructions from log.debug

          Show
          ASF subversion and git services added a comment - Commit 1685289 from Ramkumar Aiyengar in branch 'dev/trunk' [ https://svn.apache.org/r1685289 ] SOLR-7274 : Removed a few eager string constructions from log.debug
          Hide
          ASF subversion and git services added a comment -

          Commit 1685290 from Ramkumar Aiyengar in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1685290 ]

          SOLR-7274: Removed a few eager string constructions from log.debug

          Show
          ASF subversion and git services added a comment - Commit 1685290 from Ramkumar Aiyengar in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1685290 ] SOLR-7274 : Removed a few eager string constructions from log.debug
          Hide
          Anshum Gupta added a comment -

          Thanks for cleaning this up Ram but it's recommended to commit changes related to an already released issue as a new JIRA .

          Show
          Anshum Gupta added a comment - Thanks for cleaning this up Ram but it's recommended to commit changes related to an already released issue as a new JIRA .
          Hide
          Ramkumar Aiyengar added a comment -

          Sure, normally would have, just didn't bother for a couple of log lines

          Show
          Ramkumar Aiyengar added a comment - Sure, normally would have, just didn't bother for a couple of log lines
          Hide
          Anshum Gupta added a comment -

          Bulk close for 5.2.0.

          Show
          Anshum Gupta added a comment - Bulk close for 5.2.0.

            People

            • Assignee:
              Anshum Gupta
              Reporter:
              Anshum Gupta
            • Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development