Uploaded image for project: 'Apache Knox'
  1. Apache Knox
  2. KNOX-644

Limit/page results of LDAP group membership search

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.10.0
    • Component/s: Server
    • Labels:
      None

      Description

      Some users are finding that they have >1000 groups that would be returned given how Knox currently implements group lookup. ActiveDirectory currently limits search results to 1000 items and this causes failures that require workarounds at the client side. Ideally Knox's LDAP group search implementation would either limit/filter the results or page the result set that are unavoidably large.

      1. paging.patch
        34 kB
        Kevin Risden
      2. KNOX-644-paging.patch
        10 kB
        Kevin Risden
      3. KNOX-644.patch
        28 kB
        Kevin Risden
      4. create_groups_ldif.py
        0.7 kB
        Kevin Risden
      5. ad_setup.ps1
        0.7 kB
        Kevin Risden

        Issue Links

          Activity

          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ccd2fd4b3e3229b8b333053a439300ee6220edb0 in knox's branch refs/heads/v0.9.0 from Larry McCay
          [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=ccd2fd4 ]

          KNOX-644 - Limit/page results of LDAP group membership search (Kevin Risden via lmccay)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ccd2fd4b3e3229b8b333053a439300ee6220edb0 in knox's branch refs/heads/v0.9.0 from Larry McCay [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=ccd2fd4 ] KNOX-644 - Limit/page results of LDAP group membership search (Kevin Risden via lmccay)
          Hide
          risdenk Kevin Risden added a comment -

          Larry McCay - Awesome thanks! This commit should also close KNOX-508 since the update to ApacheDS got pulled in.

          Show
          risdenk Kevin Risden added a comment - Larry McCay - Awesome thanks! This commit should also close KNOX-508 since the update to ApacheDS got pulled in.
          Hide
          lmccay Larry McCay added a comment -

          Hi Kevin Risden - I managed to manually test this against an AD instance with 1000+ groups and it worked nicely!
          I've subsequently pushed this to master and will also backport it to the 0.9.0 line for any maintenance releases that come out of there.

          Thank you for this contribution - this has been a recent pain point for some users!

          Show
          lmccay Larry McCay added a comment - Hi Kevin Risden - I managed to manually test this against an AD instance with 1000+ groups and it worked nicely! I've subsequently pushed this to master and will also backport it to the 0.9.0 line for any maintenance releases that come out of there. Thank you for this contribution - this has been a recent pain point for some users!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 62a75e0e015ae35f27a08e156f792b6e5517fb6a in knox's branch refs/heads/master from Larry McCay
          [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=62a75e0 ]

          KNOX-644 - Limit/page results of LDAP group membership search (Kevin Risden via lmccay)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 62a75e0e015ae35f27a08e156f792b6e5517fb6a in knox's branch refs/heads/master from Larry McCay [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=62a75e0 ] KNOX-644 - Limit/page results of LDAP group membership search (Kevin Risden via lmccay)
          Hide
          risdenk Kevin Risden added a comment -

          I was able to test it with the embedded ApacheDS server. I had to change the following:

          Add server.setMaxSizeLimit(LdapServer.NO_SIZE_LIMIT); to SimpleLdapDirectoryServer constructor and connect with the admin user instead of sam/sam-password.

          Those two changes allowed the paging to work with the embedded ApacheDS server.

          I would love to put up a comprehensive patch with tests, but won't get to it this week most likely.

          Show
          risdenk Kevin Risden added a comment - I was able to test it with the embedded ApacheDS server. I had to change the following: Add server.setMaxSizeLimit(LdapServer.NO_SIZE_LIMIT); to SimpleLdapDirectoryServer constructor and connect with the admin user instead of sam/sam-password. Those two changes allowed the paging to work with the embedded ApacheDS server. I would love to put up a comprehensive patch with tests, but won't get to it this week most likely.
          Hide
          lmccay Larry McCay added a comment -

          Hi Kevin Risden - I have been reviewing your patches and trying to test them and am unsure whether you actually think we can even manually test with the demo LDAP server. You point above to some ApacheDS code that indicates that paging is possible under some constraints.

          Would you have happened to actually test it?

          I would really like to get paging in while we work on more efficient searches but need to know how to actually test it - hopefully without having to stand up an AD instance to do so.

          Show
          lmccay Larry McCay added a comment - Hi Kevin Risden - I have been reviewing your patches and trying to test them and am unsure whether you actually think we can even manually test with the demo LDAP server. You point above to some ApacheDS code that indicates that paging is possible under some constraints. Would you have happened to actually test it? I would really like to get paging in while we work on more efficient searches but need to know how to actually test it - hopefully without having to stand up an AD instance to do so.
          Hide
          lmccay Larry McCay added a comment -

          Agreed, I would like to see the Hadoop Group Mappings used across all of the components in the ecosystem.
          This will require work in each project and I am going to try and contribute to them in order t drive this consistency.

          What we need to determine is where is best to invest our time for this release in order to address the worst pain points.
          If Knox 0.10.0 was able to leverage the same SSD/Centrify/etc mechanisms that you have experience with, would it address the performance/limited group issues?

          Show
          lmccay Larry McCay added a comment - Agreed, I would like to see the Hadoop Group Mappings used across all of the components in the ecosystem. This will require work in each project and I am going to try and contribute to them in order t drive this consistency. What we need to determine is where is best to invest our time for this release in order to address the worst pain points. If Knox 0.10.0 was able to leverage the same SSD/Centrify/etc mechanisms that you have experience with, would it address the performance/limited group issues?
          Hide
          risdenk Kevin Risden added a comment -

          Larry McCay - I like the KIP looks like a good approach. I need to subscribe to the knox-dev/knox-user mailing lists still to comment on the thread you started for it.

          Currently for other Hadoop services, the clusters have been configured with the default Hadoop group mapping and not to use LDAP directly. This is because the cluster nodes have been configured to get groups from LDAP (via SSSD/Centrify/etc).

          Even if the Hadoop Groups Mapping solves the issue, it would be great for the group lookup code amongst Hadoop projects to be similar (Ranger, Ambari, Knox, etc). Each of them do group lookup slightly differently and causes annoyances. Ranger and Ambari do basically the same type of group lookup and don't use the Hadoop Group Mapping.

          Show
          risdenk Kevin Risden added a comment - Larry McCay - I like the KIP looks like a good approach. I need to subscribe to the knox-dev/knox-user mailing lists still to comment on the thread you started for it. Currently for other Hadoop services, the clusters have been configured with the default Hadoop group mapping and not to use LDAP directly. This is because the cluster nodes have been configured to get groups from LDAP (via SSSD/Centrify/etc). Even if the Hadoop Groups Mapping solves the issue, it would be great for the group lookup code amongst Hadoop projects to be similar (Ranger, Ambari, Knox, etc). Each of them do group lookup slightly differently and causes annoyances. Ranger and Ambari do basically the same type of group lookup and don't use the Hadoop Group Mapping.
          Hide
          lmccay Larry McCay added a comment -

          Kevin Risden - I've created a KIP for trying to rationalize what we need to do for LDAP improvements in the 0.10.0 release. See: https://cwiki.apache.org/confluence/display/KNOX/KIP-1+LDAP+Improvements

          One thing that I am wondering is whether the integration of the Hadoop Groups Mapping facility into Knox would satisfy much of the performance and limits pain that we are experiencing. Can you tell me whether you have similar issues with group lookup inside of Hadoop and which groups mapping implementation you generally use?

          Show
          lmccay Larry McCay added a comment - Kevin Risden - I've created a KIP for trying to rationalize what we need to do for LDAP improvements in the 0.10.0 release. See: https://cwiki.apache.org/confluence/display/KNOX/KIP-1+LDAP+Improvements One thing that I am wondering is whether the integration of the Hadoop Groups Mapping facility into Knox would satisfy much of the performance and limits pain that we are experiencing. Can you tell me whether you have similar issues with group lookup inside of Hadoop and which groups mapping implementation you generally use?
          Hide
          risdenk Kevin Risden added a comment -

          Few more thoughts about how to improve, won't get back to this for a few days:

          • Should make one query for groups and member instead of one search for groups and then a search for each member in the group
            • This would make 1 ldap search instead of n ldap searches - 1 for each group
            • This should simplify group lookup quite a bit
          Show
          risdenk Kevin Risden added a comment - Few more thoughts about how to improve, won't get back to this for a few days: Should make one query for groups and member instead of one search for groups and then a search for each member in the group This would make 1 ldap search instead of n ldap searches - 1 for each group This should simplify group lookup quite a bit
          Hide
          risdenk Kevin Risden added a comment -

          Looks like ApacheDS supports paging, but only in a few scenarios.

          • You are administrator with proper client side paging configured
          • You are a user and the server limit is set to NO_LIMIT

          https://github.com/apache/directory-server/blob/trunk/server-integ/src/test/java/org/apache/directory/server/operations/search/PagedSearchIT.java

          Show
          risdenk Kevin Risden added a comment - Looks like ApacheDS supports paging, but only in a few scenarios. You are administrator with proper client side paging configured You are a user and the server limit is set to NO_LIMIT https://github.com/apache/directory-server/blob/trunk/server-integ/src/test/java/org/apache/directory/server/operations/search/PagedSearchIT.java
          Hide
          risdenk Kevin Risden added a comment -

          To apply KNOX-644-paging.patch, apply KNOX-644.patch first.

          Show
          risdenk Kevin Risden added a comment - To apply KNOX-644 -paging.patch, apply KNOX-644 .patch first.
          Hide
          risdenk Kevin Risden added a comment -

          Larry McCay - Thanks for the quick attention to this.

          I'm not sure that we need to improve the ApacheDS DEMO LDAP server here but perhaps there is no other reasonable way to test for the handling of large results without doing so. We'll have to consider that carefully.

          This is definitely something that could be interesting. The paging of results doesn't seem to work with the ApacheDS demo ldap server. Also for KNOX-461 there is no support for computed attributes currently in ApacheDS.

          • We need to add some unit tests for each feature/improvement

          Agree on the unit tests. The existing tests with expanded user.ldif actually caught the problem. Wasn't sure if it made sense to add another test class for the large number of groups.

          • Both patches seem to be adding the additional groups to user.ldif - if this is the same set and we don't need additional for paging then we should just have the paging one assume that the the other one was already applied

          Yes both patches are standalone right now. I'll post the paging patch as an addon to the KNOX-644 one.

          Show
          risdenk Kevin Risden added a comment - Larry McCay - Thanks for the quick attention to this. I'm not sure that we need to improve the ApacheDS DEMO LDAP server here but perhaps there is no other reasonable way to test for the handling of large results without doing so. We'll have to consider that carefully. This is definitely something that could be interesting. The paging of results doesn't seem to work with the ApacheDS demo ldap server. Also for KNOX-461 there is no support for computed attributes currently in ApacheDS. We need to add some unit tests for each feature/improvement Agree on the unit tests. The existing tests with expanded user.ldif actually caught the problem. Wasn't sure if it made sense to add another test class for the large number of groups. Both patches seem to be adding the additional groups to user.ldif - if this is the same set and we don't need additional for paging then we should just have the paging one assume that the the other one was already applied Yes both patches are standalone right now. I'll post the paging patch as an addon to the KNOX-644 one.
          Hide
          lmccay Larry McCay added a comment -

          Couple quick review comments on the patches:

          • We need to add some unit tests for each feature/improvement
          • Both patches seem to be adding the additional groups to user.ldif - if this is the same set and we don't need additional for paging then we should just have the paging one assume that the the other one was already applied
          Show
          lmccay Larry McCay added a comment - Couple quick review comments on the patches: We need to add some unit tests for each feature/improvement Both patches seem to be adding the additional groups to user.ldif - if this is the same set and we don't need additional for paging then we should just have the paging one assume that the the other one was already applied
          Hide
          lmccay Larry McCay added a comment -

          Kevin Risden - thank you for your work on this!
          I think these patches are important and should be targeted at our 0.10.0 (maybe 1.0.0) release.
          One of the larger feature themes that I was thinking of addressing in 0.10.0 is LDAP improvements.
          These will fit in well with those.

          I'm not sure that we need to improve the ApacheDS DEMO LDAP server here but perhaps there is no other reasonable way to test for the handling of large results without doing so. We'll have to consider that carefully.

          I am going to set the Fix Version to 0.10.0 for tracking it for the next feature bearing release.

          Show
          lmccay Larry McCay added a comment - Kevin Risden - thank you for your work on this! I think these patches are important and should be targeted at our 0.10.0 (maybe 1.0.0) release. One of the larger feature themes that I was thinking of addressing in 0.10.0 is LDAP improvements. These will fit in well with those. I'm not sure that we need to improve the ApacheDS DEMO LDAP server here but perhaps there is no other reasonable way to test for the handling of large results without doing so. We'll have to consider that carefully. I am going to set the Fix Version to 0.10.0 for tracking it for the next feature bearing release.
          Hide
          risdenk Kevin Risden added a comment -

          Attaching my Powershell script for testing AD. Creates a single user, a configurable number of groups, and adds that user to those created groups.

          Show
          risdenk Kevin Risden added a comment - Attaching my Powershell script for testing AD. Creates a single user, a configurable number of groups, and adds that user to those created groups.
          Hide
          risdenk Kevin Risden added a comment - - edited

          Attaching the paging patch. There are a few things that happen related to this patch:

          • Had to update to Apache DS 2.0.0-M16 for paging
          • With AD could take a while if referrals is set to followed
            • Hardcoded to ignore right now
            • Should make this configurable like Ranger, Ambari, and others
          • If there are more results, AD causes ldap search to throw PartialResultsException
            • Ignoring this since it doesn't seem to cause any problems
          • The ApacheDS LDAP server doesn't allow more than 100 results returned even with paging
          • With AD, if there are more than 1000 results paging will allow a search
          • Currently logging is the print statements

          Probably makes sense to use memberOf instead of paging through groups - see KNOX-461.

          Would love some comments if this is a good approach or not. I can clean up the logging if this makes sense.

          Show
          risdenk Kevin Risden added a comment - - edited Attaching the paging patch. There are a few things that happen related to this patch: Had to update to Apache DS 2.0.0-M16 for paging See KNOX-508 for details With AD could take a while if referrals is set to followed Hardcoded to ignore right now Should make this configurable like Ranger, Ambari, and others If there are more results, AD causes ldap search to throw PartialResultsException Ignoring this since it doesn't seem to cause any problems The ApacheDS LDAP server doesn't allow more than 100 results returned even with paging With AD, if there are more than 1000 results paging will allow a search Currently logging is the print statements Probably makes sense to use memberOf instead of paging through groups - see KNOX-461 . Would love some comments if this is a good approach or not. I can clean up the logging if this makes sense.
          Hide
          risdenk Kevin Risden added a comment -

          Attaching the simple python file I used to generate the list of groups for users.ldif.

          Example to use it:

          # Create 110 groups and copy to clipboard on Mac
          ./create_groups_ldif.py 110 | pbcopy
          
          Show
          risdenk Kevin Risden added a comment - Attaching the simple python file I used to generate the list of groups for users.ldif. Example to use it: # Create 110 groups and copy to clipboard on Mac ./create_groups_ldif.py 110 | pbcopy
          Hide
          risdenk Kevin Risden added a comment -

          I also have a patch that implements paging, but that doesn't seem to help the issue of >100 groups with the embedded LDAP server. I haven't tested the paging against AD yet. For that patch with paging, it is going to require KNOX-508. For AD specifically, it would make sense to solve it with KNOX-461 and avoid the searching each group for the user.

          Show
          risdenk Kevin Risden added a comment - I also have a patch that implements paging, but that doesn't seem to help the issue of >100 groups with the embedded LDAP server. I haven't tested the paging against AD yet. For that patch with paging, it is going to require KNOX-508 . For AD specifically, it would make sense to solve it with KNOX-461 and avoid the searching each group for the user.
          Hide
          risdenk Kevin Risden added a comment -

          Retrieving >100 groups from the embedded LDAP server causes the same error if you request >1000 from AD. This patch adds extra groups to users.ldif, catches the SizeLimitExceededException, and prints out a message with number of groups captured. This avoids the 500 error and at least checks the groups that were able to be returned.

          The one question I had was how should logging be handled for this?

          For reference the stack trace this avoid is:

           WARN [org.eclipse.jetty.servlet.ServletHandler] 
          javax.servlet.ServletException: org.apache.shiro.authz.AuthorizationException: LDAP naming error while attempting to retrieve authorization for user [sam].
          	at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
          	at org.apache.shiro.web.filter.authc.AuthenticatingFilter.cleanup(AuthenticatingFilter.java:155)
          	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
          	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
          	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
          	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
          	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
          	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
          	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
          	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
          	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
          	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
          	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
          	at org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:50)
          	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
          	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
          	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
          	at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:139)
          	at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:91)
          	at org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:138)
          	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
          	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
          	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
          	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
          	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
          	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
          	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          	at org.apache.hadoop.gateway.trace.TraceHandler.handle(TraceHandler.java:51)
          	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          	at org.apache.hadoop.gateway.filter.CorrelationHandler.handle(CorrelationHandler.java:39)
          	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
          	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          	at org.eclipse.jetty.server.Server.handle(Server.java:499)
          	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
          	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
          	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
          	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
          	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
          	at java.lang.Thread.run(Thread.java:745)
          Caused by: org.apache.shiro.authz.AuthorizationException: LDAP naming error while attempting to retrieve authorization for user [sam].
          	at org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthorizationInfo(JndiLdapRealm.java:316)
          	at org.apache.shiro.realm.AuthorizingRealm.getAuthorizationInfo(AuthorizingRealm.java:341)
          	at org.apache.shiro.realm.AuthorizingRealm.hasRole(AuthorizingRealm.java:571)
          	at org.apache.shiro.authz.ModularRealmAuthorizer.hasRole(ModularRealmAuthorizer.java:374)
          	at org.apache.shiro.mgt.AuthorizingSecurityManager.hasRole(AuthorizingSecurityManager.java:153)
          	at org.apache.shiro.subject.support.DelegatingSubject.hasRole(DelegatingSubject.java:224)
          	at org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:69)
          	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
          	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
          	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
          	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
          	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
          	... 42 more
          Caused by: javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'ou=groups,dc=hadoop,dc=apache,dc=org'
          	at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3139)
          	at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3033)
          	at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2840)
          	at com.sun.jndi.ldap.LdapNamingEnumeration.getNextBatch(LdapNamingEnumeration.java:147)
          	at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:216)
          	at com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:189)
          	at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.rolesFor(KnoxLdapRealm.java:268)
          	at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.getRoles(KnoxLdapRealm.java:238)
          	at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.queryForAuthorizationInfo(KnoxLdapRealm.java:224)
          	at org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthorizationInfo(JndiLdapRealm.java:313)
          	... 53 more
          
          Show
          risdenk Kevin Risden added a comment - Retrieving >100 groups from the embedded LDAP server causes the same error if you request >1000 from AD. This patch adds extra groups to users.ldif, catches the SizeLimitExceededException, and prints out a message with number of groups captured. This avoids the 500 error and at least checks the groups that were able to be returned. The one question I had was how should logging be handled for this? For reference the stack trace this avoid is: WARN [org.eclipse.jetty.servlet.ServletHandler] javax.servlet.ServletException: org.apache.shiro.authz.AuthorizationException: LDAP naming error while attempting to retrieve authorization for user [sam]. at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196) at org.apache.shiro.web.filter.authc.AuthenticatingFilter.cleanup(AuthenticatingFilter.java:155) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232) at org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:50) at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232) at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:139) at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:91) at org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:138) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.apache.hadoop.gateway.trace.TraceHandler.handle(TraceHandler.java:51) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.apache.hadoop.gateway.filter.CorrelationHandler.handle(CorrelationHandler.java:39) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang. Thread .run( Thread .java:745) Caused by: org.apache.shiro.authz.AuthorizationException: LDAP naming error while attempting to retrieve authorization for user [sam]. at org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthorizationInfo(JndiLdapRealm.java:316) at org.apache.shiro.realm.AuthorizingRealm.getAuthorizationInfo(AuthorizingRealm.java:341) at org.apache.shiro.realm.AuthorizingRealm.hasRole(AuthorizingRealm.java:571) at org.apache.shiro.authz.ModularRealmAuthorizer.hasRole(ModularRealmAuthorizer.java:374) at org.apache.shiro.mgt.AuthorizingSecurityManager.hasRole(AuthorizingSecurityManager.java:153) at org.apache.shiro.subject.support.DelegatingSubject.hasRole(DelegatingSubject.java:224) at org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:69) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) ... 42 more Caused by: javax.naming.SizeLimitExceededException: [LDAP: error code 4 - Sizelimit Exceeded]; remaining name 'ou=groups,dc=hadoop,dc=apache,dc=org' at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3139) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3033) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2840) at com.sun.jndi.ldap.LdapNamingEnumeration.getNextBatch(LdapNamingEnumeration.java:147) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:216) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:189) at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.rolesFor(KnoxLdapRealm.java:268) at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.getRoles(KnoxLdapRealm.java:238) at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.queryForAuthorizationInfo(KnoxLdapRealm.java:224) at org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthorizationInfo(JndiLdapRealm.java:313) ... 53 more

            People

            • Assignee:
              risdenk Kevin Risden
              Reporter:
              kminder Kevin Minder
            • Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development