Hadoop Common
  1. Hadoop Common
  2. HADOOP-10142

Avoid groups lookup for unprivileged users such as "dr.who"

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1-win, 1.3.0, 2.3.0
    • Component/s: None
    • Labels:
      None

      Description

      Reduce the logs generated by ShellBasedUnixGroupsMapping.
      For ex: Using WebHdfs from windows generates following log for each request

      2013-12-03 11:34:56,589 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user dr.who
      org.apache.hadoop.util.Shell$ExitCodeException: id: dr.who: No such user
      
              at org.apache.hadoop.util.Shell.runCommand(Shell.java:504)
              at org.apache.hadoop.util.Shell.run(Shell.java:417)
              at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:636)
              at org.apache.hadoop.util.Shell.execCommand(Shell.java:725)
              at org.apache.hadoop.util.Shell.execCommand(Shell.java:708)
              at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
              at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
              at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
              at org.apache.hadoop.security.Groups.getGroups(Groups.java:95)
              at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1376)
              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.<init>(FSPermissionChecker.java:63)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3228)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:4063)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4052)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:748)
              at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getDirectoryListing(NamenodeWebHdfsMethods.java:715)
              at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getListingStream(NamenodeWebHdfsMethods.java:727)
              at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:675)
              at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.access$400(NamenodeWebHdfsMethods.java:114)
              at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:623)
              at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:618)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
              at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:618)
              at org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getRoot(NamenodeWebHdfsMethods.java:586)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
              at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
              at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
              at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
              at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
              at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
              at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
              at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
              at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
              at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
              at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
              at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
              at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
              at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
              at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
              at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
              at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
              at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
              at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
              at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1310)
              at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
              at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
              at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
              at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
              at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
              at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
              at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
              at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
              at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
              at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
              at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
              at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
              at org.mortbay.jetty.Server.handle(Server.java:326)
              at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
              at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
              at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
              at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
              at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
              at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
              at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
      2013-12-03 11:34:56,590 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user dr.who
      1. HADOOP-10142.patch
        10 kB
        Vinayakumar B
      2. HADOOP-10142.patch
        6 kB
        Vinayakumar B
      3. HADOOP-10142.patch
        2 kB
        Vinayakumar B
      4. HADOOP-10142.patch
        1.0 kB
        Vinayakumar B
      5. HADOOP-10142-branch-1.2.patch
        10 kB
        Xi Fang
      6. HADOOP-10142-branch-1.patch
        10 kB
        Xi Fang

        Activity

        Hide
        Vinayakumar B added a comment -

        Attaching the patch adding trace only in debug mode and just message in warn

        Show
        Vinayakumar B added a comment - Attaching the patch adding trace only in debug mode and just message in warn
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12616718/HADOOP-10142.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3330//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3330//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12616718/HADOOP-10142.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3330//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3330//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        So, I think we should just see if the user we're trying to resolve is dr.who, and if so, simply declare that it doesn't exist rather than asking the operating system. This will potentially provide a big performance increase for us, since some user lookup methods are quite slow, and negative caching (HDFS-5369) has not been implemented yet (if it ever will be). After all, the web UI picked dr.who specifically because it was a user ID that should not exist.

        Show
        Colin Patrick McCabe added a comment - So, I think we should just see if the user we're trying to resolve is dr.who , and if so, simply declare that it doesn't exist rather than asking the operating system. This will potentially provide a big performance increase for us, since some user lookup methods are quite slow, and negative caching ( HDFS-5369 ) has not been implemented yet (if it ever will be). After all, the web UI picked dr.who specifically because it was a user ID that should not exist.
        Hide
        Vinayakumar B added a comment -

        Thanks colin. Your idea make sense.
        I will post a patch for that soon.

        Show
        Vinayakumar B added a comment - Thanks colin. Your idea make sense. I will post a patch for that soon.
        Hide
        Vinayakumar B added a comment -

        Attaching a patch for exclusion of "dr.who" from group lookup

        Show
        Vinayakumar B added a comment - Attaching a patch for exclusion of "dr.who" from group lookup
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617120/HADOOP-10142.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3336//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3336//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617120/HADOOP-10142.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3336//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3336//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        dr.who is actually only a default. The HTTP user is configurable, in hadoop.http.staticuser.user. So I don't really like the idea of hard-coding this in Groups.java. What about creating a new configuration value like hadoop.unprivileged.users in CommonConfigurationKeys.java that will set a semicolon-delimited list of usernames which we never try to resolve? And then just defaulting that to dr.who?

        I'll also ask other folks what they think about this. I think it's a great improvement, since those lookups are really ugly and unnecessary.

        Show
        Colin Patrick McCabe added a comment - dr.who is actually only a default. The HTTP user is configurable, in hadoop.http.staticuser.user . So I don't really like the idea of hard-coding this in Groups.java . What about creating a new configuration value like hadoop.unprivileged.users in CommonConfigurationKeys.java that will set a semicolon-delimited list of usernames which we never try to resolve? And then just defaulting that to dr.who ? I'll also ask other folks what they think about this. I think it's a great improvement, since those lookups are really ugly and unnecessary.
        Hide
        Andrew Wang added a comment -

        I'm +1 for this idea too. Hopefully negative caching does get done eventually (I unassigned myself in case someone else wants to pick it up), but this is a good stopgap measure. Having a comma separated list (can use Configuration#getStrings) is important, because I've seen this kind of spew from users besides dr.who too.

        My only comment on the patch thus far is that we should be consistent with isDebugEnabled and LOG.debug. We could just additionally print another message with the stack trace when debug is enabled.

        Show
        Andrew Wang added a comment - I'm +1 for this idea too. Hopefully negative caching does get done eventually (I unassigned myself in case someone else wants to pick it up), but this is a good stopgap measure. Having a comma separated list (can use Configuration#getStrings ) is important, because I've seen this kind of spew from users besides dr.who too. My only comment on the patch thus far is that we should be consistent with isDebugEnabled and LOG.debug . We could just additionally print another message with the stack trace when debug is enabled.
        Hide
        Vinayakumar B added a comment -

        Thanks Colin and Andrew.
        I will make unprevileged users as configurable and post a patch soon.

        Show
        Vinayakumar B added a comment - Thanks Colin and Andrew. I will make unprevileged users as configurable and post a patch soon.
        Hide
        Vinayakumar B added a comment -

        Updated the patch as per comments.

        Andrew, I didn't understand your last comment regarding debug log. I removed that change itself as we can avoid huge logs by configuration.

        Show
        Vinayakumar B added a comment - Updated the patch as per comments. Andrew, I didn't understand your last comment regarding debug log. I removed that change itself as we can avoid huge logs by configuration.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617302/HADOOP-10142.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3339//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3339//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617302/HADOOP-10142.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3339//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3339//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        thanks, Vinay. I agree that taking the log change out is a good idea... it shouldn't be necessary (hopefully) with proper configuration. It will be nice not to see that log spew-- that's why I really like this change.

        Even though I was the one who originally suggested it, I'm not sure if hadoop.unprivileged.users is the right config key here. I guess these are more like users with no groups. Theoretically they might have privileges (we're not changing the way users are handled-- dr.who can still own files, if people are crazy enough to let him.)

        I wonder if we should have something like hadoop.user.group.mapping.overrides that contains something like "<user1>=<group1>,<group2>:<user2>=<group1>,<group2>". Then the default would be "dr.who=" (mapping dr.who to no groups). It seems like a natural extension of our pluggable user/group resolution setup.

        Show
        Colin Patrick McCabe added a comment - thanks, Vinay. I agree that taking the log change out is a good idea... it shouldn't be necessary (hopefully) with proper configuration. It will be nice not to see that log spew-- that's why I really like this change. Even though I was the one who originally suggested it, I'm not sure if hadoop.unprivileged.users is the right config key here. I guess these are more like users with no groups. Theoretically they might have privileges (we're not changing the way users are handled-- dr.who can still own files, if people are crazy enough to let him.) I wonder if we should have something like hadoop.user.group.mapping.overrides that contains something like "<user1>=<group1>,<group2>:<user2>=<group1>,<group2>". Then the default would be "dr.who=" (mapping dr.who to no groups). It seems like a natural extension of our pluggable user/group resolution setup.
        Hide
        Vinayakumar B added a comment -

        Thanks Colin.
        How about "hadoop.user.group.static.mapping" ? with group static group mappings and dr.who mapping to empty groups..

        Show
        Vinayakumar B added a comment - Thanks Colin. How about "hadoop.user.group.static.mapping" ? with group static group mappings and dr.who mapping to empty groups..
        Hide
        Colin Patrick McCabe added a comment -

        How about hadoop.user.group.static.mapping.override? I kind of feel like override should be in there somewhere, to signify that it's overriding the normal groups resolution mechanism.

        Show
        Colin Patrick McCabe added a comment - How about hadoop.user.group.static.mapping.override ? I kind of feel like override should be in there somewhere, to signify that it's overriding the normal groups resolution mechanism.
        Hide
        Colin Patrick McCabe added a comment -

        er, that should be overrides since there can be multiple

        Show
        Colin Patrick McCabe added a comment - er, that should be overrides since there can be multiple
        Hide
        Vinayakumar B added a comment -

        ok. got it . I will try to implement it and post a patch soon

        Show
        Vinayakumar B added a comment - ok. got it . I will try to implement it and post a patch soon
        Hide
        Vinayakumar B added a comment -

        Attaching the patch for the proposed hadoop.user.group.static.mapping.overrides.
        Please review

        Show
        Vinayakumar B added a comment - Attaching the patch for the proposed hadoop.user.group.static.mapping.overrides . Please review
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12617376/HADOOP-10142.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3342//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3342//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12617376/HADOOP-10142.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3342//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3342//console This message is automatically generated.
        Hide
        Chris Nauroth added a comment -

        Xi Fang or Chuan Liu, could you take a look at this patch? I recall that one of you talked to me in the past about a need to configure static user-to-group mappings in certain Windows deployments. You might want to code review this patch to see if it suits your needs.

        Show
        Chris Nauroth added a comment - Xi Fang or Chuan Liu , could you take a look at this patch? I recall that one of you talked to me in the past about a need to configure static user-to-group mappings in certain Windows deployments. You might want to code review this patch to see if it suits your needs.
        Hide
        Colin Patrick McCabe added a comment -

        I am +1 on this. Thanks, Vinay. I will hold off on committing for a few days to give other people time to check it out.

        Show
        Colin Patrick McCabe added a comment - I am +1 on this. Thanks, Vinay. I will hold off on committing for a few days to give other people time to check it out.
        Hide
        Xi Fang added a comment -

        Thanks Vinay and Chris. I think this patch also solves our static user-to-group mapping problem in certain Windows deployments. It would be good to have this patch in the Hadoop codebase.

        Show
        Xi Fang added a comment - Thanks Vinay and Chris. I think this patch also solves our static user-to-group mapping problem in certain Windows deployments. It would be good to have this patch in the Hadoop codebase.
        Hide
        Chris Nauroth added a comment -

        Xi, thanks for taking a look. Colin, I think we're good to go, and you can commit any time. Vinay, thank you for the patch!

        Show
        Chris Nauroth added a comment - Xi, thanks for taking a look. Colin, I think we're good to go, and you can commit any time. Vinay, thank you for the patch!
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #4847 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4847/)
        HADOOP-10142. Avoid groups lookup for unprivileged users such as dr.who (vinay via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1548763)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Groups.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4847 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4847/ ) HADOOP-10142 . Avoid groups lookup for unprivileged users such as dr.who (vinay via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1548763 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Groups.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java
        Hide
        Colin Patrick McCabe added a comment -

        committed. thanks, all.

        Show
        Colin Patrick McCabe added a comment - committed. thanks, all.
        Hide
        Vinayakumar B added a comment -

        Thanks all..

        Show
        Vinayakumar B added a comment - Thanks all..
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #414 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/414/)
        HADOOP-10142. Avoid groups lookup for unprivileged users such as dr.who (vinay via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1548763)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Groups.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #414 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/414/ ) HADOOP-10142 . Avoid groups lookup for unprivileged users such as dr.who (vinay via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1548763 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Groups.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1605 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1605/)
        HADOOP-10142. Avoid groups lookup for unprivileged users such as dr.who (vinay via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1548763)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Groups.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1605 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1605/ ) HADOOP-10142 . Avoid groups lookup for unprivileged users such as dr.who (vinay via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1548763 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Groups.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1631 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1631/)
        HADOOP-10142. Avoid groups lookup for unprivileged users such as dr.who (vinay via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1548763)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Groups.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1631 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1631/ ) HADOOP-10142 . Avoid groups lookup for unprivileged users such as dr.who (vinay via cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1548763 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Groups.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestGroupsCaching.java
        Hide
        Xi Fang added a comment -

        Thanks Vinay. I backport this to branch-1-win and branch-1 (HADOOP-10142-branch-1.patch).

        Show
        Xi Fang added a comment - Thanks Vinay. I backport this to branch-1-win and branch-1 ( HADOOP-10142 -branch-1.patch).
        Hide
        Chris Nauroth added a comment -

        Hi, Xi Fang. The backport looks good, but I just noticed a few minor differences from the trunk patch in terms of whitespace and line wrapping. Could you please make the whitespace and line wrapping identical to the trunk patch? Sorry to nitpick on this, but if we keep the code as similar as possible between the two branches, then it will make any future backports easier. Thank you!

        Show
        Chris Nauroth added a comment - Hi, Xi Fang . The backport looks good, but I just noticed a few minor differences from the trunk patch in terms of whitespace and line wrapping. Could you please make the whitespace and line wrapping identical to the trunk patch? Sorry to nitpick on this, but if we keep the code as similar as possible between the two branches, then it will make any future backports easier. Thank you!
        Hide
        Xi Fang added a comment -

        Hi Chris Nauroth, thanks for pointing this out. I made a new patch (HADOOP-10142-branch-1.2.patch) and tried to make the format as identical as possible.

        Thanks!

        Show
        Xi Fang added a comment - Hi Chris Nauroth , thanks for pointing this out. I made a new patch ( HADOOP-10142 -branch-1.2.patch) and tried to make the format as identical as possible. Thanks!
        Hide
        Chris Nauroth added a comment -

        +1 for the backport patch. I committed this to branch-1 and branch-1-win. Thank you again to Vinay for the original patch, and thank you to Xi for doing the backport.

        Show
        Chris Nauroth added a comment - +1 for the backport patch. I committed this to branch-1 and branch-1-win. Thank you again to Vinay for the original patch, and thank you to Xi for doing the backport.

          People

          • Assignee:
            Vinayakumar B
            Reporter:
            Vinayakumar B
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development