Hadoop Common
  1. Hadoop Common
  2. HADOOP-9937

Improvement of the group information refer frequency

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: security
    • Labels:
      None
    • Target Version/s:

      Description

      The node (e.g. NameNode, ResourceManager) uses UGI.getGroupNames() now to get the information of the user's group who accessed it.
      In UGI.getGroupsNames(), synchronized is declared, but UGI instance by various methods each time and the node get different lock in getGroupName().

      For example, when cache time limit in userToGroupsMap of Groups expired and the node accepts many requests at the same time, each refers for group information in id command or JNI.
      To improve the refer frequency of the group information, it should change about synchronization.

      1. namenode-threaddump.txt
        5 kB
        Shinichi Yamashita

        Activity

        Hide
        Shinichi Yamashita added a comment -

        I attach the NameNode's thread dump. I do Thread.sleep for an investigation in UGI.getGroupsNames().
        NameNode locks different UGI object in thread dump.
        When update of cache by the result of id command is slow, much id command will be executed.

        Show
        Shinichi Yamashita added a comment - I attach the NameNode's thread dump. I do Thread.sleep for an investigation in UGI.getGroupsNames(). NameNode locks different UGI object in thread dump. When update of cache by the result of id command is slow, much id command will be executed.
        Hide
        Colin Patrick McCabe added a comment -

        We put a lock around getgrname_r by default, since a large percentage of systems out there have an implementation that actually is not re-entrant (despite the _r). So you may not get as large a speedup as you think by removing the lock in UserGroupInformation#getGroupNames (unless you also set hadoop.workaround.non.threadsafe.getpwuid to false).

        You could create a custom lock just for group information (rather than synchronizing on the big UGI lock), but I'm not sure how much that would help. Do you have a more detailed proposal for how to change the synchronization here?

        Show
        Colin Patrick McCabe added a comment - We put a lock around getgrname_r by default, since a large percentage of systems out there have an implementation that actually is not re-entrant (despite the _r). So you may not get as large a speedup as you think by removing the lock in UserGroupInformation#getGroupNames (unless you also set hadoop.workaround.non.threadsafe.getpwuid to false ). You could create a custom lock just for group information (rather than synchronizing on the big UGI lock), but I'm not sure how much that would help. Do you have a more detailed proposal for how to change the synchronization here?
        Hide
        Shinichi Yamashita added a comment -

        In ShellBasedUnixGroupsMapping, I think that I should reduce the execution number of times of OS command.
        I am concerned about the influence (e.g. memory) by forking OS process (id) at the same time than speedup.
        I think that it should improve calling of Groups#getGroups.

        Show
        Shinichi Yamashita added a comment - In ShellBasedUnixGroupsMapping, I think that I should reduce the execution number of times of OS command. I am concerned about the influence (e.g. memory) by forking OS process (id) at the same time than speedup. I think that it should improve calling of Groups#getGroups .
        Hide
        Colin Patrick McCabe added a comment -

        Have you considered using JniBasedUnixGroupsMapping? It doesn't fork any processes.

        Show
        Colin Patrick McCabe added a comment - Have you considered using JniBasedUnixGroupsMapping ? It doesn't fork any processes.

          People

          • Assignee:
            Unassigned
            Reporter:
            Shinichi Yamashita
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development