Hadoop Common
  1. Hadoop Common
  2. HADOOP-9336

Allow UGI of current connection to be queried

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.7, 2.1.0-beta
    • Component/s: ipc
    • Labels:
      None

      Description

      Querying UGI.getCurrentUser is synch'ed and inefficient for short-lived RPC requests. Since the connection already contains the UGI, there should be a means to query it directly and avoid a call to UGI.getCurrentUser.

      1. HADOOP-9336.patch
        5 kB
        Daryn Sharp
      2. HADOOP-9336.patch
        5 kB
        Daryn Sharp
      3. HADOOP-9336.patch
        5 kB
        Daryn Sharp
      4. HADOOP-9336.branch-23.patch
        4 kB
        Daryn Sharp
      5. HADOOP-9336.branch-23.patch
        5 kB
        Daryn Sharp
      6. HADOOP-9336.branch-23.patch
        5 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1359 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1359/)
          HADOOP-9336. Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451376)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451376
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1359 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1359/ ) HADOOP-9336 . Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451376) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451376 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1331/)
          HADOOP-9336. Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451376)

          Result = FAILURE
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451376
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1331 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1331/ ) HADOOP-9336 . Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451376) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451376 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #540 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/540/)
          HADOOP-9336. Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451383)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451383
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #540 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/540/ ) HADOOP-9336 . Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451383) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451383 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #142 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/142/)
          HADOOP-9336. Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451376)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451376
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #142 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/142/ ) HADOOP-9336 . Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451376) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451376 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3400 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3400/)
          HADOOP-9336. Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451376)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451376
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3400 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3400/ ) HADOOP-9336 . Allow UGI of current connection to be queried. Contributed by Daryn Sharp. (Revision 1451376) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1451376 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestDoAsEffectiveUser.java
          Hide
          Kihwal Lee added a comment -

          I've committed the patch to branch-2 and branch-0.23.

          Show
          Kihwal Lee added a comment - I've committed the patch to branch-2 and branch-0.23.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571459/HADOOP-9336.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2246//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2246//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571459/HADOOP-9336.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2246//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2246//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Updated comments per Kihwal

          Show
          Daryn Sharp added a comment - Updated comments per Kihwal
          Hide
          Kihwal Lee added a comment -

          +1 on the code change, but more detailed comment on getRemoteUser() will be nice, in order to avoid potential misuse.

          Show
          Kihwal Lee added a comment - +1 on the code change, but more detailed comment on getRemoteUser() will be nice, in order to avoid potential misuse.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571421/HADOOP-9336.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2243//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2243//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571421/HADOOP-9336.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2243//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2243//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          I only modified existing test cases, but was dinged for no timeout which I thought only applied to new tests, so I added timeouts to the tests I modified.

          Show
          Daryn Sharp added a comment - I only modified existing test cases, but was dinged for no timeout which I thought only applied to new tests, so I added timeouts to the tests I modified.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571299/HADOOP-9336.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2241//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2241//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571299/HADOOP-9336.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2241//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2241//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Add a Server.getRemoteUser to return the current RPC connection's UGI.

          Update tests to ensure that UGI.getCurrentUser matches Server.getRemoteUser - note: as mentioned before, another doAs on the server-side intentionally won't be reflected in Server.getRemoteUser.

          Show
          Daryn Sharp added a comment - Add a Server.getRemoteUser to return the current RPC connection's UGI. Update tests to ensure that UGI.getCurrentUser matches Server.getRemoteUser - note: as mentioned before, another doAs on the server-side intentionally won't be reflected in Server.getRemoteUser .
          Hide
          Daryn Sharp added a comment -

          Testing of this jira revealed a 3.0/2.x bug, not present in 23, where auth type is set incorrectly.

          Show
          Daryn Sharp added a comment - Testing of this jira revealed a 3.0/2.x bug, not present in 23, where auth type is set incorrectly.
          Hide
          Daryn Sharp added a comment -

          The semantics will be returning the UGI of the connection, so it will always report the UGI of the original user making the connection, not of any subsequent UGI.doAs calls. However, this jira will not universally affect anything that doesn't explicitly use it. I intend for only the FSNamesystem audit calls to currently use it to reduce the significant performance bottlenecks we are encountering.

          I'm contemplating filing another jira for UGI.doAs to cache a stack of UGIs. That would greatly accelerate UGI.getCurrentUser in general.

          Show
          Daryn Sharp added a comment - The semantics will be returning the UGI of the connection, so it will always report the UGI of the original user making the connection, not of any subsequent UGI.doAs calls. However, this jira will not universally affect anything that doesn't explicitly use it. I intend for only the FSNamesystem audit calls to currently use it to reduce the significant performance bottlenecks we are encountering. I'm contemplating filing another jira for UGI.doAs to cache a stack of UGIs. That would greatly accelerate UGI.getCurrentUser in general.
          Hide
          Alejandro Abdelnur added a comment -

          Daryn, how (if) will this affect RPC calls done within the context of a doAs()?

          Show
          Alejandro Abdelnur added a comment - Daryn, how (if) will this affect RPC calls done within the context of a doAs()?

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development