Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4034

Unable to view task logs on history server with mapreduce.job.acl-view-job=*

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.2
    • Fix Version/s: 0.23.2
    • Component/s: mrv2
    • Labels:
      None

      Description

      With log aggregation enabled, users other than the app owner or admins are sometimes unable to view the task logs on the history server even though they are in the ACL for the app. The same users are able to see the configuration and counters. Sometimes the users can see some task logs but not other task logs for the same application.

        Activity

        Hide
        Jason Lowe added a comment -

        Thanks to Siddarth Seth for the analysis on this issue. He discovered that the RM is not filling out the application ACLs on the response to ApplicationMasterService.registerApplicationMaster. The RM is responsible for setting up the app ACLs on the node that runs the AM container, but the ACLs for all the tasks are set via container requests from the AM. Since the AM doesn't know the proper ACLs, any node other than the node the AM is running on will have no ACLs for the app and the corresponding task logs will be recorded without ACLs. When the history server tries to serve up these logs, the lack of ACLs causes it to fall back to the default and only allows access from the app owner and admins.

        Logs for tasks that ran on the same node as the AM work since the node had the proper ACLs setup by the RM. This explains why it was never seen on a single-node cluster and why some tasks can be seen and others can't for the same application.

        Thanks again to Siddarth!

        Show
        Jason Lowe added a comment - Thanks to Siddarth Seth for the analysis on this issue. He discovered that the RM is not filling out the application ACLs on the response to ApplicationMasterService.registerApplicationMaster. The RM is responsible for setting up the app ACLs on the node that runs the AM container, but the ACLs for all the tasks are set via container requests from the AM. Since the AM doesn't know the proper ACLs, any node other than the node the AM is running on will have no ACLs for the app and the corresponding task logs will be recorded without ACLs. When the history server tries to serve up these logs, the lack of ACLs causes it to fall back to the default and only allows access from the app owner and admins. Logs for tasks that ran on the same node as the AM work since the node had the proper ACLs setup by the RM. This explains why it was never seen on a single-node cluster and why some tasks can be seen and others can't for the same application. Thanks again to Siddarth!
        Hide
        Jason Lowe added a comment -

        Patch to set app ACLs in the AM registration response.

        Show
        Jason Lowe added a comment - Patch to set app ACLs in the AM registration response.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12519013/MAPREDUCE-4034.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2073//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2073//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519013/MAPREDUCE-4034.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2073//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2073//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        The patch looks good to me too +1.

        Show
        Robert Joseph Evans added a comment - The patch looks good to me too +1.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Jason and Sid. I just put this into 0.23.2, trunk, and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Jason and Sid. I just put this into 0.23.2, trunk, and branch-0.23
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #231 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/)
        svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302982
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #231 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/231/ ) svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034 . Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302982 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #708 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/708/)
        svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302982
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #708 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/708/ ) svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034 . Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302982 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1907 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1907/)
        MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302980
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1907 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1907/ ) MAPREDUCE-4034 . Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302980 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #699 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/699/)
        svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302982
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #699 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/699/ ) svn merge -c 1302980 from trunk to branch 0.23 FIXES MAPREDUCE-4034 . Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302982) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302982 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1981 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1981/)
        MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302980
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1981 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1981/ ) MAPREDUCE-4034 . Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302980 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/)
        MAPREDUCE-4034. Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302980
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1025 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1025/ ) MAPREDUCE-4034 . Unable to view task logs on history server with mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) (Revision 1302980) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1302980 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development