Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3496

Yarn initializes ACL operations from capacity scheduler config in a non-deterministic order

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 0.24.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None

      Description

      'mapred queue -showacls' does not output put acls in a predictable manner. This is a regression from previous versions.

      1. MAPREDUCE-3496.patch
        1 kB
        Jonathan Eagles

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #919 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/919/)
        MAPREDUCE-3496. Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #919 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/919/ ) MAPREDUCE-3496 . Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #117 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/117/)
        Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #117 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/117/ ) Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #886 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/886/)
        MAPREDUCE-3496. Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #886 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/886/ ) MAPREDUCE-3496 . Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-HAbranch-build #7 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/7/)
        Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-HAbranch-build #7 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/7/ ) Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #99 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/99/)
        Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #99 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/99/ ) Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #265 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/265/)
        Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #265 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/265/ ) Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1394 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1394/)
        MAPREDUCE-3496. Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1394 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1394/ ) MAPREDUCE-3496 . Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #244 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/244/)
        Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #244 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/244/ ) Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #254 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/254/)
        Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #254 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/254/ ) Merge -c 1210738 from trunk to branch-0.23 to fix MAPREDUCE-3496 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210739 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1441 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1441/)
        MAPREDUCE-3496. Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1441 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1441/ ) MAPREDUCE-3496 . Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1369 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1369/)
        MAPREDUCE-3496. Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1369 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1369/ ) MAPREDUCE-3496 . Fixed client to print queue acls in consistent order. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1210738 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks Jonathan!

        Show
        Arun C Murthy added a comment - I just committed this. Thanks Jonathan!
        Hide
        Jonathan Eagles added a comment -

        Findbugs not related to this patch. No tests added for the cli output only code change.

        Show
        Jonathan Eagles added a comment - Findbugs not related to this patch. No tests added for the cli output only code change.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12505784/MAPREDUCE-3496.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1379//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1379//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1379//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505784/MAPREDUCE-3496.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1379//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1379//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1379//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        The solution could have been solve on the client side or the server side. This patch implements a client side solution. This will ensure predictable results no matter what scheduler is being used. Furthermore, the server container solution is not compromised (for example switching server to using LinkedHashMap from HashMap) nor relied upon to satisfy client view requirements.

        Show
        Jonathan Eagles added a comment - The solution could have been solve on the client side or the server side. This patch implements a client side solution. This will ensure predictable results no matter what scheduler is being used. Furthermore, the server container solution is not compromised (for example switching server to using LinkedHashMap from HashMap) nor relied upon to satisfy client view requirements.

          People

          • Assignee:
            Jonathan Eagles
            Reporter:
            Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development