Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.2.0, 3.1.1
    • yarn-native-services
    • None
    • Reviewed

    Description

      Placement policy exists in the API but is not implemented yet.

      I have filed YARN-8074 to move the composite constraints implementation out of this phase-1 implementation of placement policy.

      Attachments

        1. YARN-7142.001.patch
          112 kB
          Gour Saha
        2. YARN-7142.002.patch
          94 kB
          Gour Saha
        3. YARN-7142.003.patch
          94 kB
          Gour Saha
        4. YARN-7142.004.patch
          94 kB
          Gour Saha
        5. YARN-7142-branch-3.1.004.patch
          93 kB
          Gour Saha

        Issue Links

          Activity

            gsaha Gour Saha added a comment -

            YARN-6592 has been merged to trunk. I will work on a patch for this.

            gsaha Gour Saha added a comment - YARN-6592 has been merged to trunk. I will work on a patch for this.
            gsaha Gour Saha added a comment -

            leftnoteasy, based on our offline discussions and on the documentation provided in YARN-6592 I am uploading a patch for incorporating rich placement policy in YARN Service. Please review when you get a chance.

            gsaha Gour Saha added a comment - leftnoteasy , based on our offline discussions and on the documentation provided in YARN-6592 I am uploading a patch for incorporating rich placement policy in YARN Service. Please review when you get a chance.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 16s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for branch
            +1 mvninstall 15m 16s trunk passed
            +1 compile 7m 33s trunk passed
            +1 checkstyle 1m 4s trunk passed
            +1 mvnsite 2m 16s trunk passed
            +1 shadedclient 11m 42s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            -1 findbugs 1m 6s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 1m 54s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 9s Maven dependency ordering for patch
            +1 mvninstall 1m 50s the patch passed
            +1 compile 6m 51s the patch passed
            +1 javac 6m 51s the patch passed
            -0 checkstyle 1m 11s hadoop-yarn-project/hadoop-yarn: The patch generated 33 new + 148 unchanged - 10 fixed = 181 total (was 158)
            +1 mvnsite 2m 32s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 9m 40s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 1s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            +1 findbugs 3m 47s the patch passed
            +1 javadoc 1m 52s the patch passed
                  Other Tests
            +1 unit 0m 36s hadoop-yarn-api in the patch passed.
            +1 unit 3m 13s hadoop-yarn-common in the patch passed.
            +1 unit 5m 54s hadoop-yarn-services-core in the patch passed.
            +1 unit 0m 31s hadoop-yarn-services-api in the patch passed.
            +1 unit 0m 15s hadoop-yarn-site in the patch passed.
            +1 asflicense 0m 25s The patch does not generate ASF License warnings.
            80m 54s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f
            JIRA Issue YARN-7142
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12915435/YARN-7142.001.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux b79a06d8f83f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 6c63cc7
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20028/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20028/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20028/testReport/
            Max. process+thread count 684 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20028/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 15m 16s trunk passed +1 compile 7m 33s trunk passed +1 checkstyle 1m 4s trunk passed +1 mvnsite 2m 16s trunk passed +1 shadedclient 11m 42s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 1m 6s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 54s trunk passed       Patch Compile Tests 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 50s the patch passed +1 compile 6m 51s the patch passed +1 javac 6m 51s the patch passed -0 checkstyle 1m 11s hadoop-yarn-project/hadoop-yarn: The patch generated 33 new + 148 unchanged - 10 fixed = 181 total (was 158) +1 mvnsite 2m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 9m 40s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 1s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 47s the patch passed +1 javadoc 1m 52s the patch passed       Other Tests +1 unit 0m 36s hadoop-yarn-api in the patch passed. +1 unit 3m 13s hadoop-yarn-common in the patch passed. +1 unit 5m 54s hadoop-yarn-services-core in the patch passed. +1 unit 0m 31s hadoop-yarn-services-api in the patch passed. +1 unit 0m 15s hadoop-yarn-site in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 80m 54s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f JIRA Issue YARN-7142 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12915435/YARN-7142.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux b79a06d8f83f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 6c63cc7 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20028/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20028/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20028/testReport/ Max. process+thread count 684 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/20028/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            leftnoteasy Wangda Tan added a comment -

            Thanks gsaha, my comments:
            1) Placement policy removed from Service? How user to specify common placement policies for components.
            2) Compatibility of the placement policy?
            3) validatePlacementPolicy:

            • Should we enforce target tag name == self component name?
              4) Remove expression name from Example.md:
            • "name": "CA1"
              5) Mark unsupported APIs from the documentation/definition.
              6) Add node partition to PlacementConstraint. Ref: org.apache.hadoop.yarn.api.resource.PlacementConstraints.PlacementTargets#nodePartition
              7) tags => targetTags
              8) Use SchedulingRequest when ANY component uses placement policy.
            leftnoteasy Wangda Tan added a comment - Thanks gsaha , my comments: 1) Placement policy removed from Service? How user to specify common placement policies for components. 2) Compatibility of the placement policy? 3) validatePlacementPolicy: Should we enforce target tag name == self component name? 4) Remove expression name from Example.md: "name": "CA1" 5) Mark unsupported APIs from the documentation/definition. 6) Add node partition to PlacementConstraint. Ref: org.apache.hadoop.yarn.api.resource.PlacementConstraints.PlacementTargets#nodePartition 7) tags => targetTags 8) Use SchedulingRequest when ANY component uses placement policy.
            gsaha Gour Saha added a comment -

            leftnoteasy thank you for the review.

            1) Placement policy removed from Service? How user to specify common placement policies for components.

            I don't think common/service-level placement policies make sense at least in this first phase of implementation since we support self allocation tag namespace only. We can introduce service level placement policy later once we support other namespaces and if it seems helpful. For now, it will be good to keep it simple.

            2) Compatibility of the placement policy?

            The existing placement policy was a dummy placeholder and not supported. There was no implementation backing it. Hence there should not be any backward compatibility issue.

            I incorporated all your other suggestions in patch 002.

            gsaha Gour Saha added a comment - leftnoteasy thank you for the review. 1) Placement policy removed from Service? How user to specify common placement policies for components. I don't think common/service-level placement policies make sense at least in this first phase of implementation since we support self allocation tag namespace only. We can introduce service level placement policy later once we support other namespaces and if it seems helpful. For now, it will be good to keep it simple. 2) Compatibility of the placement policy? The existing placement policy was a dummy placeholder and not supported. There was no implementation backing it. Hence there should not be any backward compatibility issue. I incorporated all your other suggestions in patch 002.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 22s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 25s Maven dependency ordering for branch
            +1 mvninstall 22m 46s trunk passed
            +1 compile 8m 11s trunk passed
            +1 checkstyle 1m 4s trunk passed
            +1 mvnsite 2m 38s trunk passed
            +1 shadedclient 13m 8s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            -1 findbugs 1m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 2m 2s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 12s Maven dependency ordering for patch
            +1 mvninstall 2m 6s the patch passed
            +1 compile 7m 18s the patch passed
            +1 javac 7m 18s the patch passed
            -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 150 unchanged - 9 fixed = 162 total (was 159)
            +1 mvnsite 2m 29s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 9m 48s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
            +1 javadoc 2m 20s the patch passed
                  Other Tests
            +1 unit 0m 44s hadoop-yarn-api in the patch passed.
            +1 unit 3m 20s hadoop-yarn-common in the patch passed.
            +1 unit 6m 36s hadoop-yarn-services-core in the patch passed.
            +1 unit 0m 34s hadoop-yarn-services-api in the patch passed.
            +1 unit 0m 22s hadoop-yarn-site in the patch passed.
            +1 asflicense 0m 34s The patch does not generate ASF License warnings.
            94m 21s



            Reason Tests
            FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
              Useless object stored in variable constraintsMap of method org.apache.hadoop.yarn.service.component.Component.requestContainers(long) At Component.java:constraintsMap of method org.apache.hadoop.yarn.service.component.Component.requestContainers(long) At Component.java:[line 470]



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b
            JIRA Issue YARN-7142
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12916500/YARN-7142.002.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux f829d4fe529a 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 3fe41c6
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.html
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20112/testReport/
            Max. process+thread count 683 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20112/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 22m 46s trunk passed +1 compile 8m 11s trunk passed +1 checkstyle 1m 4s trunk passed +1 mvnsite 2m 38s trunk passed +1 shadedclient 13m 8s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 1m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 2s trunk passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 2m 6s the patch passed +1 compile 7m 18s the patch passed +1 javac 7m 18s the patch passed -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 150 unchanged - 9 fixed = 162 total (was 159) +1 mvnsite 2m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 9m 48s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 20s the patch passed       Other Tests +1 unit 0m 44s hadoop-yarn-api in the patch passed. +1 unit 3m 20s hadoop-yarn-common in the patch passed. +1 unit 6m 36s hadoop-yarn-services-core in the patch passed. +1 unit 0m 34s hadoop-yarn-services-api in the patch passed. +1 unit 0m 22s hadoop-yarn-site in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 94m 21s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core   Useless object stored in variable constraintsMap of method org.apache.hadoop.yarn.service.component.Component.requestContainers(long) At Component.java:constraintsMap of method org.apache.hadoop.yarn.service.component.Component.requestContainers(long) At Component.java: [line 470] Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b JIRA Issue YARN-7142 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12916500/YARN-7142.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux f829d4fe529a 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 3fe41c6 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20112/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20112/testReport/ Max. process+thread count 683 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/20112/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            gsaha Gour Saha added a comment -

            Uploaded a 003 patch with the following -
            Fixed the findbug issue and 2 of the 12 checkstyle issues reported. 9 of the remaining 10 fall in the bucket of the existing builder pattern followed by other resource objects in the o.a.h.y.service.api.records package. The final one is asking for a default clause for a switch on an enum, which does not make sense/is not a good idea since it suppresses warnings when new enum values are added in the future.

            gsaha Gour Saha added a comment - Uploaded a 003 patch with the following - Fixed the findbug issue and 2 of the 12 checkstyle issues reported. 9 of the remaining 10 fall in the bucket of the existing builder pattern followed by other resource objects in the o.a.h.y.service.api.records package. The final one is asking for a default clause for a switch on an enum, which does not make sense/is not a good idea since it suppresses warnings when new enum values are added in the future.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 26s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 53s Maven dependency ordering for branch
            +1 mvninstall 30m 50s trunk passed
            +1 compile 11m 28s trunk passed
            +1 checkstyle 1m 28s trunk passed
            +1 mvnsite 3m 44s trunk passed
            +1 shadedclient 16m 58s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            -1 findbugs 1m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 2m 44s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 12s Maven dependency ordering for patch
            +1 mvninstall 2m 10s the patch passed
            +1 compile 7m 17s the patch passed
            +1 javac 7m 17s the patch passed
            -0 checkstyle 1m 21s hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 148 unchanged - 9 fixed = 158 total (was 157)
            +1 mvnsite 2m 46s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 11m 18s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            +1 findbugs 4m 29s the patch passed
            +1 javadoc 2m 21s the patch passed
                  Other Tests
            +1 unit 0m 48s hadoop-yarn-api in the patch passed.
            +1 unit 3m 24s hadoop-yarn-common in the patch passed.
            +1 unit 6m 38s hadoop-yarn-services-core in the patch passed.
            +1 unit 0m 34s hadoop-yarn-services-api in the patch passed.
            +1 unit 0m 20s hadoop-yarn-site in the patch passed.
            +1 asflicense 0m 33s The patch does not generate ASF License warnings.
            114m 25s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b
            JIRA Issue YARN-7142
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12916515/YARN-7142.003.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux f7f7434d28cc 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / 2a2ef15
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20113/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20113/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20113/testReport/
            Max. process+thread count 667 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20113/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 30m 50s trunk passed +1 compile 11m 28s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 3m 44s trunk passed +1 shadedclient 16m 58s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 1m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 44s trunk passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 2m 10s the patch passed +1 compile 7m 17s the patch passed +1 javac 7m 17s the patch passed -0 checkstyle 1m 21s hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 148 unchanged - 9 fixed = 158 total (was 157) +1 mvnsite 2m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 18s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 4m 29s the patch passed +1 javadoc 2m 21s the patch passed       Other Tests +1 unit 0m 48s hadoop-yarn-api in the patch passed. +1 unit 3m 24s hadoop-yarn-common in the patch passed. +1 unit 6m 38s hadoop-yarn-services-core in the patch passed. +1 unit 0m 34s hadoop-yarn-services-api in the patch passed. +1 unit 0m 20s hadoop-yarn-site in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 114m 25s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b JIRA Issue YARN-7142 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12916515/YARN-7142.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux f7f7434d28cc 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 2a2ef15 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20113/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20113/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20113/testReport/ Max. process+thread count 667 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/20113/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            gsaha Gour Saha added a comment -

            leftnoteasy, the -1 on findbugs is an extant issue in hadoop-yarn-api -

            org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal representation by returning Resource.resources
            
            gsaha Gour Saha added a comment - leftnoteasy , the -1 on findbugs is an extant issue in hadoop-yarn-api - org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal representation by returning Resource.resources
            gsaha Gour Saha added a comment -

            Thanks leftnoteasy for the offline comment. I missed changing tags to target_tags in the YarnServiceAPI.md file. Uploading 004 patch with the fix.

            gsaha Gour Saha added a comment - Thanks leftnoteasy for the offline comment. I missed changing tags to target_tags in the YarnServiceAPI.md file. Uploading 004 patch with the fix.
            leftnoteasy Wangda Tan added a comment -

            Thanks gsaha, the last patch looks good.

            I would prefer to let another set of eyes to look at this patch, sunilg could you help with the patch review? I plan to commit the patch by end of tomorrow if no objections / additional reviews.

            leftnoteasy Wangda Tan added a comment - Thanks gsaha , the last patch looks good. I would prefer to let another set of eyes to look at this patch, sunilg could you help with the patch review? I plan to commit the patch by end of tomorrow if no objections / additional reviews.
            gsaha Gour Saha added a comment -

            Thanks leftnoteasy.

            billie.rinaldi if possible, can you please review too.

            gsaha Gour Saha added a comment - Thanks leftnoteasy . billie.rinaldi if possible, can you please review too.
            genericqa genericqa added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 22s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  trunk Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for branch
            +1 mvninstall 23m 7s trunk passed
            +1 compile 7m 55s trunk passed
            +1 checkstyle 1m 15s trunk passed
            +1 mvnsite 3m 0s trunk passed
            +1 shadedclient 13m 51s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            -1 findbugs 1m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings.
            +1 javadoc 2m 28s trunk passed
                  Patch Compile Tests
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 2m 3s the patch passed
            +1 compile 6m 45s the patch passed
            +1 javac 6m 45s the patch passed
            -0 checkstyle 1m 13s hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 150 unchanged - 9 fixed = 160 total (was 159)
            +1 mvnsite 2m 22s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 9m 11s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            +1 findbugs 3m 50s the patch passed
            +1 javadoc 1m 49s the patch passed
                  Other Tests
            +1 unit 0m 38s hadoop-yarn-api in the patch passed.
            +1 unit 3m 2s hadoop-yarn-common in the patch passed.
            -1 unit 6m 26s hadoop-yarn-services-core in the patch failed.
            +1 unit 0m 29s hadoop-yarn-services-api in the patch passed.
            +1 unit 0m 19s hadoop-yarn-site in the patch passed.
            +1 asflicense 0m 34s The patch does not generate ASF License warnings.
            92m 32s



            Reason Tests
            Failed junit tests hadoop.yarn.service.TestYarnNativeServices



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b
            JIRA Issue YARN-7142
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12916677/YARN-7142.004.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux c63ec8203b16 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision trunk / a991e89
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_151
            findbugs v3.1.0-RC1
            findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            unit https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20129/testReport/
            Max. process+thread count 683 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20129/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 23m 7s trunk passed +1 compile 7m 55s trunk passed +1 checkstyle 1m 15s trunk passed +1 mvnsite 3m 0s trunk passed +1 shadedclient 13m 51s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 1m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 28s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 2m 3s the patch passed +1 compile 6m 45s the patch passed +1 javac 6m 45s the patch passed -0 checkstyle 1m 13s hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 150 unchanged - 9 fixed = 160 total (was 159) +1 mvnsite 2m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 9m 11s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 50s the patch passed +1 javadoc 1m 49s the patch passed       Other Tests +1 unit 0m 38s hadoop-yarn-api in the patch passed. +1 unit 3m 2s hadoop-yarn-common in the patch passed. -1 unit 6m 26s hadoop-yarn-services-core in the patch failed. +1 unit 0m 29s hadoop-yarn-services-api in the patch passed. +1 unit 0m 19s hadoop-yarn-site in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 92m 32s Reason Tests Failed junit tests hadoop.yarn.service.TestYarnNativeServices Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b JIRA Issue YARN-7142 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12916677/YARN-7142.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux c63ec8203b16 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / a991e89 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/20129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-services_hadoop-yarn-services-core.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20129/testReport/ Max. process+thread count 683 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/20129/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            hudson Hudson added a comment -

            FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #13911 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13911/)
            YARN-7142. Support placement policy in yarn native services. (Gour Saha (wangda: rev a0bde7d525911680f9e5fb0a939604865eb8e164)

            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/exceptions/RestApiErrorMessages.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/resources/definition/YARN-Simplified-V1-API-Layer-For-Services.yaml
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/SchedulingRequestPBImpl.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ConfigFile.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementType.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Resource.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementScope.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Service.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceStatus.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/timelineservice/TestServiceTimelinePublisher.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementPolicy.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/UpgradeComponentsFinder.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/timelineservice/ServiceTimelinePublisher.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Component.java
            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraint.java
            • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementConstraint.java
            hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #13911 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13911/ ) YARN-7142 . Support placement policy in yarn native services. (Gour Saha (wangda: rev a0bde7d525911680f9e5fb0a939604865eb8e164) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/exceptions/RestApiErrorMessages.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/resources/definition/YARN-Simplified-V1-API-Layer-For-Services.yaml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/SchedulingRequestPBImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ConfigFile.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementType.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceApiUtil.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Resource.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementScope.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Service.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceStatus.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/timelineservice/TestServiceTimelinePublisher.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementPolicy.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/UpgradeComponentsFinder.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/timelineservice/ServiceTimelinePublisher.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Component.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/resource/PlacementConstraint.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/PlacementConstraint.java
            leftnoteasy Wangda Tan added a comment -

            Thanks gsaha, pushed to trunk.

            leftnoteasy Wangda Tan added a comment - Thanks gsaha , pushed to trunk.
            leftnoteasy Wangda Tan added a comment -

            gsaha, does it make sense to bring this fix to branch-3.1 so we can get this fix in 3.1.1. If you agree, could you help to create a branch-3.1 patch, I saw many conflicts. Maybe caused by the upgrade patch (I will be fine to bring that to branch-3.1 as well if needed).

            leftnoteasy Wangda Tan added a comment - gsaha , does it make sense to bring this fix to branch-3.1 so we can get this fix in 3.1.1. If you agree, could you help to create a branch-3.1 patch, I saw many conflicts. Maybe caused by the upgrade patch (I will be fine to bring that to branch-3.1 as well if needed).
            gsaha Gour Saha added a comment -

            Upgrade is in the works and not complete yet. It has been broken into multiple tasks to keep the patches from becoming too big. So far only the first patch got committed and it might not be a good idea to merge an incomplete feature into branch-3.1.

            Having said that, the placement policy patch totally makes sense. I am attaching a branch-3.1 patch (YARN-7142-branch-3.1.004.patch) for this jira with conflicts resolved. You can use this to commit to branch-3.1.

            gsaha Gour Saha added a comment - Upgrade is in the works and not complete yet. It has been broken into multiple tasks to keep the patches from becoming too big. So far only the first patch got committed and it might not be a good idea to merge an incomplete feature into branch-3.1. Having said that, the placement policy patch totally makes sense. I am attaching a branch-3.1 patch ( YARN-7142 -branch-3.1.004.patch) for this jira with conflicts resolved. You can use this to commit to branch-3.1.
            gsaha Gour Saha added a comment -
            gsaha Gour Saha added a comment - /cc leftnoteasy
            leftnoteasy Wangda Tan added a comment -

            Thanks gsaha, reopening to run Jenkins

            leftnoteasy Wangda Tan added a comment - Thanks gsaha , reopening to run Jenkins
            genericqa genericqa added a comment -
            +1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 38s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                  branch-3.1 Compile Tests
            0 mvndep 3m 30s Maven dependency ordering for branch
            +1 mvninstall 20m 26s branch-3.1 passed
            +1 compile 8m 54s branch-3.1 passed
            +1 checkstyle 1m 23s branch-3.1 passed
            +1 mvnsite 3m 1s branch-3.1 passed
            +1 shadedclient 14m 49s branch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            +1 findbugs 4m 5s branch-3.1 passed
            +1 javadoc 2m 25s branch-3.1 passed
                  Patch Compile Tests
            0 mvndep 0m 12s Maven dependency ordering for patch
            +1 mvninstall 2m 10s the patch passed
            +1 compile 7m 32s the patch passed
            +1 javac 7m 32s the patch passed
            -0 checkstyle 1m 18s hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 149 unchanged - 9 fixed = 159 total (was 158)
            +1 mvnsite 2m 47s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 11m 12s patch has no errors when building and testing our client artifacts.
            0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            +1 findbugs 4m 30s the patch passed
            +1 javadoc 2m 19s the patch passed
                  Other Tests
            +1 unit 0m 46s hadoop-yarn-api in the patch passed.
            +1 unit 3m 10s hadoop-yarn-common in the patch passed.
            +1 unit 6m 2s hadoop-yarn-services-core in the patch passed.
            +1 unit 0m 33s hadoop-yarn-services-api in the patch passed.
            +1 unit 0m 20s hadoop-yarn-site in the patch passed.
            +1 asflicense 0m 35s The patch does not generate ASF License warnings.
            100m 27s



            Subsystem Report/Notes
            Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f
            JIRA Issue YARN-7142
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12917630/YARN-7142-branch-3.1.004.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
            uname Linux 3206b4e51b81 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/patchprocess/precommit/personality/provided.sh
            git revision branch-3.1 / a336561
            maven version: Apache Maven 3.3.9
            Default Java 1.8.0_162
            findbugs v3.1.0-RC1
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20250/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20250/testReport/
            Max. process+thread count 669 (vs. ulimit of 10000)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/20250/console
            Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            genericqa genericqa added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 38s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       branch-3.1 Compile Tests 0 mvndep 3m 30s Maven dependency ordering for branch +1 mvninstall 20m 26s branch-3.1 passed +1 compile 8m 54s branch-3.1 passed +1 checkstyle 1m 23s branch-3.1 passed +1 mvnsite 3m 1s branch-3.1 passed +1 shadedclient 14m 49s branch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 4m 5s branch-3.1 passed +1 javadoc 2m 25s branch-3.1 passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 2m 10s the patch passed +1 compile 7m 32s the patch passed +1 javac 7m 32s the patch passed -0 checkstyle 1m 18s hadoop-yarn-project/hadoop-yarn: The patch generated 10 new + 149 unchanged - 9 fixed = 159 total (was 158) +1 mvnsite 2m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 12s patch has no errors when building and testing our client artifacts. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 4m 30s the patch passed +1 javadoc 2m 19s the patch passed       Other Tests +1 unit 0m 46s hadoop-yarn-api in the patch passed. +1 unit 3m 10s hadoop-yarn-common in the patch passed. +1 unit 6m 2s hadoop-yarn-services-core in the patch passed. +1 unit 0m 33s hadoop-yarn-services-api in the patch passed. +1 unit 0m 20s hadoop-yarn-site in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 100m 27s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f JIRA Issue YARN-7142 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12917630/YARN-7142-branch-3.1.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 3206b4e51b81 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision branch-3.1 / a336561 maven version: Apache Maven 3.3.9 Default Java 1.8.0_162 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/20250/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20250/testReport/ Max. process+thread count 669 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/20250/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            cheersyang Weiwei Yang added a comment - - edited

            Hi gsaha/leftnoteasy

            Thanks for backing port this to branch-3.1. Not related to this task, I have a question about the format of placement policy in yaml file. It looks like it is more like an interpretation of how we specify placement constraints using Java API. I think we should be able to support a simple PC language, by specifying something like:

            notin,node,foo
            

            see more in this doc in YARN-7921. I know this is only used distributed shell as a demo, but I think if we find this more easier to write, maybe we can use such expression here too? Just want to know your opinion.

            Thanks

            cheersyang Weiwei Yang added a comment - - edited Hi gsaha / leftnoteasy Thanks for backing port this to branch-3.1. Not related to this task, I have a question about the format of placement policy in yaml file. It looks like it is more like an interpretation of how we specify placement constraints using Java API. I think we should be able to support a simple PC language, by specifying something like: notin,node,foo see more in this doc  in  YARN-7921 . I know this is only used distributed shell as a demo, but I think if we find this more easier to write, maybe we can use such expression here too? Just want to know your opinion. Thanks
            cheersyang Weiwei Yang added a comment -

            Hi leftnoteasy/gsaha

            Is the patch for 3.1 branch ready to commit? I see some checkstyle issues but looks like those are already included in trunk so probably doesn't matter? Please take a look, thanks.

            cheersyang Weiwei Yang added a comment - Hi leftnoteasy / gsaha Is the patch for 3.1 branch ready to commit? I see some checkstyle issues but looks like those are already included in trunk so probably doesn't matter? Please take a look, thanks.
            gsaha Gour Saha added a comment -

            I think we should be able to support a simple PC language, by specifying something like: notin,node,foo

            cheersyang thank you for the good suggestions. In my opinion, YARN Service should be viewed as a higher level abstraction for layman users who do not understand and don't want to understand YARN internals. They only understand their own application and their app deployment model. If the YARN Service API and spec are not simple and crisp it will immediately cause a barrier to such application-owners from coming to YARN. What do you think?

            gsaha Gour Saha added a comment - I think we should be able to support a simple PC language, by specifying something like: notin,node,foo cheersyang thank you for the good suggestions. In my opinion, YARN Service should be viewed as a higher level abstraction for layman users who do not understand and don't want to understand YARN internals. They only understand their own application and their app deployment model. If the YARN Service API and spec are not simple and crisp it will immediately cause a barrier to such application-owners from coming to YARN. What do you think?
            leftnoteasy Wangda Tan added a comment -

            cheersyang, thanks for reviewing this Jira, I agree with gsaha: unlike DS mostly for dev testing, placement spec of native service should be more clear. The proposed one in this Jira is clearer than DS spec for end user to use. 

            Currently we're planning to backport several dependencies to branch-3.1 so YARN-7142 can be backported w/o modification and makes native service implementation less divergency between trunk and branch-3.1. Once YARN-8118 backported, we can backport this one.

            leftnoteasy Wangda Tan added a comment - cheersyang , thanks for reviewing this Jira, I agree with gsaha : unlike DS mostly for dev testing, placement spec of native service should be more clear. The proposed one in this Jira is clearer than DS spec for end user to use.  Currently we're planning to backport several dependencies to branch-3.1 so YARN-7142 can be backported w/o modification and makes native service implementation less divergency between trunk and branch-3.1. Once YARN-8118 backported, we can backport this one.
            cheersyang Weiwei Yang added a comment -

            Thanks gsaha/leftnoteasy, that's OK lets keep it this way. I was just hesitating about which one is more descriptive. Thanks for your feedback.

            cheersyang Weiwei Yang added a comment - Thanks gsaha / leftnoteasy , that's OK lets keep it this way. I was just hesitating about which one is more descriptive. Thanks for your feedback.
            billie Billie Rinaldi added a comment -

            YARN-8018 has been committed to branch-3.1 (8118 was a typo), so this patch will cherry-pick cleanly now. I will go ahead and do this today.

            billie Billie Rinaldi added a comment - YARN-8018 has been committed to branch-3.1 (8118 was a typo), so this patch will cherry-pick cleanly now. I will go ahead and do this today.
            leftnoteasy Wangda Tan added a comment -

            Just cherry-picked to branch-3.1 

            Apologize that I missed last comment from billie.rinaldi.

            leftnoteasy Wangda Tan added a comment - Just cherry-picked to branch-3.1  Apologize that I missed last comment from billie.rinaldi .
            billie Billie Rinaldi added a comment -

            No problem! Thanks for taking care of that, leftnoteasy.

            billie Billie Rinaldi added a comment - No problem! Thanks for taking care of that, leftnoteasy .

            People

              gsaha Gour Saha
              billie Billie Rinaldi
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: