Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.2.0, 3.1.1
    • None
    • None
    • Reviewed

    Description

      It is observed that transitive dependencies are not resolved as a result one of the component is started earlier.
      Ex : In HBase app,
      master is independent component,
      regionserver is depends on master.
      hbaseclient depends on regionserver,
      but I always see that HBaseClient is launched before regionserver.

      Attachments

        1. YARN-7941.1.patch
          4 kB
          Billie Rinaldi

        Activity

          haibochen Haibo Chen added a comment -

          rohithsharma Is this a HBase issue or a YARN issue? How is it related to transitive dependencies?

          haibochen Haibo Chen added a comment - rohithsharma Is this a HBase issue or a YARN issue? How is it related to transitive dependencies?

          Sorry for confusion. It is totally different issue in native service. I will make this as subtask for native service i.e YARN-7054

          rohithsharma Rohith Sharma K S added a comment - Sorry for confusion. It is totally different issue in native service. I will make this as subtask for native service i.e YARN-7054
          rohithsharma Rohith Sharma K S added a comment - cc :/ jianhe billie.rinaldi
          billie Billie Rinaldi added a comment - - edited

          I think I see the problem. The dependency readiness evaluation is checking whether the number of ready containers is less than the number of desired containers. But the number of desired containers is not being set until a flex event is issued for the component, so we are checking that the number of ready containers is not less than 0. I think we can fix this by initializing the number of desired containers in the Component constructor.

          billie Billie Rinaldi added a comment - - edited I think I see the problem. The dependency readiness evaluation is checking whether the number of ready containers is less than the number of desired containers. But the number of desired containers is not being set until a flex event is issued for the component, so we are checking that the number of ready containers is not less than 0. I think we can fix this by initializing the number of desired containers in the Component constructor.
          genericqa genericqa added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 44s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 26m 53s trunk passed
          +1 compile 0m 26s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 shadedclient 11m 3s branch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 37s trunk passed
          +1 javadoc 0m 16s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 27s the patch passed
          +1 compile 0m 22s the patch passed
          +1 javac 0m 22s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 11m 32s patch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 44s the patch passed
          +1 javadoc 0m 14s the patch passed
                Other Tests
          +1 unit 6m 45s hadoop-yarn-services-core in the patch passed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          62m 7s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b
          JIRA Issue YARN-7941
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918263/YARN-7941.1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 0e277bde1947 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9059376
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_151
          findbugs v3.1.0-RC1
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20279/testReport/
          Max. process+thread count 669 (vs. ulimit of 10000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/20279/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 44s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 26m 53s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 28s trunk passed +1 shadedclient 11m 3s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 37s trunk passed +1 javadoc 0m 16s trunk passed       Patch Compile Tests +1 mvninstall 0m 27s the patch passed +1 compile 0m 22s the patch passed +1 javac 0m 22s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 32s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 14s the patch passed       Other Tests +1 unit 6m 45s hadoop-yarn-services-core in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 62m 7s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b JIRA Issue YARN-7941 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12918263/YARN-7941.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 0e277bde1947 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / 9059376 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/20279/testReport/ Max. process+thread count 669 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core Console output https://builds.apache.org/job/PreCommit-YARN-Build/20279/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          +lgtm, verified the patch in cluster.

          2018-04-10 12:01:12,982 [pool-7-thread-1] INFO  monitor.ServiceMonitor - [COMPONENT regionserver]: Dependencies satisfied, ramping up.
          .....
          2018-04-10 12:01:42,880 [pool-7-thread-1] INFO  monitor.ServiceMonitor - [COMPONENT hbaseclient]: Dependencies satisfied, ramping up.
          
          rohithsharma Rohith Sharma K S added a comment - +lgtm, verified the patch in cluster. 2018-04-10 12:01:12,982 [pool-7-thread-1] INFO monitor.ServiceMonitor - [COMPONENT regionserver]: Dependencies satisfied, ramping up. ..... 2018-04-10 12:01:42,880 [pool-7-thread-1] INFO monitor.ServiceMonitor - [COMPONENT hbaseclient]: Dependencies satisfied, ramping up.

          committing shortly

          rohithsharma Rohith Sharma K S added a comment - committing shortly

          committed to trunk/branch-3.1. thanks to billie.rinaldi for the patch!

          rohithsharma Rohith Sharma K S added a comment - committed to trunk/branch-3.1. thanks to billie.rinaldi for the patch!
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13964 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13964/)
          YARN-7941. Transitive dependencies for component are not resolved. (rohithsharmaks: rev c0487110990958fa985d273eb178bdf76002cf3a)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13964 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13964/ ) YARN-7941 . Transitive dependencies for component are not resolved. (rohithsharmaks: rev c0487110990958fa985d273eb178bdf76002cf3a) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java
          billie Billie Rinaldi added a comment -

          Thanks for the review, rohithsharma!

          billie Billie Rinaldi added a comment - Thanks for the review, rohithsharma !

          People

            billie Billie Rinaldi
            rohithsharma Rohith Sharma K S
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: