Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5101

YARN_APPLICATION_UPDATED event is parsed in ApplicationHistoryManagerOnTimelineStore#convertToApplicationReport with reversed order

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Right now, the application events are parsed in in ApplicationHistoryManagerOnTimelineStore#convertToApplicationReport with timestamp descending order, which means the later events would be parsed first, and the previous same type of events would override the information. In
      https://issues.apache.org/jira/browse/YARN-4044, we have introduced YARN_APPLICATION_UPDATED events which might be submitted by RM multiple times in one application life cycle. This could cause problem.

      1. YARN-5101.0001.patch
        5 kB
        Sunil G
      2. YARN-5101.0002.patch
        6 kB
        Sunil G

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10556 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10556/)
          YARN-5101. YARN_APPLICATION_UPDATED event is parsed in (rohithsharmaks: rev 4d2f380d787a6145f45c87ba663079fedbf645b8)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10556 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10556/ ) YARN-5101 . YARN_APPLICATION_UPDATED event is parsed in (rohithsharmaks: rev 4d2f380d787a6145f45c87ba663079fedbf645b8) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerOnTimelineStore.java
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          committed to trunk/branch-2/branch-2.8.. thanks Sunil for contributions and thanks Xuan for review..

          Show
          rohithsharma Rohith Sharma K S added a comment - committed to trunk/branch-2/branch-2.8.. thanks Sunil for contributions and thanks Xuan for review..
          Hide
          xgong Xuan Gong added a comment -

          Thanks for the patch, Sunil G, and Thanks for the review, Rohith Sharma K S

          The patch looks good to me. +1

          Show
          xgong Xuan Gong added a comment - Thanks for the patch, Sunil G , and Thanks for the review, Rohith Sharma K S The patch looks good to me. +1
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          the patch looks good to me.. I will wait for a day, Xuan Gong would like to look at the patch?

          Show
          rohithsharma Rohith Sharma K S added a comment - the patch looks good to me.. I will wait for a day, Xuan Gong would like to look at the patch?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 8m 35s trunk passed
          +1 compile 1m 52s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 40s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 59s the patch passed
          +1 compile 1m 49s the patch passed
          +1 javac 1m 50s the patch passed
          -1 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 220 unchanged - 1 fixed = 221 total (was 221)
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 0m 34s the patch passed
          -1 unit 2m 52s hadoop-yarn-server-applicationhistoryservice in the patch failed.
          -1 unit 36m 27s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          63m 45s



          Reason Tests
          Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices
            hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817894/YARN-5101.0002.patch
          JIRA Issue YARN-5101
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 29b5d758d768 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7442084
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13266/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13266/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 8m 35s trunk passed +1 compile 1m 52s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 40s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 1m 49s the patch passed +1 javac 1m 50s the patch passed -1 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 220 unchanged - 1 fixed = 221 total (was 221) +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 0m 34s the patch passed -1 unit 2m 52s hadoop-yarn-server-applicationhistoryservice in the patch failed. -1 unit 36m 27s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 63m 45s Reason Tests Failed junit tests hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices   hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817894/YARN-5101.0002.patch JIRA Issue YARN-5101 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 29b5d758d768 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7442084 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt https://builds.apache.org/job/PreCommit-YARN-Build/13266/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13266/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/13266/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Rohith Sharma K S/Xuan Gong. Could you please take a look.

          Show
          sunilg Sunil G added a comment - Rohith Sharma K S / Xuan Gong . Could you please take a look.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 33s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 12s Maven dependency ordering for branch
          +1 mvninstall 8m 30s trunk passed
          +1 compile 1m 42s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 39s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 1m 44s the patch passed
          +1 javac 1m 44s the patch passed
          -1 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 221 unchanged - 1 fixed = 222 total (was 222)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 30s the patch passed
          +1 unit 3m 17s hadoop-yarn-server-applicationhistoryservice in the patch passed.
          -1 unit 37m 28s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          65m 41s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817894/YARN-5101.0002.patch
          JIRA Issue YARN-5101
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3d7ce7f4ad37 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbe97aa
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12322/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12322/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12322/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12322/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12322/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 12s Maven dependency ordering for branch +1 mvninstall 8m 30s trunk passed +1 compile 1m 42s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 39s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 1m 44s the patch passed +1 javac 1m 44s the patch passed -1 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 221 unchanged - 1 fixed = 222 total (was 222) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 30s the patch passed +1 unit 3m 17s hadoop-yarn-server-applicationhistoryservice in the patch passed. -1 unit 37m 28s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 65m 41s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817894/YARN-5101.0002.patch JIRA Issue YARN-5101 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3d7ce7f4ad37 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbe97aa Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12322/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12322/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12322/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12322/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/12322/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Thanks Rohith Sharma K S. It makes sense.
          Updated a patch with this approach, however SystemClock was used by RMAppImpl which is internally using System.currentTimeMillis and it was available only in RMAppImpl, so CS#updateApplicationPriority had to use old approach. I will raise a separate jira to use MonotonicClock for RMAppImpl.

          Show
          sunilg Sunil G added a comment - Thanks Rohith Sharma K S . It makes sense. Updated a patch with this approach, however SystemClock was used by RMAppImpl which is internally using System.currentTimeMillis and it was available only in RMAppImpl, so CS#updateApplicationPriority had to use old approach. I will raise a separate jira to use MonotonicClock for RMAppImpl.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks Sunil for providing patch. Overall patch looks good to me.
          Not from your patch , small nit. Can you use systemClock for publishing appUpdate event instead of System.currentTimeMillis() in below classes

          1. RMAppImple : line number 1057
          2. CS : line number 2114
          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks Sunil for providing patch. Overall patch looks good to me. Not from your patch , small nit. Can you use systemClock for publishing appUpdate event instead of System.currentTimeMillis() in below classes RMAppImple : line number 1057 CS : line number 2114
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 47s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 50s trunk passed
          +1 compile 0m 19s trunk passed
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 33s trunk passed
          +1 javadoc 0m 14s trunk passed
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 15s the patch passed
          +1 javac 0m 15s the patch passed
          -1 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice: The patch generated 5 new + 99 unchanged - 1 fixed = 104 total (was 100)
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 37s the patch passed
          +1 javadoc 0m 11s the patch passed
          +1 unit 2m 44s hadoop-yarn-server-applicationhistoryservice in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          27m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816852/YARN-5101.0001.patch
          JIRA Issue YARN-5101
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2afee52774e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 30ee57c
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12232/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12232/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12232/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 47s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 50s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 15s the patch passed +1 javac 0m 15s the patch passed -1 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice: The patch generated 5 new + 99 unchanged - 1 fixed = 104 total (was 100) +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 37s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 2m 44s hadoop-yarn-server-applicationhistoryservice in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 27m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816852/YARN-5101.0001.patch JIRA Issue YARN-5101 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2afee52774e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 30ee57c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12232/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12232/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice Console output https://builds.apache.org/job/PreCommit-YARN-Build/12232/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sunilg Sunil G added a comment -

          Attaching a patch addressing this issue.
          As mentioned earlier, we could check for the time stamp as a reference to identify the latest event. During appUpdate event, since we are ensuring that all params (queue name, priority) are updated, we do not need to handle cases like partial updates for one parameter in certain events.

          Xuan Gong, Rohith Sharma K S, please help to check the patch.

          Show
          sunilg Sunil G added a comment - Attaching a patch addressing this issue. As mentioned earlier, we could check for the time stamp as a reference to identify the latest event. During appUpdate event, since we are ensuring that all params (queue name, priority) are updated, we do not need to handle cases like partial updates for one parameter in certain events. Xuan Gong , Rohith Sharma K S , please help to check the patch.
          Hide
          sunilg Sunil G added a comment -

          I will try to provide a patch based on above comments. Also will write a test case to cover this scenario. Xuan Gong, pls share your thoughts if I understood the problem wrongly.

          Show
          sunilg Sunil G added a comment - I will try to provide a patch based on above comments. Also will write a test case to cover this scenario. Xuan Gong , pls share your thoughts if I understood the problem wrongly.
          Hide
          sunilg Sunil G added a comment -

          HI Xuan Gong
          pls correct me if I am wrong.

          } else if (event.getEventType().equals(
                      ApplicationMetricsConstants.UPDATED_EVENT_TYPE)) {
                    Map<String, Object> eventInfo = event.getEventInfo();
                    if (eventInfo == null) {
                      continue;
                    }
                    applicationPriority = Integer
                        .parseInt(eventInfo.get(
                            ApplicationMetricsConstants.APPLICATION_PRIORITY_INFO)
                            .toString());
                    queue = eventInfo.get(ApplicationMetricsConstants.QUEUE_ENTITY_INFO)
                        .toString();
          }
          

          From UPDATED_EVENT_TYPE, we were only fetching applicationPriority and queueName. And the assumption was that we were fetching the LAST_EVENT_ONLY event. Had few discussions on this earlier link. As per current code, we will consider all events as sorted in timestamp order. And I agree that this will cause last changed event to overwrite with old value.

          So we can look in to that last updated time for event, and then skip until we update for last event. This seems a viable option, and if no objections, I can help to share a patch. Xuan Gong/Rohith Sharma K S Thoughts?

          Show
          sunilg Sunil G added a comment - HI Xuan Gong pls correct me if I am wrong. } else if (event.getEventType().equals( ApplicationMetricsConstants.UPDATED_EVENT_TYPE)) { Map< String , Object > eventInfo = event.getEventInfo(); if (eventInfo == null ) { continue ; } applicationPriority = Integer .parseInt(eventInfo.get( ApplicationMetricsConstants.APPLICATION_PRIORITY_INFO) .toString()); queue = eventInfo.get(ApplicationMetricsConstants.QUEUE_ENTITY_INFO) .toString(); } From UPDATED_EVENT_TYPE , we were only fetching applicationPriority and queueName . And the assumption was that we were fetching the LAST_EVENT_ONLY event. Had few discussions on this earlier link . As per current code, we will consider all events as sorted in timestamp order. And I agree that this will cause last changed event to overwrite with old value. So we can look in to that last updated time for event, and then skip until we update for last event. This seems a viable option, and if no objections, I can help to share a patch. Xuan Gong / Rohith Sharma K S Thoughts?
          Hide
          xgong Xuan Gong added a comment -

          Sunil G Rohith Sharma K S
          Could you take a look at this, please ?

          Show
          xgong Xuan Gong added a comment - Sunil G Rohith Sharma K S Could you take a look at this, please ?

            People

            • Assignee:
              sunilg Sunil G
              Reporter:
              xgong Xuan Gong
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development