Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4228

FileSystemRMStateStore use IOUtils#close instead of fs#close

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      NPE on FileSystemRMStateStore#closeWithRetries when active service initialization fails on rm start up

      2015-10-05 19:56:38,626 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore failed in state STOPPED; cause: java.lang.NullPointerException
      java.lang.NullPointerException
          at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore$13.run(FileSystemRMStateStore.java:721)
          at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore$13.run(FileSystemRMStateStore.java:718)
          at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore$FSAction.runWithRetries(FileSystemRMStateStore.java:734)
          at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.closeWithRetries(FileSystemRMStateStore.java:718)
          at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.closeInternal(FileSystemRMStateStore.java:169)
          at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStop(RMStateStore.java:618)
          at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
          at org.apache.hadoop.service.AbstractService.close(AbstractService.java:250)
          at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStop(ResourceManager.java:609)
          at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
          at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
          at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
          at org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
          at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:965)
          at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:256)
          at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
          at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1195)
      
      1. 0001-YARN-4228.patch
        1 kB
        Bibin A Chundatt

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2403 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2403/)
        YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2403 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2403/ ) YARN-4228 . FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #464 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/464/)
        YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #464 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/464/ ) YARN-4228 . FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2435 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2435/)
        YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2435 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2435/ ) YARN-4228 . FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #499 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/499/)
        YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #499 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/499/ ) YARN-4228 . FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1228 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1228/)
        YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1228 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1228/ ) YARN-4228 . FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #491 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/491/)
        YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #491 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/491/ ) YARN-4228 . FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8582 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8582/)
        YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8582 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8582/ ) YARN-4228 . FileSystemRMStateStore use IOUtils#close instead of fs#close. (rohithsharmaks: rev 3793cbe4c3cce5d03c4a18d562cbcb7cacd8f743) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/FileSystemRMStateStore.java hadoop-yarn-project/CHANGES.txt
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        committed to branch-2/trunk.. thanks Bibin!!

        Show
        rohithsharma Rohith Sharma K S added a comment - committed to branch-2/trunk.. thanks Bibin!!
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Rohith Sharma K S
        Thnks for looking in to this issue. Failures are not related . YARN-3342 jira is already available for the same

        Show
        bibinchundatt Bibin A Chundatt added a comment - Rohith Sharma K S Thnks for looking in to this issue. Failures are not related . YARN-3342 jira is already available for the same
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        +1 lgtm.. can you check test failures?

        Show
        rohithsharma Rohith Sharma K S added a comment - +1 lgtm.. can you check test failures?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 2s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 55s There were no new javac warning messages.
        +1 javadoc 10m 12s There were no new javadoc warning messages.
        -1 release audit 0m 16s The applied patch generated 1 release audit warnings.
        +1 checkstyle 0m 49s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 27s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 55m 49s Tests failed in hadoop-yarn-server-resourcemanager.
            95m 34s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12765137/0001-YARN-4228.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5f6edb3
        Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9360/artifact/patchprocess/patchReleaseAuditProblems.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9360/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9360/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9360/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 2s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 12s There were no new javadoc warning messages. -1 release audit 0m 16s The applied patch generated 1 release audit warnings. +1 checkstyle 0m 49s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 55m 49s Tests failed in hadoop-yarn-server-resourcemanager.     95m 34s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12765137/0001-YARN-4228.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5f6edb3 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9360/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9360/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9360/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9360/console This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Attaching patch for the same

        Show
        bibinchundatt Bibin A Chundatt added a comment - Attaching patch for the same

          People

          • Assignee:
            bibinchundatt Bibin A Chundatt
            Reporter:
            bibinchundatt Bibin A Chundatt
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development