Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4365

FileSystemNodeLabelStore should check for root dir existence on startup

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If the namenode is in safe mode for some reason then FileSystemNodeLabelStore will prevent the RM from starting since it unconditionally tries to create the root directory for the label store. If the root directory already exists and no labels are changing then we shouldn't prevent the RM from starting even if the namenode is in safe mode.

      1. YARN-4365-1.patch
        4 kB
        Kuhu Shukla
      2. YARN-4365-2.patch
        3 kB
        Kuhu Shukla
      3. YARN-4365-3.patch
        3 kB
        Kuhu Shukla

        Activity

        Hide
        jlowe Jason Lowe added a comment -

        Sample stacktrace when the problem occurs:

        org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /path/to/labelstore. Name node is in safe mode.
        It was turned on manually. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.
                at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1322)
                at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3887)
                at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:988)
                at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
                at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
                at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
                at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
                at org.apache.hadoop.ipc.Server.call(Server.java:2297)
                at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:654)
                at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:621)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:422)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1680)
                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2247)
        
                at org.apache.hadoop.ipc.Client.call(Client.java:1458)
                at org.apache.hadoop.ipc.Client.call(Client.java:1395)
                at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
                at com.sun.proxy.$Proxy22.mkdirs(Unknown Source)
                at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:558)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:483)
                at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
                at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
                at com.sun.proxy.$Proxy23.mkdirs(Unknown Source)
                at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3008)
                at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2978)
                at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047)
                at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043)
                at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
                at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061)
                at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036)
                at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877)
                at org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.init(FileSystemNodeLabelsStore.java:87)
                at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:219)
                at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:233)
                at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
                at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:579)
                at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:964)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1005)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1001)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:422)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1680)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1001)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1041)
                at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
                at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1178)
        
        Show
        jlowe Jason Lowe added a comment - Sample stacktrace when the problem occurs: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /path/to/labelstore. Name node is in safe mode. It was turned on manually. Use "hdfs dfsadmin -safemode leave" to turn safe mode off. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1322) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3887) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:988) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server.call(Server.java:2297) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:654) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:621) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1680) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2247) at org.apache.hadoop.ipc.Client.call(Client.java:1458) at org.apache.hadoop.ipc.Client.call(Client.java:1395) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy22.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:558) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:3008) at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2978) at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1047) at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:1043) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1061) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1036) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1877) at org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.init(FileSystemNodeLabelsStore.java:87) at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:219) at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:233) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:579) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:964) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1005) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1001) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1680) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1001) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1041) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1178)
        Hide
        kshukla Kuhu Shukla added a comment -

        Checks whether path exists before calling mkdir() for root path. Unit Test included.

        Show
        kshukla Kuhu Shukla added a comment - Checks whether path exists before calling mkdir() for root path. Unit Test included.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 7s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 28s trunk passed
        +1 compile 0m 31s trunk passed with JDK v1.8.0_66
        +1 compile 0m 32s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 41s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        -1 findbugs 1m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
        +1 javadoc 0m 35s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 40s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 29s the patch passed with JDK v1.8.0_66
        +1 javac 0m 29s the patch passed
        +1 compile 0m 33s the patch passed with JDK v1.7.0_85
        +1 javac 0m 33s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 43s the patch passed
        +1 javadoc 0m 35s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 36s the patch passed with JDK v1.7.0_85
        -1 unit 2m 17s hadoop-yarn-common in the patch failed with JDK v1.8.0_66.
        -1 unit 2m 24s hadoop-yarn-common in the patch failed with JDK v1.7.0_85.
        +1 asflicense 0m 26s Patch does not generate ASF License warnings.
        26m 29s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.webapp.TestWebApp
        JDK v1.7.0_85 Failed junit tests hadoop.yarn.webapp.TestWebApp



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:date2015-11-20
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773553/YARN-4365-1.patch
        JIRA Issue YARN-4365
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ff28fcdb5368 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
        git revision trunk / 4539131
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_85.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_85.txt
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9747/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Max memory used 77MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9747/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 28s trunk passed +1 compile 0m 31s trunk passed with JDK v1.8.0_66 +1 compile 0m 32s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 41s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 1m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 0m 35s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 40s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 33s the patch passed +1 compile 0m 29s the patch passed with JDK v1.8.0_66 +1 javac 0m 29s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_85 +1 javac 0m 33s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 43s the patch passed +1 javadoc 0m 35s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 36s the patch passed with JDK v1.7.0_85 -1 unit 2m 17s hadoop-yarn-common in the patch failed with JDK v1.8.0_66. -1 unit 2m 24s hadoop-yarn-common in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 26m 29s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.webapp.TestWebApp JDK v1.7.0_85 Failed junit tests hadoop.yarn.webapp.TestWebApp Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-20 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773553/YARN-4365-1.patch JIRA Issue YARN-4365 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ff28fcdb5368 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh git revision trunk / 4539131 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9747/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9747/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Max memory used 77MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9747/console This message was automatically generated.
        Hide
        kshukla Kuhu Shukla added a comment -

        Test failure is irreproducible locally and is unrelated to the patch as far as I can see. Findbugs warnings are coming from org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl, which are not related to this patch.

        Show
        kshukla Kuhu Shukla added a comment - Test failure is irreproducible locally and is unrelated to the patch as far as I can see. Findbugs warnings are coming from org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl , which are not related to this patch.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for the patch, Kuhu!

        There appears to be a mix of overriding and mocking approaches in the test that leads to a confusing test. For example, setFileSystem was promoted to protected scope yet that's unnecessary in the current patch. The test is also spying on the node label manager and mocking Configuration unnecessarily.

        Instead of all the mocking and stubbing, I think it would be more straightforward to simply override setFileSystem and have the test use a "real" FileSystemNodeLabelsStore rather than a mocked one where we pass through various methods. The only mock at that point would be the filesystem that would be set in the overridden setFileSystem method.

        There's also a misleading comment in the test:

            // File Exists returns true the third time
            Mockito.when(myStore.fs.exists(Mockito.any(Path.class))).thenReturn(false);
        
        Show
        jlowe Jason Lowe added a comment - Thanks for the patch, Kuhu! There appears to be a mix of overriding and mocking approaches in the test that leads to a confusing test. For example, setFileSystem was promoted to protected scope yet that's unnecessary in the current patch. The test is also spying on the node label manager and mocking Configuration unnecessarily. Instead of all the mocking and stubbing, I think it would be more straightforward to simply override setFileSystem and have the test use a "real" FileSystemNodeLabelsStore rather than a mocked one where we pass through various methods. The only mock at that point would be the filesystem that would be set in the overridden setFileSystem method. There's also a misleading comment in the test: // File Exists returns true the third time Mockito.when(myStore.fs.exists(Mockito.any(Path.class))).thenReturn( false );
        Hide
        kshukla Kuhu Shukla added a comment -

        Thanks a lot Jason. Will update revised patch shorty.

        Show
        kshukla Kuhu Shukla added a comment - Thanks a lot Jason. Will update revised patch shorty.
        Hide
        kshukla Kuhu Shukla added a comment -

        Attaching revised patch, with a better test case and changing the scope of setFileSystem() from private to package private.

        Show
        kshukla Kuhu Shukla added a comment - Attaching revised patch, with a better test case and changing the scope of setFileSystem() from private to package private.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 30s trunk passed
        +1 compile 0m 25s trunk passed with JDK v1.8.0_66
        +1 compile 0m 27s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 31s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 24s trunk passed
        +1 javadoc 0m 29s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 35s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.8.0_66
        +1 javac 0m 25s the patch passed
        +1 compile 0m 27s the patch passed with JDK v1.7.0_85
        +1 javac 0m 27s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 28s the patch passed
        +1 javadoc 0m 29s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 35s the patch passed with JDK v1.7.0_85
        +1 unit 1m 54s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
        +1 unit 2m 8s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
        +1 asflicense 0m 27s Patch does not generate ASF License warnings.
        23m 14s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774142/YARN-4365-2.patch
        JIRA Issue YARN-4365
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d650dc1aa099 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / db4cab2
        findbugs v3.0.0
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9784/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9784/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 30s trunk passed +1 compile 0m 25s trunk passed with JDK v1.8.0_66 +1 compile 0m 27s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 29s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 35s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 30s the patch passed +1 compile 0m 25s the patch passed with JDK v1.8.0_66 +1 javac 0m 25s the patch passed +1 compile 0m 27s the patch passed with JDK v1.7.0_85 +1 javac 0m 27s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 28s the patch passed +1 javadoc 0m 29s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 35s the patch passed with JDK v1.7.0_85 +1 unit 1m 54s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 2m 8s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 23m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774142/YARN-4365-2.patch JIRA Issue YARN-4365 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d650dc1aa099 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / db4cab2 findbugs v3.0.0 JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9784/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9784/console This message was automatically generated.
        Hide
        kshukla Kuhu Shukla added a comment -

        Jason Lowe, Request for comments/review. Thanks a lot.

        Show
        kshukla Kuhu Shukla added a comment - Jason Lowe , Request for comments/review. Thanks a lot.
        Hide
        jlowe Jason Lowe added a comment -

        Rather than poking the store into the manager object, we don't need the manager object at all. We can just test the store object directly. Note how the test is simply using the manager object as a place to hold the store but only ever manipulates the store directly by reaching into the manager object.

        Show
        jlowe Jason Lowe added a comment - Rather than poking the store into the manager object, we don't need the manager object at all. We can just test the store object directly. Note how the test is simply using the manager object as a place to hold the store but only ever manipulates the store directly by reaching into the manager object.
        Hide
        kshukla Kuhu Shukla added a comment -

        Thank you Jason Lowe. Updated the patch.

        Show
        kshukla Kuhu Shukla added a comment - Thank you Jason Lowe . Updated the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 11m 24s trunk passed
        +1 compile 0m 48s trunk passed with JDK v1.8.0_66
        +1 compile 0m 39s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 48s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 56s trunk passed
        +1 javadoc 0m 54s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 46s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 43s the patch passed
        +1 compile 0m 49s the patch passed with JDK v1.8.0_66
        +1 javac 0m 49s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_85
        +1 javac 0m 41s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 0m 46s the patch passed
        +1 mvneclipse 0m 16s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 10s the patch passed
        +1 javadoc 0m 51s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 49s the patch passed with JDK v1.7.0_85
        +1 unit 3m 4s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
        +1 unit 2m 50s hadoop-yarn-common in the patch passed with JDK v1.7.0_85.
        +1 asflicense 0m 29s Patch does not generate ASF License warnings.
        33m 18s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774169/YARN-4365-3.patch
        JIRA Issue YARN-4365
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c0148886fec2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4ac6799
        findbugs v3.0.0
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9787/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Max memory used 76MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9787/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 11m 24s trunk passed +1 compile 0m 48s trunk passed with JDK v1.8.0_66 +1 compile 0m 39s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 48s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 54s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 46s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 43s the patch passed +1 compile 0m 49s the patch passed with JDK v1.8.0_66 +1 javac 0m 49s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_85 +1 javac 0m 41s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 46s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 10s the patch passed +1 javadoc 0m 51s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 49s the patch passed with JDK v1.7.0_85 +1 unit 3m 4s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 2m 50s hadoop-yarn-common in the patch passed with JDK v1.7.0_85. +1 asflicense 0m 29s Patch does not generate ASF License warnings. 33m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774169/YARN-4365-3.patch JIRA Issue YARN-4365 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c0148886fec2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4ac6799 findbugs v3.0.0 JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9787/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9787/console This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        +1 latest patch lgtm. Committing this.

        Show
        jlowe Jason Lowe added a comment - +1 latest patch lgtm. Committing this.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Kuhu! I committed this to trunk, branch-2, branch-2.7, and branch-2.6.

        Show
        jlowe Jason Lowe added a comment - Thanks, Kuhu! I committed this to trunk, branch-2, branch-2.7, and branch-2.6.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8882 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8882/)
        YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8882 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8882/ ) YARN-4365 . FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2660 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2660/)
        YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2660 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2660/ ) YARN-4365 . FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #719 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/719/)
        YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #719 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/719/ ) YARN-4365 . FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1451 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1451/)
        YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1451 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1451/ ) YARN-4365 . FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #729 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/729/)
        YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #729 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/729/ ) YARN-4365 . FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2577 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2577/)
        YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2577 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2577/ ) YARN-4365 . FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #640 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/640/)
        YARN-4365. FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #640 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/640/ ) YARN-4365 . FileSystemNodeLabelStore should check for root dir existence (jlowe: rev f5acf94ecafb301a0cc8e8f91f19c8bcbc8da701) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/nodelabels/TestFileSystemNodeLabelsStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/FileSystemNodeLabelsStore.java hadoop-yarn-project/CHANGES.txt
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same.

          People

          • Assignee:
            kshukla Kuhu Shukla
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development