Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10592

Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This jira is to fix the TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning test case failure.

      Reference Build_15973

      1. HDFS-10592-00.patch
        8 kB
        Rakesh R
      2. HDFS-10592-01.patch
        9 kB
        Rakesh R

        Activity

        Hide
        rakeshr Rakesh R added a comment -

        As per the analysis the test case failure is due to the ConcurrentModificationException while iterating over the mockito stubs.

        Following are the two possible concurrent operations which results in the above exception.

        operation-1) NameNodeResourceMonitor is running and invoked FSNamesystem.checkAvailableResources
        operation-2) At the same time, adding the answer Mockito.when(mockResourceChecker.hasAvailableDiskSpace()).thenReturn(false); to the InvocationContainerImpl#stubbed linked list data structure.

        Reference log:

        2016-07-04 12:00:05,657 [DataNode: [[[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data1/, [DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:46553] INFO  datanode.DataNode (BPOfferService.java:processCommandFromActive(696)) - Got finalize command for block pool BP-1174120436-172.17.0.2-1467633603414
        2016-07-04 12:00:05,757 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2543)) - Cluster is active
        2016-07-04 12:00:05,760 [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@640f11a1] ERROR namenode.FSNamesystem (FSNamesystem.java:run(3808)) - Exception in NameNodeResourceMonitor: 
        java.util.ConcurrentModificationException
        	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
        	at java.util.LinkedList$ListItr.next(LinkedList.java:888)
        	at org.mockito.internal.stubbing.InvocationContainerImpl.findAnswerFor(InvocationContainerImpl.java:71)
        	at org.mockito.internal.MockHandler.handle(MockHandler.java:93)
        	at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)
        	at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$$EnhancerByMockitoWithCGLIB$$e4714180.hasAvailableDiskSpace(<generated>)
        	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAvailableResources(FSNamesystem.java:3763)
        	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:3791)
        	at java.lang.Thread.run(Thread.java:745)
        2016-07-04 12:01:05,772 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1907)) - Shutting down the Mini HDFS Cluster
        2016-07-04 12:01:05,773 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdownDataNodes(1947)) - Shutting down DataNode 0
        

        org.mockito.internal.stubbing.InvocationContainerImpl#findAnswerFor is not thread safe.

        org.mockito.internal.stubbing.InvocationContainerImpl.java
            public StubbedInvocationMatcher findAnswerFor(Invocation invocation) {
                for (StubbedInvocationMatcher s : stubbed) {
                    if (s.matches(invocation)) {
                        s.markStubUsed(invocation);
                        invocation.markStubbed(new StubInfo(s));
                        return s;
                    }
                }
        
                return null;
            }
        

        Proposed Fix:
        Instead of using Mockito, it is pretty simple to use NameNodeResourceCheckerStub stubbing and sets the resource availability explicitly. I will attach a patch soon.

        Show
        rakeshr Rakesh R added a comment - As per the analysis the test case failure is due to the ConcurrentModificationException while iterating over the mockito stubs. Following are the two possible concurrent operations which results in the above exception. operation-1) NameNodeResourceMonitor is running and invoked FSNamesystem.checkAvailableResources operation-2) At the same time, adding the answer Mockito.when(mockResourceChecker.hasAvailableDiskSpace()).thenReturn(false); to the InvocationContainerImpl#stubbed linked list data structure. Reference log: 2016-07-04 12:00:05,657 [DataNode: [[[DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data1/, [DISK]file:/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/3/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:46553] INFO datanode.DataNode (BPOfferService.java:processCommandFromActive(696)) - Got finalize command for block pool BP-1174120436-172.17.0.2-1467633603414 2016-07-04 12:00:05,757 [main] INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:waitActive(2543)) - Cluster is active 2016-07-04 12:00:05,760 [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@640f11a1] ERROR namenode.FSNamesystem (FSNamesystem.java:run(3808)) - Exception in NameNodeResourceMonitor: java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.mockito.internal.stubbing.InvocationContainerImpl.findAnswerFor(InvocationContainerImpl.java:71) at org.mockito.internal.MockHandler.handle(MockHandler.java:93) at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47) at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$$EnhancerByMockitoWithCGLIB$$e4714180.hasAvailableDiskSpace(<generated>) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAvailableResources(FSNamesystem.java:3763) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:3791) at java.lang. Thread .run( Thread .java:745) 2016-07-04 12:01:05,772 [main] INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1907)) - Shutting down the Mini HDFS Cluster 2016-07-04 12:01:05,773 [main] INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdownDataNodes(1947)) - Shutting down DataNode 0 org.mockito.internal.stubbing.InvocationContainerImpl#findAnswerFor is not thread safe. org.mockito.internal.stubbing.InvocationContainerImpl.java public StubbedInvocationMatcher findAnswerFor(Invocation invocation) { for (StubbedInvocationMatcher s : stubbed) { if (s.matches(invocation)) { s.markStubUsed(invocation); invocation.markStubbed( new StubInfo(s)); return s; } } return null ; } Proposed Fix: Instead of using Mockito , it is pretty simple to use NameNodeResourceCheckerStub stubbing and sets the resource availability explicitly. I will attach a patch soon.
        Hide
        rakeshr Rakesh R added a comment -

        Attached patch where it replaces mockito using normal class stubbing, kindly review the patch. Thanks!

        Show
        rakeshr Rakesh R added a comment - Attached patch where it replaces mockito using normal class stubbing, kindly review the patch. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
        +1 mvninstall 7m 16s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 44s trunk passed
        +1 javadoc 1m 2s trunk passed
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 19 unchanged - 4 fixed = 20 total (was 23)
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 1m 0s the patch passed
        -1 unit 73m 11s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        93m 43s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:85209cc
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816079/HDFS-10592-00.patch
        JIRA Issue HDFS-10592
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux afae75ce10b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8b4b525
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15975/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15975/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15975/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15975/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 7m 16s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 1m 2s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 19 unchanged - 4 fixed = 20 total (was 23) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 1m 0s the patch passed -1 unit 73m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 93m 43s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816079/HDFS-10592-00.patch JIRA Issue HDFS-10592 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux afae75ce10b0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b4b525 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15975/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15975/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15975/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15975/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Please ignore test case failure which is not related to the patch. Appreciate reviews, thanks!

        Show
        rakeshr Rakesh R added a comment - Please ignore test case failure which is not related to the patch. Appreciate reviews, thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
        +1 mvninstall 7m 47s trunk passed
        +1 compile 0m 59s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 1m 10s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 2s trunk passed
        +1 javadoc 1m 1s trunk passed
        +1 mvninstall 1m 4s the patch passed
        +1 compile 0m 54s the patch passed
        +1 javac 0m 54s the patch passed
        +1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 19 unchanged - 4 fixed = 19 total (was 23)
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 57s the patch passed
        -1 unit 75m 19s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        97m 13s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.hdfs.server.namenode.TestEditLog



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:85209cc
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816122/HDFS-10592-01.patch
        JIRA Issue HDFS-10592
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7f3144a609c3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8b4b525
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/15980/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15980/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15980/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 7m 47s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 1m 4s the patch passed +1 compile 0m 54s the patch passed +1 javac 0m 54s the patch passed +1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 19 unchanged - 4 fixed = 19 total (was 23) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 57s the patch passed -1 unit 75m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 97m 13s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.TestEditLog Subsystem Report/Notes Docker Image:yetus/hadoop:85209cc JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816122/HDFS-10592-01.patch JIRA Issue HDFS-10592 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7f3144a609c3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b4b525 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15980/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15980/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15980/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Please ignore the test case failures, it unrelated to my patch. I could see HDFS-10169 is handling TestEditLog failure and I've commented in that jira. Can someone help me by reviewing the proposed patch/fix. Thanks!

        Show
        rakeshr Rakesh R added a comment - Please ignore the test case failures, it unrelated to my patch. I could see HDFS-10169 is handling TestEditLog failure and I've commented in that jira. Can someone help me by reviewing the proposed patch/fix. Thanks!
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Thanks Rakesh for working on this.
        +1 for the patch. Will commit this.

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Thanks Rakesh for working on this. +1 for the patch. Will commit this.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        I have just committed this to trunk and branch-2, 2.8. Thanks

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - I have just committed this to trunk and branch-2, 2.8. Thanks
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10062 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10062/)
        HDFS-10592:Fix intermittent test failure of (uma.gangumalla: rev ab092c56c251f27a9426e3c69a784417fdf9a26e)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/MockNameNodeResourceChecker.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10062 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10062/ ) HDFS-10592 :Fix intermittent test failure of (uma.gangumalla: rev ab092c56c251f27a9426e3c69a784417fdf9a26e) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/MockNameNodeResourceChecker.java

          People

          • Assignee:
            rakeshr Rakesh R
            Reporter:
            rakeshr Rakesh R
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development