Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11818

TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2, 2.8.2
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Saw a weird Mockito failure in last night's build with the following stack trace:

      org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
      INodeFile cannot be returned by isRunning()
      isRunning() should return boolean
      	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.addBlockOnNodes(TestBlockManager.java:555)
      	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.doTestSufficientlyReplBlocksUsesNewRack(TestBlockManager.java:404)
      	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.testSufficientlyReplBlocksUsesNewRack(TestBlockManager.java:397)
      

      This is pretty confusing since we explicitly set isRunning() to return true in TestBlockManager's @Before method

      154    Mockito.doReturn(true).when(fsn).isRunning();
      

      Also saw the following exception in the logs:

      2017-05-12 05:42:27,903 ERROR blockmanagement.BlockManager (BlockManager.java:run(2796)) - Error while processing replication queues async
      org.mockito.exceptions.base.MockitoException: 
      'writeLockInterruptibly' is a *void method* and it *cannot* be stubbed with a *return value*!
      Voids are usually stubbed with Throwables:
          doThrow(exception).when(mock).someVoidMethod();
      If the method you are trying to stub is *overloaded* then make sure you are calling the right overloaded version.
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processMisReplicatesAsync(BlockManager.java:2841)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.access$100(BlockManager.java:120)
      	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$1.run(BlockManager.java:2792)
      

      This is also weird since we don't do any explicit mocking with writeLockInterruptibly via fsn in the test. It has to be something changing the mocks or non-thread safe access or something like that. I can't explain the failures otherwise.

      1. HDFS-11818.patch
        2 kB
        Nathan Roberts
      2. HDFS-11818-branch-2.patch
        2 kB
        Nathan Roberts

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11732 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11732/)
          HDFS-11818. TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails (jlowe: rev 2397a2626e22d002174f4a36891d713a7e1f1b20)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11732 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11732/ ) HDFS-11818 . TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails (jlowe: rev 2397a2626e22d002174f4a36891d713a7e1f1b20) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Nathan for the contribution and to Eric for additional review! I committed this to trunk, branch-2, and branch-2.8.

          Show
          jlowe Jason Lowe added a comment - Thanks to Nathan for the contribution and to Eric for additional review! I committed this to trunk, branch-2, and branch-2.8.
          Hide
          jlowe Jason Lowe added a comment -

          Test failures are unrelated.

          +1 lgtm. Committing this.

          Show
          jlowe Jason Lowe added a comment - Test failures are unrelated. +1 lgtm. Committing this.
          Hide
          ebadger Eric Badger added a comment -

          lgtm +1 (non-binding)

          Show
          ebadger Eric Badger added a comment - lgtm +1 (non-binding)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 15m 31s trunk passed
          +1 compile 0m 51s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          -1 findbugs 1m 45s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          +1 checkstyle 0m 37s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 0m 45s the patch passed
          -1 unit 88m 54s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          118m 20s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-11818
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867834/HDFS-11818.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fde7f9a06906 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a9e24a1
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19419/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/19419/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19419/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19419/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 31s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 15s trunk passed -1 findbugs 1m 45s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed +1 checkstyle 0m 37s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 45s the patch passed -1 unit 88m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 118m 20s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11818 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867834/HDFS-11818.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fde7f9a06906 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a9e24a1 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19419/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/19419/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19419/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19419/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Patches for trunk and branch-2. branch-2 patch picks cleanly to 2.8

          Show
          nroberts Nathan Roberts added a comment - Patches for trunk and branch-2. branch-2 patch picks cleanly to 2.8
          Hide
          nroberts Nathan Roberts added a comment -

          Know what the issue is. Will post a patch shortly.

          Show
          nroberts Nathan Roberts added a comment - Know what the issue is. Will post a patch shortly.

            People

            • Assignee:
              nroberts Nathan Roberts
              Reporter:
              ebadger Eric Badger
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development