Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4925

TestQuorumJournalManager.testPurgeLogs intermittently Fails assertNoThreadsMatching

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 3.0.0-alpha1
    • None
    • qjm, test
    • None

    Description

      On Jenkins, I saw this assert fail in the @After shutdown() for
      TestQuorumJournalManager.testPurgeLogs():

          // Should not leak clients between tests -- this can cause flaky tests.
          // (See HDFS-4643)
          GenericTestUtils.assertNoThreadsMatching(".*IPC Client.*");
      

      https://builds.apache.org/job/PreCommit-HDFS-Build/4546//testReport/

      I could not reproduce locally.

      Attachments

        Issue Links

          Activity

            taoluo Tao Luo added a comment -

            Stacktrace:

            java.lang.AssertionError: Leaked thread: "IPC Client (5584739) connection to /127.0.0.1:46788 from jenkins" Id=912 RUNNABLE
            	at sun.nio.ch.FileDispatcher.preClose0(Native Method)
            	at sun.nio.ch.SocketDispatcher.preClose(SocketDispatcher.java:41)
            	at sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(SocketChannelImpl.java:677)
            	at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:201)
            	at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
            	at org.apache.hadoop.net.SocketOutputStream.close(SocketOutputStream.java:137)
            	at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
            	at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
            	...
            
            
            sun.nio.ch.FileDispatcher.preClose0(Native Method)
            sun.nio.ch.SocketDispatcher.preClose(SocketDispatcher.java:41)
            sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(SocketChannelImpl.java:677)
            java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:201)
            java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
            org.apache.hadoop.net.SocketOutputStream.close(SocketOutputStream.java:137)
            java.io.FilterOutputStream.close(FilterOutputStream.java:143)
            java.io.FilterOutputStream.close(FilterOutputStream.java:143)
            org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:239)
            org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:256)
            org.apache.hadoop.ipc.Client$Connection.close(Client.java:1036)
            org.apache.hadoop.ipc.Client$Connection.run(Client.java:855)
            	at org.junit.Assert.fail(Assert.java:93)
            	at org.apache.hadoop.test.GenericTestUtils.assertNoThreadsMatching(GenericTestUtils.java:360)
            	at org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager.shutdown(TestQuorumJournalManager.java:108)
            	at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            	at java.lang.reflect.Method.invoke(Method.java:597)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
            	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
            	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
            	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
            	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
            	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
            	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
            	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
            	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
            	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
            	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            	at java.lang.reflect.Method.invoke(Method.java:597)
            	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
            	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
            	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
            	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
            	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
            
            taoluo Tao Luo added a comment - Stacktrace: java.lang.AssertionError: Leaked thread: "IPC Client (5584739) connection to /127.0.0.1:46788 from jenkins" Id=912 RUNNABLE at sun.nio.ch.FileDispatcher.preClose0(Native Method) at sun.nio.ch.SocketDispatcher.preClose(SocketDispatcher.java:41) at sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(SocketChannelImpl.java:677) at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:201) at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97) at org.apache.hadoop.net.SocketOutputStream.close(SocketOutputStream.java:137) at java.io.FilterOutputStream.close(FilterOutputStream.java:143) at java.io.FilterOutputStream.close(FilterOutputStream.java:143) ... sun.nio.ch.FileDispatcher.preClose0(Native Method) sun.nio.ch.SocketDispatcher.preClose(SocketDispatcher.java:41) sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(SocketChannelImpl.java:677) java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:201) java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97) org.apache.hadoop.net.SocketOutputStream.close(SocketOutputStream.java:137) java.io.FilterOutputStream.close(FilterOutputStream.java:143) java.io.FilterOutputStream.close(FilterOutputStream.java:143) org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:239) org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:256) org.apache.hadoop.ipc.Client$Connection.close(Client.java:1036) org.apache.hadoop.ipc.Client$Connection.run(Client.java:855) at org.junit.Assert.fail(Assert.java:93) at org.apache.hadoop.test.GenericTestUtils.assertNoThreadsMatching(GenericTestUtils.java:360) at org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager.shutdown(TestQuorumJournalManager.java:108) at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

            Thanks to taoluo. I have identified the root cause of the bug inHDFS-9347 and HADOOP-12532 and would like to work on this issue.

            weichiu Wei-Chiu Chuang added a comment - Thanks to taoluo . I have identified the root cause of the bug inHDFS-9347 and HADOOP-12532 and would like to work on this issue.

            I am going to resolve this as fixed. The assertion failure was fixed via HDFS-9347.

            weichiu Wei-Chiu Chuang added a comment - I am going to resolve this as fixed. The assertion failure was fixed via HDFS-9347 .
            aajisaka Akira Ajisaka added a comment - - edited

            Thanks weichiu for fixing the test. I'll reopen and (re)close it to change the resolution from "Fixed" to "Duplicate".

            aajisaka Akira Ajisaka added a comment - - edited Thanks weichiu for fixing the test. I'll reopen and (re)close it to change the resolution from "Fixed" to "Duplicate".

            People

              Unassigned Unassigned
              taoluo Tao Luo
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: