Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12706

TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      TestLocalFsFCStatistics has been failing sometimes, and when it fails it appears to be from FCStatisticsBaseTest.testStatisticsThreadLocalDataCleanUp. The test is timing out when it fails.

      1. HADOOP-12706.002.patch
        1 kB
        Colin P. McCabe
      2. HADOOP-12706.01.patch
        3 kB
        Sangjin Lee
      3. HADOOP-12706.03.patch
        2 kB
        Sangjin Lee

        Issue Links

          Activity

          Hide
          sjlee0 Sangjin Lee added a comment -

          Do you have the output of the test when it fails? I'm curious where it times out. I'll try to reproduce it later.

          Show
          sjlee0 Sangjin Lee added a comment - Do you have the output of the test when it fails? I'm curious where it times out. I'll try to reproduce it later.
          Hide
          jlowe Jason Lowe added a comment -

          Searching JIRA for "hadoop TestLocalFsFCStatistics" finds a significant number of JIRAs where this has been reported as failing during precommit builds. Sample test run:

          Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.079 sec <<< FAILURE! - in org.apache.hadoop.fs.TestLocalFsFCStatistics
          testStatisticsThreadLocalDataCleanUp(org.apache.hadoop.fs.TestLocalFsFCStatistics)  Time elapsed: 10.168 sec  <<< ERROR!
          java.util.concurrent.TimeoutException: Timed out waiting for condition. Thread diagnostics:
          Timestamp: 2016-01-11 07:16:42,102
          
          "Finalizer" daemon prio=8 tid=3 in Object.wait()
          java.lang.Thread.State: WAITING (on object monitor)
                  at java.lang.Object.wait(Native Method)
                  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
                  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
                  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
          "Signal Dispatcher" daemon prio=9 tid=4 runnable
          java.lang.Thread.State: RUNNABLE
          "Thread-6"  prio=5 tid=25 runnable
          java.lang.Thread.State: RUNNABLE
                  at java.lang.Thread.dumpThreads(Native Method)
                  at java.lang.Thread.getAllStackTraces(Thread.java:1603)
                  at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87)
                  at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73)
                  at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:188)
                  at org.apache.hadoop.fs.FCStatisticsBaseTest.testStatisticsThreadLocalDataCleanUp(FCStatisticsBaseTest.java:143)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:497)
                  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
                  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
                  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
                  at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
                  at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon prio=5 tid=22 in Object.wait()
          java.lang.Thread.State: WAITING (on object monitor)
                  at java.lang.Object.wait(Native Method)
                  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
                  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
                  at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3180)
                  at java.lang.Thread.run(Thread.java:745)
          "Reference Handler" daemon prio=10 tid=2 in Object.wait()
          java.lang.Thread.State: WAITING (on object monitor)
                  at java.lang.Object.wait(Native Method)
                  at java.lang.Object.wait(Object.java:502)
                  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
          "process reaper" daemon prio=10 tid=19 timed_waiting
          java.lang.Thread.State: TIMED_WAITING
                  at sun.misc.Unsafe.park(Native Method)
                  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
                  at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
                  at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
                  at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
                  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                  at java.lang.Thread.run(Thread.java:745)
          "main"  prio=5 tid=1 timed_waiting
          java.lang.Thread.State: TIMED_WAITING
                  at java.lang.Object.wait(Native Method)
                  at java.lang.Thread.join(Thread.java:1253)
                  at org.junit.internal.runners.statements.FailOnTimeout.evaluateStatement(FailOnTimeout.java:26)
                  at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:17)
                  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
                  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
                  at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
                  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
                  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
                  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
                  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
                  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
                  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
                  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
                  at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
                  at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
                  at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
                  at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
                  at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
                  at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
                  at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          
          
          	at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:188)
          	at org.apache.hadoop.fs.FCStatisticsBaseTest.testStatisticsThreadLocalDataCleanUp(FCStatisticsBaseTest.java:143)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Show
          jlowe Jason Lowe added a comment - Searching JIRA for "hadoop TestLocalFsFCStatistics" finds a significant number of JIRAs where this has been reported as failing during precommit builds. Sample test run: Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.079 sec <<< FAILURE! - in org.apache.hadoop.fs.TestLocalFsFCStatistics testStatisticsThreadLocalDataCleanUp(org.apache.hadoop.fs.TestLocalFsFCStatistics) Time elapsed: 10.168 sec <<< ERROR! java.util.concurrent.TimeoutException: Timed out waiting for condition. Thread diagnostics: Timestamp: 2016-01-11 07:16:42,102 "Finalizer" daemon prio=8 tid=3 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Signal Dispatcher" daemon prio=9 tid=4 runnable java.lang.Thread.State: RUNNABLE "Thread-6" prio=5 tid=25 runnable java.lang.Thread.State: RUNNABLE at java.lang.Thread.dumpThreads(Native Method) at java.lang.Thread.getAllStackTraces(Thread.java:1603) at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87) at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73) at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:188) at org.apache.hadoop.fs.FCStatisticsBaseTest.testStatisticsThreadLocalDataCleanUp(FCStatisticsBaseTest.java:143) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon prio=5 tid=22 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3180) at java.lang.Thread.run(Thread.java:745) "Reference Handler" daemon prio=10 tid=2 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157) "process reaper" daemon prio=10 tid=19 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "main" prio=5 tid=1 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1253) at org.junit.internal.runners.statements.FailOnTimeout.evaluateStatement(FailOnTimeout.java:26) at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:188) at org.apache.hadoop.fs.FCStatisticsBaseTest.testStatisticsThreadLocalDataCleanUp(FCStatisticsBaseTest.java:143) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          Hide
          cmccabe Colin P. McCabe added a comment -

          I can't seem to reproduce this failure locally. As far as I can tell from the stack trace, the failure just comes from the phantom reference not getting cleaned up within the 10 second timeout.

          Suggest increasing the timeout to something long, in case there is contention on Jenkins, and adding log messages to determine what the number of references are when a check fails.

          Show
          cmccabe Colin P. McCabe added a comment - I can't seem to reproduce this failure locally. As far as I can tell from the stack trace, the failure just comes from the phantom reference not getting cleaned up within the 10 second timeout. Suggest increasing the timeout to something long, in case there is contention on Jenkins, and adding log messages to determine what the number of references are when a check fails.
          Hide
          jlowe Jason Lowe added a comment -

          I can reproduce it fairly quickly by simply running mvn surefire:test -Dtest=TestLocalFsFCStatistics in a loop within the hadoop-common directory. The system is not heavily loaded when the test runs, so I'm wondering if adding more time is going to help or just have the test take longer. I tried increasing the wait duration from 10 seconds to 30 seconds, and it still can fail. I'll try to dig into this more tomorrow to see why the references aren't getting cleaned up when it fails.

          Show
          jlowe Jason Lowe added a comment - I can reproduce it fairly quickly by simply running mvn surefire:test -Dtest=TestLocalFsFCStatistics in a loop within the hadoop-common directory. The system is not heavily loaded when the test runs, so I'm wondering if adding more time is going to help or just have the test take longer. I tried increasing the wait duration from 10 seconds to 30 seconds, and it still can fail. I'll try to dig into this more tomorrow to see why the references aren't getting cleaned up when it fails.
          Hide
          cmccabe Colin P. McCabe added a comment -

          I just reproduced this locally by running the test in a loop until it happened to fail. Just like you experienced, the system doesn't seem to be heavily loaded.

          I don't know how deterministic PhantomReference behavior is "supposed" to be. It could be that the JVM just doesn't feel like it needs to run the GC in this small unit test, and so the P.R.s never get cleaned up. I am going to try adding a call to System.gc when the waitFor comes up false, to see if that fixes things.

          Show
          cmccabe Colin P. McCabe added a comment - I just reproduced this locally by running the test in a loop until it happened to fail. Just like you experienced, the system doesn't seem to be heavily loaded. I don't know how deterministic PhantomReference behavior is "supposed" to be. It could be that the JVM just doesn't feel like it needs to run the GC in this small unit test, and so the P.R.s never get cleaned up. I am going to try adding a call to System.gc when the waitFor comes up false, to see if that fixes things.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I was unable to reproduce it on my mac OS + JDK 1.7, but was able to reproduce it on ubuntu:

          java version "1.7.0_91"
          OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.15.10.1)
          OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)
          

          The frequency is about 1 out of 20 - 30 iterations. My strong suspicion is whether System.gc() would force the phantom reference to be enqueued in a timely manner. System.gc() itself is not deterministic (it's only a hint to the JVM after all), and the manner in which the garbage collector determines objects to be phantom-reachable seems to vary depending on the JVM/OS.

          As Colin P. McCabe and Jason Lowe observed, simply increasing the wait time in this case doesn't seem to fix it.

          I introduced a loop around GenericTestUtils.waitFor(), and if it times out I'm now forcing another System.gc() call and trying again. I can verify that we do hit that condition, and forcing another System.gc() call fixes it promptly. I ran the modified code 100 times on the problem OS/JVM combination, and it now passes all 100 times.

          I'll post a patch shortly.

          Show
          sjlee0 Sangjin Lee added a comment - I was unable to reproduce it on my mac OS + JDK 1.7, but was able to reproduce it on ubuntu: java version "1.7.0_91" OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.15.10.1) OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode) The frequency is about 1 out of 20 - 30 iterations. My strong suspicion is whether System.gc() would force the phantom reference to be enqueued in a timely manner. System.gc() itself is not deterministic (it's only a hint to the JVM after all), and the manner in which the garbage collector determines objects to be phantom-reachable seems to vary depending on the JVM/OS. As Colin P. McCabe and Jason Lowe observed, simply increasing the wait time in this case doesn't seem to fix it. I introduced a loop around GenericTestUtils.waitFor(), and if it times out I'm now forcing another System.gc() call and trying again. I can verify that we do hit that condition, and forcing another System.gc() call fixes it promptly. I ran the modified code 100 times on the problem OS/JVM combination, and it now passes all 100 times. I'll post a patch shortly.
          Hide
          sjlee0 Sangjin Lee added a comment -

          When it fails, it usually stands at 1 (one got cleaned up, the other didn't).

          Show
          sjlee0 Sangjin Lee added a comment - When it fails, it usually stands at 1 (one got cleaned up, the other didn't).
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.1.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.1.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Thanks for the patch, Sangjin Lee. I agree that it seems a bit iffy to rely on System.gc, but I'm not sure if we have any other choice. The behavior we are testing is gc-related, after all.

          The patch looks good, but I wonder if we could make it simpler by avoiding adding the "for" loop. I posted HADOOP-12706.002.patch which implements this-- let me know what you think.

          I have been running this in a loop for a half hour and so far have seen no failures.

          Show
          cmccabe Colin P. McCabe added a comment - Thanks for the patch, Sangjin Lee . I agree that it seems a bit iffy to rely on System.gc , but I'm not sure if we have any other choice. The behavior we are testing is gc-related, after all. The patch looks good, but I wonder if we could make it simpler by avoiding adding the "for" loop. I posted HADOOP-12706 .002.patch which implements this-- let me know what you think. I have been running this in a loop for a half hour and so far have seen no failures.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for your patch Colin. Yours seems much simpler and has the same behavior. Let's go with that. I might still preserve the initial System.gc() call, though... We can also add some logging as we were discussing? I can touch up the second patch if you're fine with that. Let me know.

          Regarding System.gc(), my point was not so much about whether we can avoid System.gc() in testing this (we can't) as about System.gc() being not very deterministic. Making this type of tests work across JVMs seems to be somewhat of an art.

          Still, I think this is a test-only issue in that in a live JVM there should be no cases where a phantom-reachable reference is never enqueued, short of a JVM bug, given the contract:

          If the garbage collector determines at a certain point in time that the referent of a phantom reference is phantom reachable, then at that time or at some later time it will enqueue the reference.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for your patch Colin. Yours seems much simpler and has the same behavior. Let's go with that. I might still preserve the initial System.gc() call, though... We can also add some logging as we were discussing? I can touch up the second patch if you're fine with that. Let me know. Regarding System.gc(), my point was not so much about whether we can avoid System.gc() in testing this (we can't) as about System.gc() being not very deterministic. Making this type of tests work across JVMs seems to be somewhat of an art. Still, I think this is a test-only issue in that in a live JVM there should be no cases where a phantom-reachable reference is never enqueued, short of a JVM bug, given the contract: If the garbage collector determines at a certain point in time that the referent of a phantom reference is phantom reachable, then at that time or at some later time it will enqueue the reference.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.3 based on Colin's v.2 patch. Mostly added more logging.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.3 based on Colin's v.2 patch. Mostly added more logging.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Sure, preserving the initial System.gc sounds good to me. Feel free to post another version.

          Yeah, this is definitely a test-only issue, not a bug in the code itself. Hopefully this solution will stabilize jenkins! If we find JVMs where System.gc doesn't work, we might have to disable the test entirely, which would be too bad.

          Show
          cmccabe Colin P. McCabe added a comment - Sure, preserving the initial System.gc sounds good to me. Feel free to post another version. Yeah, this is definitely a test-only issue, not a bug in the code itself. Hopefully this solution will stabilize jenkins! If we find JVMs where System.gc doesn't work, we might have to disable the test entirely, which would be too bad.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 54s trunk passed
          +1 compile 10m 10s trunk passed with JDK v1.8.0_66
          +1 compile 9m 19s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 5s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 31s the patch passed
          +1 compile 10m 40s the patch passed with JDK v1.8.0_66
          +1 javac 10m 40s the patch passed
          +1 compile 9m 30s the patch passed with JDK v1.7.0_91
          +1 javac 9m 30s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 7s the patch passed with JDK v1.7.0_91
          -1 unit 8m 26s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 9m 21s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 31s Patch does not generate ASF License warnings.
          80m 4s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.ipc.TestRPC



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782165/HADOOP-12706.03.patch
          JIRA Issue HADOOP-12706
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0061c7553e08 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8315582
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8406/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8406/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8406/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8406/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 54s trunk passed +1 compile 10m 10s trunk passed with JDK v1.8.0_66 +1 compile 9m 19s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 5s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 31s the patch passed +1 compile 10m 40s the patch passed with JDK v1.8.0_66 +1 javac 10m 40s the patch passed +1 compile 9m 30s the patch passed with JDK v1.7.0_91 +1 javac 9m 30s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 7s the patch passed with JDK v1.7.0_91 -1 unit 8m 26s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 9m 21s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 80m 4s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag   hadoop.ipc.TestRPC Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782165/HADOOP-12706.03.patch JIRA Issue HADOOP-12706 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0061c7553e08 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8315582 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8406/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8406/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8406/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8406/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Can I get your review? The test failures are unrelated.

          Show
          sjlee0 Sangjin Lee added a comment - Can I get your review? The test failures are unrelated.
          Hide
          andrew.wang Andrew Wang added a comment -

          LGTM +1

          Show
          andrew.wang Andrew Wang added a comment - LGTM +1
          Hide
          cmccabe Colin P. McCabe added a comment -

          +1 as well. Thanks, all.

          Show
          cmccabe Colin P. McCabe added a comment - +1 as well. Thanks, all.
          Hide
          jlowe Jason Lowe added a comment -

          +1 committing this.

          Show
          jlowe Jason Lowe added a comment - +1 committing this.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks everyone for the quick analysis, fix, and reviews! I committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6.

          Show
          jlowe Jason Lowe added a comment - Thanks everyone for the quick analysis, fix, and reviews! I committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks Jason for reporting the issue and others for the reviews!

          Show
          sjlee0 Sangjin Lee added a comment - Thanks Jason for reporting the issue and others for the reviews!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9116 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9116/)
          HADOOP-12706. (jlowe: rev cdf88952599a43b1ef5adda792bfb195c7529fad)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FCStatisticsBaseTest.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9116 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9116/ ) HADOOP-12706 . (jlowe: rev cdf88952599a43b1ef5adda792bfb195c7529fad) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FCStatisticsBaseTest.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              sjlee0 Sangjin Lee
              Reporter:
              jlowe Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development