Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12849

TestSymlinkLocalFSFileSystem fails intermittently

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Error Message

      expected:<1456523612000> but was:<1456523613000>

      Stacktrace

      java.lang.AssertionError: expected:<1456523612000> but was:<1456523613000>
      at org.junit.Assert.fail(Assert.java:88)
      at org.junit.Assert.failNotEquals(Assert.java:743)
      at org.junit.Assert.assertEquals(Assert.java:118)
      at org.junit.Assert.assertEquals(Assert.java:555)
      at org.junit.Assert.assertEquals(Assert.java:542)
      at org.apache.hadoop.fs.SymlinkBaseTest.testSetTimesDanglingLink(SymlinkBaseTest.java:1410)
      at org.apache.hadoop.fs.TestSymlinkLocalFS.testSetTimesDanglingLink(TestSymlinkLocalFS.java:239)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

      It happens in recent builds:

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9381 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9381/)
          HADOOP-12849. TestSymlinkLocalFSFileSystem fails intermittently. (cnauroth: rev 798babf661311264fa895c89a33d0f2da927da33)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9381 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9381/ ) HADOOP-12849 . TestSymlinkLocalFSFileSystem fails intermittently. (cnauroth: rev 798babf661311264fa895c89a33d0f2da927da33) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/SymlinkBaseTest.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          liuml07 Mingliang Liu added a comment -

          Thank you Chris Nauroth very much for your diagnosing the root cause, reviewing and committing the patch.

          Show
          liuml07 Mingliang Liu added a comment - Thank you Chris Nauroth very much for your diagnosing the root cause, reviewing and committing the patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the patch. I have committed this to trunk, branch-2 and branch-2.8. Mingliang Liu, thank you for contributing this fix.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the patch. I have committed this to trunk, branch-2 and branch-2.8. Mingliang Liu , thank you for contributing this fix.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 37s trunk passed
          +1 compile 5m 34s trunk passed with JDK v1.8.0_72
          +1 compile 6m 34s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 20s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 33s trunk passed
          +1 javadoc 0m 52s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 41s the patch passed
          +1 compile 5m 49s the patch passed with JDK v1.8.0_72
          +1 javac 5m 49s the patch passed
          +1 compile 6m 34s the patch passed with JDK v1.7.0_95
          +1 javac 6m 34s the patch passed
          +1 checkstyle 0m 20s the patch passed
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 55s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 3s the patch passed with JDK v1.7.0_95
          +1 unit 6m 57s hadoop-common in the patch passed with JDK v1.8.0_72.
          +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          72m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790255/HADOOP-12849.000.patch
          JIRA Issue HADOOP-12849
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d4202f80d505 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d1d4e16
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8733/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8733/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 14m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 5m 34s trunk passed with JDK v1.8.0_72 +1 compile 6m 34s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 20s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 33s trunk passed +1 javadoc 0m 52s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 41s the patch passed +1 compile 5m 49s the patch passed with JDK v1.8.0_72 +1 javac 5m 49s the patch passed +1 compile 6m 34s the patch passed with JDK v1.7.0_95 +1 javac 6m 34s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 55s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 3s the patch passed with JDK v1.7.0_95 +1 unit 6m 57s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 7m 26s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 72m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790255/HADOOP-12849.000.patch JIRA Issue HADOOP-12849 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d4202f80d505 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d1d4e16 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8733/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8733/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          The v0 patch is to relax the access time assertion, as HADOOP-12603 did (see Chris Nauroth's analysis above). I went through all the similar tests and found two cases.

          Show
          liuml07 Mingliang Liu added a comment - The v0 patch is to relax the access time assertion, as HADOOP-12603 did (see Chris Nauroth 's analysis above). I went through all the similar tests and found two cases.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Chris Nauroth very much for pointing out the related issue, which helps me a lot finding the root cause. I think we may also relax the getAccessTime() assertion here as well?

          Show
          liuml07 Mingliang Liu added a comment - Thanks Chris Nauroth very much for pointing out the related issue, which helps me a lot finding the root cause. I think we may also relax the getAccessTime() assertion here as well?
          Hide
          cnauroth Chris Nauroth added a comment -

          This looks similar to HADOOP-12603. This is an assertion on equality of atime. Unfortunately, the act of checking the inode might inadvertently trigger another update of atime. Assuming the underlying local file system tracks atime at 1-second granularity, this can manifest as a 1-second difference in expected vs. actual atime value seen by the assertion. In HADOOP-12603, we relaxed some of the assertions in SymlinkBaseTest to allow for this. It looks like we didn't catch them all though.

          Mingliang Liu, thank you for the bug report.

          Show
          cnauroth Chris Nauroth added a comment - This looks similar to HADOOP-12603 . This is an assertion on equality of atime. Unfortunately, the act of checking the inode might inadvertently trigger another update of atime. Assuming the underlying local file system tracks atime at 1-second granularity, this can manifest as a 1-second difference in expected vs. actual atime value seen by the assertion. In HADOOP-12603 , we relaxed some of the assertions in SymlinkBaseTest to allow for this. It looks like we didn't catch them all though. Mingliang Liu , thank you for the bug report.
          Hide
          liuml07 Mingliang Liu added a comment -

          Not investigated in detail, but it seems that there are several tests creating the file named link which links to non-existent file. Is it possible that the link file already exits for different tests? If true, we may need unique file names along with unique directory names.

          Show
          liuml07 Mingliang Liu added a comment - Not investigated in detail, but it seems that there are several tests creating the file named link which links to non-existent file. Is it possible that the link file already exits for different tests? If true, we may need unique file names along with unique directory names.

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              liuml07 Mingliang Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development