Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: fuse-dfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestFuseDFS.java has two bugs:

      • there is a race condition between mounting the filesystem and testing it
      • it doesn't clear the mount directory before it tries to mount there
      1. HDFS-3758-b1.001.patch
        4 kB
        Colin Patrick McCabe
      2. HDFS-3758.003.patch
        4 kB
        Colin Patrick McCabe
      3. TestFuseDFS-fix-0002.patch
        4 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2. Thanks Colin!

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2571 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2571/)
          HDFS-3758. TestFuseDFS test failing. Contributed by Colin Patrick McCabe (Revision 1371555)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371555
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/TestFuseDFS.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2571 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2571/ ) HDFS-3758 . TestFuseDFS test failing. Contributed by Colin Patrick McCabe (Revision 1371555) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371555 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/test/TestFuseDFS.java
          Hide
          Colin Patrick McCabe added a comment -

          Just to be clear, the reason for foregrounding fuse_dfs is so we can capture the log output, which we otherwise would not see. Not having log output makes debugging difficult, as you might imagine.

          Show
          Colin Patrick McCabe added a comment - Just to be clear, the reason for foregrounding fuse_dfs is so we can capture the log output, which we otherwise would not see. Not having log output makes debugging difficult, as you might imagine.
          Hide
          Colin Patrick McCabe added a comment -

          Here's a little more explanation of the patch: -ononempty allows FUSE to mount over a non-empty directory. Since we previously had a bug which could result in the fuse mount directory getting full of junk, you can see why this is useful.

          This patch also changes the way we run fuse_dfs slightly. Rather than running it in the background, we run it in the foreground, piping its stdout and stderr to java threads. This is the meaning of the -f option.

          Show
          Colin Patrick McCabe added a comment - Here's a little more explanation of the patch: -ononempty allows FUSE to mount over a non-empty directory. Since we previously had a bug which could result in the fuse mount directory getting full of junk, you can see why this is useful. This patch also changes the way we run fuse_dfs slightly. Rather than running it in the background, we run it in the foreground, piping its stdout and stderr to java threads. This is the meaning of the -f option.
          Hide
          Eli Collins added a comment -

          Actually looks like branch-1 will need 423, 856, 857, 858, 859, 860, 861, 961, 780 and 420 before it's up to speed enough for this change so can probably hold off on branch-1 for now.

          I'll commit your patch for trunk, and let's follow up on the test timeout in HDFS-3634 . I suspect we can remove this test entirely and replace it with your new one.

          Show
          Eli Collins added a comment - Actually looks like branch-1 will need 423, 856, 857, 858, 859, 860, 861, 961, 780 and 420 before it's up to speed enough for this change so can probably hold off on branch-1 for now. I'll commit your patch for trunk, and let's follow up on the test timeout in HDFS-3634 . I suspect we can remove this test entirely and replace it with your new one.
          Hide
          Eli Collins added a comment -

          The branch-1 patch doesn't apply, looks like HDFS-423 and HDFS-780 need to be applied first.

          Show
          Eli Collins added a comment - The branch-1 patch doesn't apply, looks like HDFS-423 and HDFS-780 need to be applied first.
          Hide
          Eli Collins added a comment -

          This is fixed in HDFS-3634. It just needs a review.

          Not really, HDFS-3634 doesn't modify this test, it introduces a new test that doesn't have this issue but it doesn't replace this test, so this test will still fail if the 5s timeout is insufficient. When do you think the HDFS-3634 tests will be sufficient to remove this TestFuseDFS entirely?

          Show
          Eli Collins added a comment - This is fixed in HDFS-3634 . It just needs a review. Not really, HDFS-3634 doesn't modify this test, it introduces a new test that doesn't have this issue but it doesn't replace this test, so this test will still fail if the 5s timeout is insufficient. When do you think the HDFS-3634 tests will be sufficient to remove this TestFuseDFS entirely?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539117/HDFS-3758-b1.001.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2953//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539117/HDFS-3758-b1.001.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2953//console This message is automatically generated.
          Hide
          Colin Patrick McCabe added a comment -

          Worth considering checking the mount status instead of waiting 5s as a follow-on improvement?

          This is fixed in HDFS-3634. It just needs a review.

          Show
          Colin Patrick McCabe added a comment - Worth considering checking the mount status instead of waiting 5s as a follow-on improvement? This is fixed in HDFS-3634 . It just needs a review.
          Hide
          Eli Collins added a comment -

          Marking for branch-1 as well. IIRC the test code is the same so the same change should apply.

          Show
          Eli Collins added a comment - Marking for branch-1 as well. IIRC the test code is the same so the same change should apply.
          Hide
          Eli Collins added a comment -

          +1 looks good. Worth considering checking the mount status instead of waiting 5s as a follow-on improvement?

          Show
          Eli Collins added a comment - +1 looks good. Worth considering checking the mount status instead of waiting 5s as a follow-on improvement?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539082/HDFS-3758.003.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestLeaseRenewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2947//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2947//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539082/HDFS-3758.003.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestLeaseRenewer +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2947//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2947//console This message is automatically generated.
          Hide
          Colin Patrick McCabe added a comment -

          fix path

          Show
          Colin Patrick McCabe added a comment - fix path
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539071/TestFuseDFS-fix-0002.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2944//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539071/TestFuseDFS-fix-0002.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2944//console This message is automatically generated.

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development