Hadoop Common
  1. Hadoop Common
  2. HADOOP-4399

fuse-dfs per FD context is not thread safe and can cause segfaults and corruptions

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.18.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      for reads, optimal solution would be to have a per thread (per FD) context - including the buffer. Otherwise, protect the FD context with a mutex as in hadoop-4397 and hadoop-4398.

      for writes, should just protect the context with a mutex as in hadoop-4398.

      1. TEST-TestFuseDFS.txt
        33 kB
        Pete Wyckoff
      2. TEST-TestFuseDFS.txt
        35 kB
        Pete Wyckoff
      3. HADOOP-4399.3.txt
        50 kB
        Pete Wyckoff
      4. HADOOP-4399.2.txt
        50 kB
        Pete Wyckoff
      5. HADOOP-4399.1.txt
        7 kB
        Pete Wyckoff

        Issue Links

          Activity

          Hide
          Pete Wyckoff added a comment -

          building on hadoop-4397, I did the following:

          1. protect dfs_read accesses to the read buffer with a critical section based on a per file handle mutex.
          2. open inits the per file handle mutex
          3. release destroys the mutex

          Also put a comment in dfs_getattr and a "soft" assertion that dfs->fs global filesystem handle is already set in dfs_init.

          No new tests as this is a concurrency fix, but all the existing 10 tests pass.

          Show
          Pete Wyckoff added a comment - building on hadoop-4397, I did the following: 1. protect dfs_read accesses to the read buffer with a critical section based on a per file handle mutex. 2. open inits the per file handle mutex 3. release destroys the mutex Also put a comment in dfs_getattr and a "soft" assertion that dfs->fs global filesystem handle is already set in dfs_init. No new tests as this is a concurrency fix, but all the existing 10 tests pass.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12392026/HADOOP-4399.1.txt
          against trunk revision 704186.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12392026/HADOOP-4399.1.txt against trunk revision 704186. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/console This message is automatically generated.
          Hide
          Pete Wyckoff added a comment -

          TestJobQueueInformation failed because of https://issues.apache.org/jira/browse/HADOOP-4378.

          Show
          Pete Wyckoff added a comment - TestJobQueueInformation failed because of https://issues.apache.org/jira/browse/HADOOP-4378 .
          Hide
          Zheng Shao added a comment -

          +1 Code looks good.

          Show
          Zheng Shao added a comment - +1 Code looks good.
          Hide
          Pete Wyckoff added a comment -

          the (successful) output of ant -Dlibhdfs=1 -Dfusedfs=1 -Dtestcase=TestFuseDFS

          Show
          Pete Wyckoff added a comment - the (successful) output of ant -Dlibhdfs=1 -Dfusedfs=1 -Dtestcase=TestFuseDFS
          Hide
          Pete Wyckoff added a comment -

          ready to commit.

          Show
          Pete Wyckoff added a comment - ready to commit.
          Hide
          Pete Wyckoff added a comment -

          this patch it turns out did not capture all concurrency problems.

          Show
          Pete Wyckoff added a comment - this patch it turns out did not capture all concurrency problems.
          Hide
          Pete Wyckoff added a comment -

          newest patch's output of ant -Dlibhdfs=1 -Dfusedfs=1 -Dtestcase=TestFuseDFS test-contrib.

          Show
          Pete Wyckoff added a comment - newest patch's output of ant -Dlibhdfs=1 -Dfusedfs=1 -Dtestcase=TestFuseDFS test-contrib.
          Hide
          Pete Wyckoff added a comment -

          this is the ubber patch that fixes all concurrency issues for all .

          Show
          Pete Wyckoff added a comment - this is the ubber patch that fixes all concurrency issues for all .
          Hide
          Pete Wyckoff added a comment -

          submitting for hudson.

          Show
          Pete Wyckoff added a comment - submitting for hudson.
          Hide
          Pete Wyckoff added a comment -

          fixed some comments.

          Show
          Pete Wyckoff added a comment - fixed some comments.
          Hide
          Zheng Shao added a comment -

          +1
          Talked with Pete. Looks good to me.

          Show
          Zheng Shao added a comment - +1 Talked with Pete. Looks good to me.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12392297/HADOOP-4399.3.txt
          against trunk revision 705420.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 18 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12392297/HADOOP-4399.3.txt against trunk revision 705420. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/console This message is automatically generated.
          Hide
          Pete Wyckoff added a comment -

          this is ready to commit please.
          thx, pete

          Show
          Pete Wyckoff added a comment - this is ready to commit please. thx, pete
          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Pete!

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Pete!
          Hide
          dhruba borthakur added a comment -

          This fix is needed for 0.18.2 release as well.

          Show
          dhruba borthakur added a comment - This fix is needed for 0.18.2 release as well.
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #640 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/640/ )

            People

            • Assignee:
              Pete Wyckoff
              Reporter:
              Pete Wyckoff
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development