fuse-dfs per FD context is not thread safe and can cause segfaults and corruptions

• Key: HADOOP-4399
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Pete Wyckoff
• Reporter: Pete Wyckoff
• Affects Version/s: 0.19.0
• Fix Version/s: 0.18.2

Description

for reads, optimal solution would be to have a per thread (per FD) context - including the buffer. Otherwise, protect the FD context with a mutex as in hadoop-4397 and hadoop-4398.

for writes, should just protect the context with a mutex as in hadoop-4398.

Comments

[Pete Wyckoff]

building on hadoop-4397, I did the following:

1. protect dfs_read accesses to the read buffer with a critical section based on a per file handle mutex.
2. open inits the per file handle mutex
3. release destroys the mutex

Also put a comment in dfs_getattr and a "soft" assertion that dfs->fs global filesystem handle is already set in dfs_init.

No new tests as this is a concurrency fix, but all the existing 10 tests pass.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392026/HADOOP-4399.1.txt
against trunk revision 704186.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3451/console

This message is automatically generated.

[Pete Wyckoff]

TestJobQueueInformation failed because of https://issues.apache.org/jira/browse/HADOOP-4378.

[Zheng Shao]

+1 Code looks good.

[Pete Wyckoff]

the (successful) output of ant -Dlibhdfs=1 -Dfusedfs=1 -Dtestcase=TestFuseDFS

[Pete Wyckoff]

ready to commit.

[Pete Wyckoff]

this patch it turns out did not capture all concurrency problems.

[Pete Wyckoff]

newest patch's output of ant -Dlibhdfs=1 -Dfusedfs=1 -Dtestcase=TestFuseDFS test-contrib.

[Pete Wyckoff]

this is the ubber patch that fixes all concurrency issues for all .

[Pete Wyckoff]

submitting for hudson.

[Pete Wyckoff]

fixed some comments.

[Zheng Shao]

+1
Talked with Pete. Looks good to me.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392297/HADOOP-4399.3.txt
against trunk revision 705420.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 18 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3479/console

This message is automatically generated.

[Pete Wyckoff]

this is ready to commit please.
thx, pete

[dhruba borthakur]

I just committed this. Thanks Pete!

[dhruba borthakur]

This fix is needed for 0.18.2 release as well.

[Hudson]

Integrated in Hadoop-trunk #640 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/640/)