KFS: Allow KFS layer to interface with multiple KFS namenodes

• Key: HADOOP-4234
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Minor
• Assignee: Sriram Rao
• Reporter: Sriram Rao
• Component/s: fs
• Affects Version/s: 0.19.0
• Fix Version/s: 0.20.0

Description

The KFS "glue" layer code in Hadoop, currently, only allows an application to interface with a single KFS namenode. The KFS client side library has been modified to allow applications to interface with multiple KFS namenodes. This jira issue is for incorporating the change into the KFS code.

Comments

[Sriram Rao]

A patch file that fixes the issue is attached. This issue was found in a KFS deployment and was tested there. Testing this fix requires a KFS deployment.

[Lohit Vijayarenu]

+1 Looks good.

[Tsz Wo (Nicholas), SZE]

Should we keep checking whether kfsImpl == null?

[Sriram Rao]

@Tsz: In the patch that is submitted, in the Init() method, we don't need to check whether kfsImpl == null. The KFS client library code keeps track of all the servers to which the client is connected to; it build a new "handle" for each new server.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390672/patch.4234
against trunk revision 697306.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

-1 patch. The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3347/console

This message is automatically generated.

[Sriram Rao]

Providing a new patch since the previous one didn't work.

Re: no new test, this issue was found in a KFS deployment. To test this code path, a KFS+Hadoop deployment is required; that was done elsewhere.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390719/patch.4234
against trunk revision 698187.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

-1 core tests. The patch failed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/console

This message is automatically generated.

[Sriram Rao]

Updated patch that passes the tests.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390805/patch.4234
against trunk revision 698385.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/console

This message is automatically generated.

[Lohit Vijayarenu]

I just committed this. Thank you Sriram.

[Lohit Vijayarenu]

contrib test failure is tracked in HADOOP-4257. And test cases are not included as this was tested on KFS deployment.

[Owen O'Malley]

The change comment for this should probably be in new feature or improvement rather than bug.

[Lohit Vijayarenu]

The change comment for this should probably be in new feature or improvement rather than bug.

Thanks Owen. Changed it with a comment.

[Hudson]

Integrated in Hadoop-trunk #615 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/615/)