Hadoop Common
  1. Hadoop Common
  2. HADOOP-4234

KFS: Allow KFS layer to interface with multiple KFS namenodes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.20.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed KFS glue layer to allow applications to interface with multiple KFS metaservers.

      Description

      The KFS "glue" layer code in Hadoop, currently, only allows an application to interface with a single KFS namenode. The KFS client side library has been modified to allow applications to interface with multiple KFS namenodes. This jira issue is for incorporating the change into the KFS code.

      1. patch.4234
        1 kB
        Sriram Rao

        Activity

        Sriram Rao created issue -
        Sriram Rao made changes -
        Field Original Value New Value
        Attachment patch.4234 [ 12390672 ]
        Hide
        Sriram Rao added a comment -

        A patch file that fixes the issue is attached. This issue was found in a KFS deployment and was tested there. Testing this fix requires a KFS deployment.

        Show
        Sriram Rao added a comment - A patch file that fixes the issue is attached. This issue was found in a KFS deployment and was tested there. Testing this fix requires a KFS deployment.
        Sriram Rao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Lohit Vijayarenu added a comment -

        +1 Looks good.

        Show
        Lohit Vijayarenu added a comment - +1 Looks good.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Should we keep checking whether kfsImpl == null?

        Show
        Tsz Wo Nicholas Sze added a comment - Should we keep checking whether kfsImpl == null?
        Tsz Wo Nicholas Sze made changes -
        Component/s fs [ 12310689 ]
        Assignee Sriram Rao [ sriramsrao ]
        Hide
        Sriram Rao added a comment -

        @Tsz: In the patch that is submitted, in the Init() method, we don't need to check whether kfsImpl == null. The KFS client library code keeps track of all the servers to which the client is connected to; it build a new "handle" for each new server.

        Show
        Sriram Rao added a comment - @Tsz: In the patch that is submitted, in the Init() method, we don't need to check whether kfsImpl == null. The KFS client library code keeps track of all the servers to which the client is connected to; it build a new "handle" for each new server.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390672/patch.4234
        against trunk revision 697306.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3347/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390672/patch.4234 against trunk revision 697306. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3347/console This message is automatically generated.
        Sriram Rao made changes -
        Attachment patch.4234 [ 12390672 ]
        Hide
        Sriram Rao added a comment -

        Providing a new patch since the previous one didn't work.

        Re: no new test, this issue was found in a KFS deployment. To test this code path, a KFS+Hadoop deployment is required; that was done elsewhere.

        Show
        Sriram Rao added a comment - Providing a new patch since the previous one didn't work. Re: no new test, this issue was found in a KFS deployment. To test this code path, a KFS+Hadoop deployment is required; that was done elsewhere.
        Sriram Rao made changes -
        Attachment patch.4234 [ 12390719 ]
        Lohit Vijayarenu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Lohit Vijayarenu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390719/patch.4234
        against trunk revision 698187.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390719/patch.4234 against trunk revision 698187. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3355/console This message is automatically generated.
        Sriram Rao made changes -
        Attachment patch.4234 [ 12390719 ]
        Hide
        Sriram Rao added a comment -

        Updated patch that passes the tests.

        Show
        Sriram Rao added a comment - Updated patch that passes the tests.
        Sriram Rao made changes -
        Attachment patch.4234 [ 12390805 ]
        Lohit Vijayarenu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Lohit Vijayarenu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390805/patch.4234
        against trunk revision 698385.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390805/patch.4234 against trunk revision 698385. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3359/console This message is automatically generated.
        Hide
        Lohit Vijayarenu added a comment -

        I just committed this. Thank you Sriram.

        Show
        Lohit Vijayarenu added a comment - I just committed this. Thank you Sriram.
        Lohit Vijayarenu made changes -
        Hadoop Flags [Reviewed]
        Fix Version/s 0.20.0 [ 12313438 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.19.0 [ 12313211 ]
        Release Note Fix KFS "glue" layer to allow applications to interface with multiple KFS metaservers.
        Resolution Fixed [ 1 ]
        Lohit Vijayarenu made changes -
        Affects Version/s 0.19.0 [ 12313211 ]
        Hide
        Lohit Vijayarenu added a comment -

        contrib test failure is tracked in HADOOP-4257. And test cases are not included as this was tested on KFS deployment.

        Show
        Lohit Vijayarenu added a comment - contrib test failure is tracked in HADOOP-4257 . And test cases are not included as this was tested on KFS deployment.
        Hide
        Owen O'Malley added a comment -

        The change comment for this should probably be in new feature or improvement rather than bug.

        Show
        Owen O'Malley added a comment - The change comment for this should probably be in new feature or improvement rather than bug.
        Hide
        Lohit Vijayarenu added a comment -

        The change comment for this should probably be in new feature or improvement rather than bug.

        Thanks Owen. Changed it with a comment.

        Show
        Lohit Vijayarenu added a comment - The change comment for this should probably be in new feature or improvement rather than bug. Thanks Owen. Changed it with a comment.
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #615 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/615/ )
        Robert Chansler made changes -
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Release Note Fix KFS "glue" layer to allow applications to interface with multiple KFS metaservers. Changed KFS glue layer to allow applications to interface with multiple KFS metaservers.
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Sriram Rao
            Reporter:
            Sriram Rao
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development