Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1914

Federation: namenode storage directory must be configurable specific to a namenode

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Federation allows common configuration where namenode specific configuration are in the same configuration suffixed by nameservice ID. When namenodes use an external storage directory (NFS), in order to make namenodes use different directories on the external server, the storage directory configuration must also allow specific configuration, using nameservice ID.

      1. HDFS-1914.patch
        10 kB
        Suresh Srinivas
      2. HDFS-1914.1.patch
        11 kB
        Suresh Srinivas

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #660 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/660/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #660 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/660/ )
        Hide
        Suresh Srinivas added a comment -

        I committed the patch.

        Show
        Suresh Srinivas added a comment - I committed the patch.
        Hide
        Suresh Srinivas added a comment -

        > Does it make sense to change DFSUtil.setGenericConf to take a String array as input instead of String..
        String... is essentially a String array right?

        The advantage of this approach is - caller side one could pass variable number of args without having to explicitly construct an array.

        Show
        Suresh Srinivas added a comment - > Does it make sense to change DFSUtil.setGenericConf to take a String array as input instead of String.. String... is essentially a String array right? The advantage of this approach is - caller side one could pass variable number of args without having to explicitly construct an array.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479408/HDFS-1914.1.patch
        against trunk revision 1103987.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.tools.TestJMXGet

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/534//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/534//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/534//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479408/HDFS-1914.1.patch against trunk revision 1103987. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/534//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/534//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/534//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479408/HDFS-1914.1.patch
        against trunk revision 1103958.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI
        org.apache.hadoop.hdfs.TestCrcCorruption
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.tools.TestJMXGet

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/530//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/530//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/530//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479408/HDFS-1914.1.patch against trunk revision 1103958. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/530//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/530//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/530//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        +1 for the patch.
        Minor: Does it make sense to change DFSUtil.setGenericConf to take a String array as input instead of String... ? However, it can be addressed in a different jira.

        Show
        Jitendra Nath Pandey added a comment - +1 for the patch. Minor: Does it make sense to change DFSUtil.setGenericConf to take a String array as input instead of String... ? However, it can be addressed in a different jira.
        Hide
        Suresh Srinivas added a comment -

        Updated, simplified patch.

        Show
        Suresh Srinivas added a comment - Updated, simplified patch.
        Hide
        Suresh Srinivas added a comment -

        Changes:

        1. While merging federation, TestDFSUtil.java changes was missed. I have added it back. In this patch, I have mostly modified testConfModification() method from that test.
        2. The patch makes all the namenode and secondary namenode dir keys nameservice specific.
        Show
        Suresh Srinivas added a comment - Changes: While merging federation, TestDFSUtil.java changes was missed. I have added it back. In this patch, I have mostly modified testConfModification() method from that test. The patch makes all the namenode and secondary namenode dir keys nameservice specific.

          People

          • Assignee:
            Suresh Srinivas
            Reporter:
            Suresh Srinivas
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development