Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1052 HDFS scalability with multiple namenodes
  3. HDFS-1637

Federation: FSDataset in Datanode should be created after initial handshake with namenode

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: Federation Branch
    • Fix Version/s: Federation Branch
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently storage is initialized only after handshake with a namenode.However FSDataset is created before handshake and hence before storage initialization. At this time, storage does not have storage directories resulting in FSDataset initialized with no volumes.

      1. HDFS-1637.1.patch
        9 kB
        Jitendra Nath Pandey

        Activity

        Hide
        Jitendra Nath Pandey added a comment -

        Patch uploaded.

        Show
        Jitendra Nath Pandey added a comment - Patch uploaded.
        Hide
        Jitendra Nath Pandey added a comment -

        In this patch, the initFsDataSet is called when setupBP (which performs handshake with namenode) has succeeded for a namenode. Although, initFsDataSet will be called multiple times (once after handshake with each namenode), but FSDataset is initialized only once.

        Show
        Jitendra Nath Pandey added a comment - In this patch, the initFsDataSet is called when setupBP (which performs handshake with namenode) has succeeded for a namenode. Although, initFsDataSet will be called multiple times (once after handshake with each namenode), but FSDataset is initialized only once.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch with test-patch results

        Show
        Suresh Srinivas added a comment - +1 for the patch with test-patch results
        Hide
        Jitendra Nath Pandey added a comment -

        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] -1 system tests framework. The patch failed system tests framework compile.

        The system tests framework errors seem to be unrelated. For example:

        [exec] [iajc] ..../branches/HDFS-1052/src/test/system/aop/org/apache/hadoop/hdfs/HDFSPolicyProviderAspect.aj:24 [error] The import org.apache.hadoop.test.system cannot be resolved
        [exec] [iajc] import org.apache.hadoop.test.system.DaemonProtocol;
        [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

        Show
        Jitendra Nath Pandey added a comment - [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 system tests framework. The patch failed system tests framework compile. The system tests framework errors seem to be unrelated. For example: [exec] [iajc] ..../branches/ HDFS-1052 /src/test/system/aop/org/apache/hadoop/hdfs/HDFSPolicyProviderAspect.aj:24 [error] The import org.apache.hadoop.test.system cannot be resolved [exec] [iajc] import org.apache.hadoop.test.system.DaemonProtocol; [exec] [iajc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        Hide
        Jitendra Nath Pandey added a comment -

        I committed this to the Federation branch.

        Show
        Jitendra Nath Pandey added a comment - I committed this to the Federation branch.

          People

          • Assignee:
            Jitendra Nath Pandey
            Reporter:
            Jitendra Nath Pandey
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development