Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3092 Enable journal protocol based editlog streaming for standby namenode
  3. HDFS-3201

Implement Journal reader for reading transactions from local edits log or remote Journal node

    Details

    • Hadoop Flags:
      Reviewed
    1. HDFS-3201.HDFS-3092.patch
      23 kB
      Brandon Li
    2. HDFS-3201.HDFS-3092.patch
      22 kB
      Brandon Li
    3. HDFS-3201.HDFS-3092.patch
      27 kB
      Brandon Li
    4. HDFS-3201.HDFS-3092.patch
      30 kB
      Brandon Li

      Issue Links

        Activity

        Brandon Li created issue -
        Hide
        Brandon Li added a comment -

        After a new(or a rejoining) JournalNode joins the cluster and starts accepting streamed transactions, it may not have all the previously finalized edit segments in its local storage directory.

        To synchronize this JournalNode (J1) with other active JournalNode, this JournalNode needs to copy the finalized edit segments from another JournalNode (J2) which has all the finalized edit segments.

        The remote-reader is used by J1 to get edit segments from J2. The local-reader is used by J2 to read edit segment from its local storage directory.

        Show
        Brandon Li added a comment - After a new(or a rejoining) JournalNode joins the cluster and starts accepting streamed transactions, it may not have all the previously finalized edit segments in its local storage directory. To synchronize this JournalNode (J1) with other active JournalNode, this JournalNode needs to copy the finalized edit segments from another JournalNode (J2) which has all the finalized edit segments. The remote-reader is used by J1 to get edit segments from J2. The local-reader is used by J2 to read edit segment from its local storage directory.
        Hide
        Brandon Li added a comment -

        The current plan is to use http server to transfer finalized edit log segments.

        This patch includes the following changes:
        1. create a http server for journal node
        2. have a new get image severlet to be used only by journal node
        3. have a test to test file copy between journal nodes

        Show
        Brandon Li added a comment - The current plan is to use http server to transfer finalized edit log segments. This patch includes the following changes: 1. create a http server for journal node 2. have a new get image severlet to be used only by journal node 3. have a test to test file copy between journal nodes
        Brandon Li made changes -
        Field Original Value New Value
        Attachment HDFS-3201.HDFS-3092.patch [ 12522598 ]
        Brandon Li made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522598/HDFS-3201.HDFS-3092.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2274//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522598/HDFS-3201.HDFS-3092.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2274//console This message is automatically generated.
        Brandon Li made changes -
        Link This issue relates to HDFS-3283 [ HDFS-3283 ]
        Hide
        Brandon Li added a comment -

        Split this JIRA into two for better issue management:
        HDFS-3283 is used for creating JN http server
        and this JIRA is to enable the edit log transfer.

        Show
        Brandon Li added a comment - Split this JIRA into two for better issue management: HDFS-3283 is used for creating JN http server and this JIRA is to enable the edit log transfer.
        Hide
        Brandon Li added a comment -

        rebased the patch with the branch head

        Show
        Brandon Li added a comment - rebased the patch with the branch head
        Brandon Li made changes -
        Attachment HDFS-3201.HDFS-3092.patch [ 12523051 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12523051/HDFS-3201.HDFS-3092.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2293//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523051/HDFS-3201.HDFS-3092.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2293//console This message is automatically generated.
        Brandon Li made changes -
        Attachment HDFS-3201.HDFS-3092.patch [ 12523278 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12523278/HDFS-3201.HDFS-3092.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2303//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523278/HDFS-3201.HDFS-3092.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2303//console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi Brandon, you don't have to submit the patch since it is not for trunk.

        Show
        Tsz Wo Nicholas Sze added a comment - Hi Brandon, you don't have to submit the patch since it is not for trunk.
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -
        • You are actually using DFS_JOURNAL_USER_NAME_KEY in the code, please update the TODO.
          +  // TODO: journal service should use DFS_JOURNAL_USER_NAME_KEY when security is
          +  // enabled
          
        • Please remove Journal.getStorageInfo()
        • Please use JOURNAL_ATTRIBUTE_KEY
          +          httpServer.setAttribute("localjournal", localJournal);
          
        • Move editsToDownload(..) to the test since it is only used there.
        Show
        Tsz Wo Nicholas Sze added a comment - You are actually using DFS_JOURNAL_USER_NAME_KEY in the code, please update the TODO. + // TODO: journal service should use DFS_JOURNAL_USER_NAME_KEY when security is + // enabled Please remove Journal.getStorageInfo() Please use JOURNAL_ATTRIBUTE_KEY + httpServer.setAttribute( "localjournal" , localJournal); Move editsToDownload(..) to the test since it is only used there.
        Hide
        Brandon Li added a comment -

        The new patch addressed Nicholas' comments. Thanks!

        Show
        Brandon Li added a comment - The new patch addressed Nicholas' comments. Thanks!
        Brandon Li made changes -
        Attachment HDFS-3201.HDFS-3092.patch [ 12523287 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        I have committed this.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good. I have committed this.
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s Shared journals (HDFS-3092) [ 12320553 ]
        Resolution Unresolved [ 9 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        8d 19h 32m 1 Brandon Li 13/Apr/12 21:40
        Patch Available Patch Available Open Open
        5d 4h 39m 1 Tsz Wo Nicholas Sze 19/Apr/12 02:20
        Open Open Resolved Resolved
        3h 56m 1 Tsz Wo Nicholas Sze 19/Apr/12 06:17

          People

          • Assignee:
            Brandon Li
            Reporter:
            Brandon Li
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development