Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3092

Enable journal protocol based editlog streaming for standby namenode


    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.23.3, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: ha, namenode
    • Labels:


      Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits.

      1. MultipleSharedJournals.pdf
        150 kB
        Suresh Srinivas
      2. MultipleSharedJournals.pdf
        161 kB
        Suresh Srinivas
      3. MultipleSharedJournals.pdf
        191 kB
        Tsz Wo Nicholas Sze
      4. ComparisonofApproachesforHAJournals.pdf
        72 kB
        Bikas Saha
      5. JNStates.png
        48 kB
        Tsz Wo Nicholas Sze
      6. Removingfilerdependency.pdf
        59 kB
        Eli Collins

        Issue Links

        Add a service that enables JournalDaemon Sub-task Resolved Suresh Srinivas
        Journal stream from the namenode to backup needs to have a timeout Sub-task Resolved Hari Mankude
        Add states for journal synchronization in journal daemon Sub-task Resolved Tsz Wo Nicholas Sze
        4. Add JournalManager implementation to JournalDaemons for storing edits Sub-task Open Unassigned
        Setup configuration for Journal Manager and Journal Services Sub-task Resolved Hari Mankude
        Sync lagging journal service from the active journal service Sub-task Resolved Brandon Li
        Implement JournalListener for writing journal to local disk Sub-task Resolved Tsz Wo Nicholas Sze
        Implement Journal reader for reading transactions from local edits log or remote Journal node Sub-task Resolved Brandon Li
        JournalProtocol changes required for introducing epoch and fencing Sub-task Resolved Suresh Srinivas
        10. Persist the epoch received by the JournalService Sub-task Open Unassigned
        JournalDaemon (server) should persist the cluster id and nsid in the storage directory Sub-task Resolved Hari Mankude
        add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. Sub-task Resolved Brandon Li
        Fix synchronization issues with journal service Sub-task Resolved Hari Mankude
        Refactor BackupImage and FSEditLog Sub-task Resolved Tsz Wo Nicholas Sze
        Create a new journal_edits_dir key to support journal nodes Sub-task Resolved Hari Mankude
        create http server for Journal Service Sub-task Resolved Brandon Li
        17. Handle block pool ID in journal service Sub-task Open Unassigned
        18. handle creation time also in journal service Sub-task Open Unassigned
        19. stream the edit segments to NameNode when NameNode starts up Sub-task Open Brandon Li
        create a protocol for journal service synchronziation Sub-task Resolved Brandon Li
        21. remove JournalServiceProtocols Sub-task Open Brandon Li
        GetJournalEditServlet should close output stream only if the stream is used. Sub-task Resolved Brandon Li
        23. Enable standby namenode to tail edits from journal service Sub-task Open Brandon Li
        24. Replaced Kerberized SSL for journal segment transfer with SPNEGO-based solution Sub-task Open Brandon Li
        fix the build failure in branch HDFS-3092 Sub-task Resolved Brandon Li


          Suresh Srinivas created issue -
          Suresh Srinivas made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-3077 [ HDFS-3077 ]
          Suresh Srinivas made changes -
          Assignee Suresh Srinivas [ sureshms ]
          Suresh Srinivas made changes -
          Attachment MultipleSharedJournals.pdf [ 12520984 ]
          Suresh Srinivas made changes -
          Attachment MultipleSharedJournals.pdf [ 12521243 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment MultipleSharedJournals.pdf [ 12521444 ]
          Suresh Srinivas made changes -
          Link This issue relates to HDFS-3204 [ HDFS-3204 ]
          Sanjay Radia made changes -
          Link This issue is depended upon by HDFS-3278 [ HDFS-3278 ]
          Bikas Saha made changes -
          Attachment ComparisonofApproachesforHAJournals.pdf [ 12523531 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment JNStates.png [ 12524775 ]
          Eli Collins made changes -
          Attachment Removingfilerdependency.pdf [ 12526102 ]
          Suresh Srinivas made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Allen Wittenauer made changes -
          Affects Version/s 2.0.0-alpha [ 12320353 ]
          Affects Version/s 0.24.0 [ 12317653 ]


            • Assignee:
              Suresh Srinivas
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              35 Start watching this issue


              • Created: