Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3092

Enable journal protocol based editlog streaming for standby namenode

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.23.3, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: ha, namenode
    • Labels:
      None

      Description

      Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits.

      1. MultipleSharedJournals.pdf
        150 kB
        Suresh Srinivas
      2. MultipleSharedJournals.pdf
        161 kB
        Suresh Srinivas
      3. MultipleSharedJournals.pdf
        191 kB
        Tsz Wo Nicholas Sze
      4. ComparisonofApproachesforHAJournals.pdf
        72 kB
        Bikas Saha
      5. JNStates.png
        48 kB
        Tsz Wo Nicholas Sze
      6. Removingfilerdependency.pdf
        59 kB
        Eli Collins

        Issue Links

        1.
        Add a service that enables JournalDaemon Sub-task Resolved Suresh Srinivas
         
        2.
        Journal stream from the namenode to backup needs to have a timeout Sub-task Resolved Hari Mankude
         
        3.
        Add states for journal synchronization in journal daemon Sub-task Resolved Tsz Wo Nicholas Sze
         
        4. Add JournalManager implementation to JournalDaemons for storing edits Sub-task Open Unassigned
         
        5.
        Setup configuration for Journal Manager and Journal Services Sub-task Resolved Hari Mankude
         
        6.
        Sync lagging journal service from the active journal service Sub-task Resolved Brandon Li
         
        7.
        Implement JournalListener for writing journal to local disk Sub-task Resolved Tsz Wo Nicholas Sze
         
        8.
        Implement Journal reader for reading transactions from local edits log or remote Journal node Sub-task Resolved Brandon Li
         
        9.
        JournalProtocol changes required for introducing epoch and fencing Sub-task Resolved Suresh Srinivas
         
        10. Persist the epoch received by the JournalService Sub-task Open Unassigned
         
        11.
        JournalDaemon (server) should persist the cluster id and nsid in the storage directory Sub-task Resolved Hari Mankude
         
        12.
        add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. Sub-task Resolved Brandon Li
         
        13.
        Fix synchronization issues with journal service Sub-task Resolved Hari Mankude
         
        14.
        Refactor BackupImage and FSEditLog Sub-task Resolved Tsz Wo Nicholas Sze
         
        15.
        Create a new journal_edits_dir key to support journal nodes Sub-task Resolved Hari Mankude
         
        16.
        create http server for Journal Service Sub-task Resolved Brandon Li
         
        17. Handle block pool ID in journal service Sub-task Open Unassigned
         
        18. handle creation time also in journal service Sub-task Open Unassigned
         
        19. stream the edit segments to NameNode when NameNode starts up Sub-task Open Brandon Li
         
        20.
        create a protocol for journal service synchronziation Sub-task Resolved Brandon Li
         
        21. remove JournalServiceProtocols Sub-task Open Brandon Li
         
        22.
        GetJournalEditServlet should close output stream only if the stream is used. Sub-task Resolved Brandon Li
         
        23. Enable standby namenode to tail edits from journal service Sub-task Open Brandon Li
         
        24. Replaced Kerberized SSL for journal segment transfer with SPNEGO-based solution Sub-task Open Brandon Li
         
        25.
        fix the build failure in branch HDFS-3092 Sub-task Resolved Brandon Li
         

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              35 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development