Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3092

Enable journal protocol based editlog streaming for standby namenode

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 0.23.3, 2.0.0-alpha
    • None
    • ha, namenode
    • None

    Description

      Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits.

      Attachments

        1. Removingfilerdependency.pdf
          59 kB
          Eli Collins
        2. MultipleSharedJournals.pdf
          150 kB
          Suresh Srinivas
        3. MultipleSharedJournals.pdf
          161 kB
          Suresh Srinivas
        4. MultipleSharedJournals.pdf
          191 kB
          Tsz-wo Sze
        5. JNStates.png
          48 kB
          Tsz-wo Sze
        6. ComparisonofApproachesforHAJournals.pdf
          72 kB
          Bikas Saha

        Issue Links

          1.
          Add a service that enables JournalDaemon Sub-task Resolved Suresh Srinivas
          2.
          Journal stream from the namenode to backup needs to have a timeout Sub-task Closed Hari Mankude
          3.
          Add states for journal synchronization in journal daemon Sub-task Resolved Tsz-wo Sze
          4.
          Add JournalManager implementation to JournalDaemons for storing edits Sub-task Open Unassigned
          5.
          Setup configuration for Journal Manager and Journal Services Sub-task Resolved Hari Mankude
          6.
          Sync lagging journal service from the active journal service Sub-task Resolved Brandon Li
          7.
          Implement JournalListener for writing journal to local disk Sub-task Resolved Tsz-wo Sze
          8.
          Implement Journal reader for reading transactions from local edits log or remote Journal node Sub-task Resolved Brandon Li
          9.
          JournalProtocol changes required for introducing epoch and fencing Sub-task Closed Suresh Srinivas
          10.
          Persist the epoch received by the JournalService Sub-task Open Unassigned
          11.
          JournalDaemon (server) should persist the cluster id and nsid in the storage directory Sub-task Resolved Hari Mankude
          12.
          add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. Sub-task Resolved Brandon Li
          13.
          Fix synchronization issues with journal service Sub-task Resolved Hari Mankude
          14.
          Refactor BackupImage and FSEditLog Sub-task Resolved Tsz-wo Sze
          15.
          Create a new journal_edits_dir key to support journal nodes Sub-task Resolved Hari Mankude
          16.
          create http server for Journal Service Sub-task Resolved Brandon Li
          17.
          Handle block pool ID in journal service Sub-task Open Unassigned
          18.
          handle creation time also in journal service Sub-task Open Unassigned
          19.
          stream the edit segments to NameNode when NameNode starts up Sub-task Open Brandon Li
          20.
          create a protocol for journal service synchronziation Sub-task Resolved Brandon Li
          21.
          remove JournalServiceProtocols Sub-task Open Brandon Li
          22.
          GetJournalEditServlet should close output stream only if the stream is used. Sub-task Resolved Brandon Li
          23.
          Enable standby namenode to tail edits from journal service Sub-task Open Brandon Li
          24.
          Replaced Kerberized SSL for journal segment transfer with SPNEGO-based solution Sub-task Open Brandon Li
          25.
          fix the build failure in branch HDFS-3092 Sub-task Resolved Brandon Li

          Activity

            People

              sureshms Suresh Srinivas
              sureshms Suresh Srinivas
              Votes:
              0 Vote for this issue
              Watchers:
              36 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: