Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1623 High Availability Framework for HDFS NN
  3. HDFS-2737

HA: Automatically trigger log rolls periodically on the active NN

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HA branch (HDFS-1623)
    • Fix Version/s: HA branch (HDFS-1623)
    • Component/s: ha, namenode
    • Labels:
      None

      Description

      Currently, the edit log tailing process can only read finalized log segments. So, if the active NN is not rolling its logs periodically, the SBN will lag a lot. This also causes many datanode messages to be queued up in the PendingDatanodeMessage structure.

      To combat this, the active NN needs to roll its logs periodically – perhaps based on a time threshold, or perhaps based on a number of transactions. I'm not sure yet whether it's better to have the NN roll on its own or to have the SBN ask the active NN to roll its logs.

        Attachments

        1. HDFS-2737-HDFS-1623.patch
          38 kB
          Aaron Myers
        2. HDFS-2737-HDFS-1623.patch
          38 kB
          Aaron Myers
        3. HDFS-2737-HDFS-1623.patch
          38 kB
          Aaron Myers
        4. HDFS-2737-HDFS-1623.patch
          37 kB
          Aaron Myers
        5. hdfs-2737-prelim.txt
          35 kB
          Todd Lipcon

        Issue Links

          Activity

            People

            • Assignee:
              atm Aaron Myers
              Reporter:
              tlipcon Todd Lipcon

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment