Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5037

Active NN should trigger its own edit log rolls

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta, 3.0.0-alpha1
    • Fix Version/s: 2.3.0
    • Component/s: ha, namenode
    • Labels:
      None

      Description

      We've seen cases where the SBN/2NN went down, and then users accumulated very very large edit log segments. This causes a slow startup time because the last edit log segment must be read fully to recover it before the NN can start up again. Additionally, in the case of QJM, it can trigger timeouts on recovery or edit log syncing because the very-large segment has to get processed within a certain time bound.

      We could easily improve this by having the NN trigger its own edit log rolls on a configurable size (eg every 256MB)

        Attachments

        1. hdfs-5037-5.patch
          14 kB
          Andrew Wang
        2. hdfs-5037-4.patch
          14 kB
          Andrew Wang
        3. hdfs-5037-3.patch
          13 kB
          Andrew Wang
        4. hdfs-5037-2.patch
          12 kB
          Andrew Wang
        5. hdfs-5037-1.patch
          8 kB
          Andrew Wang

          Activity

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              tlipcon Todd Lipcon
            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: