Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14500

NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
    • Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
    • Component/s: namenode
    • Labels:
      None

      Description

      When testing out a cluster with the edit log tailing fast path feature enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in safe mode for an extended period of time, preventing the NameNode from fully completing its startup sequence. We noticed that the Startup Progress web UI displayed many edit log segments (millions of them).

      I traced this problem back to StartupProgress. Within FSEditLogLoader, the loader continually tries to update the startup progress with a new Step any time that it loads edits. Per the Javadoc for StartupProgress, this should be a no-op once startup is completed:

      StartupProgress.java
       * After startup completes, the tracked data is frozen.  Any subsequent updates
       * or counter increments are no-ops.
      

      However, StartupProgress only implements that logic once the entire startup sequence has been completed. When FSEditLogLoader calls addStep(), it adds it into the LOADING_EDITS phase:

      FSEditLogLoader.java
          StartupProgress prog = NameNode.getStartupProgress();
          Step step = createStartupProgressStep(edits);
          prog.beginStep(Phase.LOADING_EDITS, step);
      

      This phase, in our case, ended long before, so it is nonsensical to continue to add steps to it. I believe it is a bug that StartupProgress accepts such steps instead of ignoring them; once a phase is complete, it should no longer change.

        Attachments

        1. HDFS-14500-branch-2.001.patch
          7 kB
          Erik Krogen
        2. HDFS-14500.001.patch
          8 kB
          Erik Krogen
        3. HDFS-14500.000.patch
          7 kB
          Erik Krogen

          Issue Links

            Activity

              People

              • Assignee:
                xkrogen Erik Krogen
                Reporter:
                xkrogen Erik Krogen
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: