When testing out a cluster with the edit log tailing fast path feature enabled (
HDFS-13150), an unrelated issue caused the NameNode to remain in safe mode for an extended period of time, preventing the NameNode from fully completing its startup sequence. We noticed that the Startup Progress web UI displayed many edit log segments (millions of them).
I traced this problem back to StartupProgress. Within FSEditLogLoader, the loader continually tries to update the startup progress with a new Step any time that it loads edits. Per the Javadoc for StartupProgress, this should be a no-op once startup is completed:
However, StartupProgress only implements that logic once the entire startup sequence has been completed. When FSEditLogLoader calls addStep(), it adds it into the LOADING_EDITS phase:
This phase, in our case, ended long before, so it is nonsensical to continue to add steps to it. I believe it is a bug that StartupProgress accepts such steps instead of ignoring them; once a phase is complete, it should no longer change.