[SPARK-11020] HistoryServer fails to come up if HDFS takes too long to come out of safe mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.1, 1.6.0
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 09/Oct/15 01:25
• Updated: 01/Nov/15 22:43
• Resolved: 01/Nov/15 22:43

Description

When HDFS is starting up, it starts in safe mode until the NN is able to read the whole fs image and initialize everything. For a really large NN that can take a while.

If the HS is started at the same time, it may give up trying to check whether the event log directory exists, and exit. That's a little sub-optimal; the HS could wait until HDFS came out of safe mode instead.

Issue Links

• links to: [Github] Pull Request #9043 (vanzin)