[HDFS-3921] NN will prematurely consider blocks missing when entering active state while still in safe mode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2-alpha
• Fix Version/s: 2.0.3-alpha
• Target Version/s: 2.0.3-alpha
• Hadoop Flags: Reviewed
• Assignee: Aaron Myers
• Reporter: Stephen Chu
• Created: 11/Sep/12 22:01
• Updated: 15/Feb/13 13:11
• Resolved: 13/Nov/12 01:31

Description

I shut down all the HDFS daemons in an Highly Available (automatic failover) cluster.

Then I started one NN and it transitioned it to active. No DNs were started, and I saw the red warning link on the NN web UI:
WARNING : There are 36 missing blocks. Please check the logs or run fsck in order to identify the missing blocks.

I clicked this to go to the corrupt_files.jsp page, which ran into the following error:

HTTP ERROR 500

Problem accessing /corrupt_files.jsp. Reason:

    Cannot run listCorruptFileBlocks because replication queues have not been initialized.
Caused by:

java.io.IOException: Cannot run listCorruptFileBlocks because replication queues have not been initialized.
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.listCorruptFileBlocks(FSNamesystem.java:5035)
	at org.apache.hadoop.hdfs.server.namenode.corrupt_005ffiles_jsp._jspService(corrupt_005ffiles_jsp.java:78)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1039)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)