Deadlock while leaving safe mode.

• Key: HADOOP-4904
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Konstantin Shvachko
• Reporter: Konstantin Shvachko
• Affects Version/s: 0.18.3
• Fix Version/s: 0.18.3

Description

SafeModeInfo.leave() acquires locks in an incorrect order, which causes the deadlock.
It first acquires the SafeModeInfo lock, then calls FSNamesystem.processMisReplicatedBlocks(), which requires the global FSNamesystem lock.
It should be the other way around: first FSNamesystem lock, then SafeModeInfo.

Comments

[Konstantin Shvachko]

Thanks Koji for detecting this.
Here's part of jstack trace.

"org.apache.hadoop.dfs.FSNamesystem$SafeModeMonitor@2b7f6b6d":
 at org.apache.hadoop.dfs.FSNamesystem.processMisReplicatedBlocks(FSNamesystem.java:2918)
 - waiting to lock <0x0000002ada38f558> (a org.apache.hadoop.dfs.FSNamesystem)
 at org.apache.hadoop.dfs.FSNamesystem.access$800(FSNamesystem.java:72)
 at org.apache.hadoop.dfs.FSNamesystem$SafeModeInfo.leave(FSNamesystem.java:3833)
 - locked <0x0000002d34fb4c80> (a org.apache.hadoop.dfs.FSNamesystem$SafeModeInfo)
 at org.apache.hadoop.dfs.FSNamesystem$SafeModeMonitor.run(FSNamesystem.java:4033)
 at java.lang.Thread.run(Thread.java:619)

"IPC Server handler 38 on 8020":
 at org.apache.hadoop.dfs.FSNamesystem$SafeModeInfo.isOn(FSNamesystem.java:3796)
 - waiting to lock <0x0000002d34fb4c80> (a org.apache.hadoop.dfs.FSNamesystem$SafeModeInfo)
 at org.apache.hadoop.dfs.FSNamesystem.isInSafeMode(FSNamesystem.java:4068)
 at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:2820)
 - locked <0x0000002ada38f558> (a org.apache.hadoop.dfs.FSNamesystem)
 at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:2718)
 - locked <0x0000002ada38f558> (a org.apache.hadoop.dfs.FSNamesystem)
 at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:613)
 at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

Found 1 deadlock.

[Konstantin Shvachko]

This should solve the problem. leaveSafeMode() first acquires the FSNamesystem lock and then the SafeMode lock.

[Konstantin Shvachko]

Patch for 0.18

[Tsz Wo (Nicholas), SZE]

+1 patch looks good.

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no tests are needed for this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

[Konstantin Shvachko]

I ran tests. Only one failed: TestMapReduceLocal. This is related to HADOOP-4907.

[Konstantin Shvachko]

I just committed this.

[Hudson]

Integrated in Hadoop-trunk #698 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/698/)