[HDFS-8179] DFSClient#getServerDefaults returns null within 1 hour of system start

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Xiaoyu Yao
• Reporter: Xiaoyu Yao
• Created: 17/Apr/15 21:59
• Updated: 05/Jan/17 22:41
• Resolved: 20/Apr/15 22:44

Description

We recently hit NPE during Ambari Oozie service check. The failed hdfs command is below. It repros sometimes and then go away after the cluster runs for a while.

[ambari-qa@c6401 ~]$ hadoop --config /etc/hadoop/conf fs -rm -r /user/ambari-qa/mapredsmokeoutput
rm: Failed to get server trash configuration: null. Consider using -skipTrash option

With additional tracing, the failure was located to the following stack.

15/04/17 20:57:12 DEBUG fs.Trash: Failed to get server trash configuration
java.lang.NullPointerException
	at org.apache.hadoop.fs.Trash.moveToAppropriateTrash(Trash.java:86)
	at org.apache.hadoop.fs.shell.Delete$Rm.moveToTrash(Delete.java:117)
	at org.apache.hadoop.fs.shell.Delete$Rm.processPath(Delete.java:104)
	at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:321)
	at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:293)
	at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:275)
	at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:259)
	at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:205)
	at org.apache.hadoop.fs.shell.Command.run(Command.java:166)
	at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
rm: Failed to get server trash configuration: null. Consider using -skipTrash option

Issue Links

• is broken by: HDFS-6841 Use Time.monotonicNow() wherever applicable instead of Time.now()
• is related to: HDFS-8163 Using monotonicNow for block report scheduling causes test failures on recently restarted systems