[HDFS-12319] DirectoryScanner will throw IllegalStateException when Multiple BP's are present

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2, 2.7.8
• Target Version/s: 3.0.0-beta1, 2.8.2
• Hadoop Flags: Reviewed
• Assignee: Brahma Reddy Battula
• Reporter: Brahma Reddy Battula
• Created: 18/Aug/17 11:22
• Updated: 25/Oct/19 20:26
• Resolved: 25/Aug/17 18:12

Description

Scenario:
Configure "dfs.datanode.directoryscan.interval" as 60 and start federated cluster atleast with two nameservices.

2017-08-18 19:06:37,150 [java.util.concurrent.ThreadPoolExecutor$Worker@37d68b4e[State = -1, empty queue]] ERROR datanode.DirectoryScanner (DirectoryScanner.java:getDiskReport(551)) - Error compiling report for the volume, StorageId: DS-258b5e16-caa3-48c8-a0c8-b16934eb8a0c
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: StopWatch is already running
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:542)
	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:392)
	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:373)
	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:318)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: StopWatch is already running
	at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:612)
	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:579)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more

Issue Links

• is broken by: HDFS-8873 Allow the directoryScanner to be rate-limited