Details
-
Bug
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
0.17.0
-
None
-
None
-
17.0 + H1979-H2159-H3442
-
Reviewed
Description
I see bunch of datanodes stop verifying local blocks.
".out" showed
-rw-r--r-- 1 hdfs users 614 Jun 23 10:24 datanode.out Exception in thread "org.apache.hadoop.dfs.DataBlockScanner@aadc97" java.lang.NumberFormatException: For input string: "121195228080date=" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Long.parseLong(Long.java:412) at java.lang.Long.valueOf(Long.java:518) at org.apache.hadoop.dfs.DataBlockScanner$LogEntry.parseEntry(DataBlockScanner.java:351) at org.apache.hadoop.dfs.DataBlockScanner.assignInitialVerificationTimes(DataBlockScanner.java:481) at org.apache.hadoop.dfs.DataBlockScanner.run(DataBlockScanner.java:534) at java.lang.Thread.run(Thread.java:619)
Namenode log also showed
2008-06-23 10:24:12,831 WARN org.apache.hadoop.dfs.DataBlockScanner: RuntimeException during DataBlockScanner.run() : java.lang.NumberFormatException: For input string: "121195228080date=" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Long.parseLong(Long.java:412) at java.lang.Long.valueOf(Long.java:518) at org.apache.hadoop.dfs.DataBlockScanner$LogEntry.parseEntry(DataBlockScanner.java:351) at org.apache.hadoop.dfs.DataBlockScanner.assignInitialVerificationTimes(DataBlockScanner.java:481) at org.apache.hadoop.dfs.DataBlockScanner.run(DataBlockScanner.java:534) at java.lang.Thread.run(Thread.java:619)
Datanode was still up and running but no verification.
Jstack didn't show DataBlockScanner.
DataBlockScanner object reads log files "dncp_block_verification.log.curr" and "dncp_block_verification.log.prev". These log files contain lines like
From the error message in the description, it seems to me that the log file was corrupted. ("121195228080date=": the number looks like a timestamp but it is somehow followed by another line.) Could you check your log files? They can be found in your datanode directory .../hadoop-SYSTEM/dfs/data/current