[ACCUMULO-1759] Empty walogs block recovery after power outage.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.5.0

Environment

• HDP 1.3
  • dfs.durable.sync=true
  • dfs.datanode.synconclose=true
• encrytion patch from ACCUMULO-998

• Assignee: Unassigned
• Reporter: Luke Brassard
• Created: 09/Oct/13 20:00
• Updated: 20/Feb/15 21:04
• Resolved: 20/Feb/15 21:04

Description

Power was abruptly cut to the cluster. Upon restart of HDFS, there was a single Rfile that was missing a block.

Here are some details about the cluster:

• HDP 1.3
  • dfs.durable.sync=true
  • dfs.datanode.synconclose=true
• encrytion patch from ACCUMULO-998

After restarting Accumulo, the Master was complaining with a series of these:

2013-10-09 18:15:16,649 [recovery.HadoopLogCloser] INFO : Waiting for file to be closed /accumulo/wal/10.10.0.1+9997/d52ab315-5ac1-4a5c-9085-67ae29b98b88
2013-10-09 18:15:16,663 [recovery.HadoopLogCloser] INFO : Waiting for file to be closed /accumulo/wal/10.10.0.2+9997/d0192739-74e2-43a0-985f-3ed668259995
2013-10-09 18:15:16,742 [recovery.HadoopLogCloser] INFO : Waiting for file to be closed /accumulo/wal/10.10.0.3+9997/de54e6dc-964a-4b33-b4fb-052e81749913
2013-10-09 18:15:16,833 [recovery.HadoopLogCloser] INFO : Waiting for file to be closed /accumulo/wal/10.10.0.4+9997/cda5daec-25f3-443b-818a-990d3eddd56f

Inspection of the files above showed that they were all empty, but referenced in the !METADATA table. The solution was to move or remove the files from HDFS and delete the references from the metadata. The instance was then able to stabilize and assign the rest of the tablets.

It is unclear why these empty walogs existed in the first place. Is it possible that there should have been data in these walogs? Or should the files have been disregarded since they were empty?

Regarding the Rfile that was missing the block, it was not being referenced by the !METADATA table, so removing it had no negative effect on the instance. Should there have been some reference to this file?

Issue Links

• duplicates: ACCUMULO-3182 Empty or partial WAL header blocks successful recovery