[HBASE-1994] Master will lose hlog entries while splitting if region has empty oldlogfile.log

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.90.0
• Fix Version/s: 0.20.3, 0.90.0
• Component/s: master
• Assignee: Lars George
• Reporter: Cosmin Lehene
• Created: 19/Nov/09 14:22
• Updated: 12/Oct/12 06:14
• Resolved: 04/Dec/09 18:38

Description

I don't know yet how an empty oldlogfile.log can exist, however it happened.
Master will fail to put the splits in the region oldlogfile.log if an empty oldlogfile.log already exists there.
This is the master log after I artificially reproduced it by placing an empty oldlogfile.log in /hbase/.META./1028785192/oldlogfile.log and then killed the regionserver that was holding the .META. table

2009-11-19 09:08:36,012 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 1 hlog(s) in hdfs://b0:9000/hbase/.logs/b4,60020,1258637492773
2009-11-19 09:08:36,012 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting hlog 1 of 1: hdfs://b0:9000/hbase/.logs/b4,60020,1258637492773/hlog.dat.1258637493128, length=0
2009-11-19 09:08:36,019 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Adding queue for .META.,,1
2009-11-19 09:08:36,037 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Pushed=795 entries from hdfs://b0:9000/hbase/.logs/b4,60020,1258637492773/hlog.dat.1258637493128
2009-11-19 09:08:36,038 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Thread got 795 to process
2009-11-19 09:08:36,043 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: Old hlog file hdfs://b0:9000/hbase/.META./1028785192/oldlogfile.log already exists. Copying existing file to new file
2009-11-19 09:08:36,079 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: Got while writing region .META.,,1 log java.io.EOFException
2009-11-19 09:08:36,081 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: hlog file splitting completed in 70 millis for hdfs://b0:9000/hbase/.logs/b4,60020,1258637492773

Issue Links

• is related to: HBASE-2358 Store doReconstructionLog will fail if oldlogfile.log is empty and won't load region
• is related to: HBASE-1364 [performance] Distributed splitting of regionserver commit logs