[HBASE-21651] when splitting hlog，occurred exception

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.3.1, 2.1.0
• Assignee: Unassigned
• Reporter: Bo Cui
• Created: 27/Dec/18 10:00
• Updated: 21/Jan/19 10:32
• Resolved: 21/Jan/19 10:32

Description

if hlog contains too many regions,when hlog splits, WALSplitter will open a discovered.edits(FSDataOutputStream) for each region,

WALSplitter.java

protected Map<byte[], SinkWriter> writers = Collections
        .synchronizedMap(new TreeMap<byte[], SinkWriter>(Bytes.BYTES_COMPARATOR));

but datanode has a limit (dfs.datanode.max.transfer.threads), which will cause splithlog to be very slow,even failed.

solution:add an max openEdits value（a configuration）, when openEdits exceeds this value, the oldest FSDataOutputStream will be closed

Issue Links

• is duplicated by: HBASE-19358 Improve the stability of splitting log when do fail over