[HADOOP-6834] TFile.append compares initial key against null lastKey

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.1, 0.20.2
• Fix Version/s: 0.22.0
• Component/s: io
• Hadoop Flags: Reviewed
• Assignee: Hong Tang
• Reporter: Ahad Rana
• Created: 22/Jun/10 07:23
• Updated: 12/Dec/11 06:18
• Resolved: 16/Jul/10 18:38

Description

The following code in TFile.KeyReigster.close:

byte[] lastKey = lastKeyBufferOS.getBuffer();
int lastLen = lastKeyBufferOS.size();
if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
lastLen) < 0)

{ throw new IOException("Keys are not added in sorted order"); }

compares the initial key (passed in via TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1 or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results.