[MAPREDUCE-1820] InputSampler does not create a deep copy of the key object when creating a sample, which causes problems with some formats like SequenceFile<Text,Text>

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.22.0
• Hadoop Flags: Reviewed
• Assignee: Alex Kozlov
• Reporter: Alex Kozlov
• Created: 26/May/10 22:30
• Updated: 08/May/13 21:13
• Resolved: 08/Jul/10 00:11

Description

I tried to use the InputSampler on a SequenceFile<Text,Text> and found that it comes up with duplicate keys in the sample. The problem was tracked down to the fact that the Text object returned from the reader is essentially a wrapper pointing to a byte array, which changes as the sequence file reader progresses. There was also a bug in that the reader should be initialized before the use. The am attaching a patch that fixes both of the issues. --Alex K

Issue Links

• is related to: MAPREDUCE-366 Change org.apache.hadoop.mapred.lib.TotalOrderPartitioner to use new api
• is related to: MAPREDUCE-5225 SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits