[MAPREDUCE-4593] CombineFileInputFormat must ensure it doesn't dupe locations in its InputSplit objects

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.0.0-alpha
• Component/s: client
• Assignee: Unassigned
• Reporter: Harsh J
• Created: 25/Aug/12 10:52
• Updated: 26/Aug/12 14:39
• Resolved: 26/Aug/12 14:39

Description

Currently it seems possible for CombineFileInputFormat's InputSplit objects to grow to very large sizes due to its non-de-duplication of the locations field. We should probably use a set structure to prevent dupe locations from rising the block locations size of InputSplits sent over by CombineFileInputFormat, as that will help performance and help fix unnecessary warnings/errors over block location limits at the JT/MR AM.

Issue Links

• duplicates: MAPREDUCE-2021 CombineFileInputFormat returns duplicate hostnames in split locations