[GIRAPH-477] Fetching locality info in InputSplitPathOrganizer causes jobs to hang

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Assignee: Alessandro Presta
• Reporter: Alessandro Presta
• Created: 09/Jan/13 19:33
• Updated: 11/Jan/13 05:48
• Resolved: 11/Jan/13 05:24

Description

In the presence of many input splits (>6000 in our case) and input split threads (3000), the loop that fetches locality info for all splits from ZooKeeper becomes a bottleneck. A few workers aren't able to even iterate once over the list, run into increased GC pauses, and eventually time out.
Furthermore, depending on the cluster configuration, it's not always possible/useful to exploit locality.
We should add a flag so that the feature can be optionally disabled.