[TEZ-3245] Race condition between addKnowInput and clearAndGetOnepartition of InputHost

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Assignee: Zhiyuan Yang
• Reporter: Zhiyuan Yang
• Created: 06/May/16 16:48
• Updated: 22/Aug/17 00:02
• Resolved: 09/May/16 21:11

Description

In ShuffleManager::addKnownInput, InputHost::addKnowInput is invoked without locking. At the same time, InputHost::clearAndGetOnePartition can be invoked with lock(ShuffleManager.RunShuffleCallable::callInternal => ShuffleManager::constructFetcherForHost => InputHosst::clearAndGetOnePartition). The method clearAndGetOnePartition does two things: drain input attempt of a host and then delete the entry for this host. If addKnownInput happens between these two operation, the added input will be deleted without being handled.

Issue Links

• relates to: TEZ-3207 Add support for fetching multiple partitions from the same source task to UnorderedKVInput