[PIG-4581] thread safe issue in NodeIdGenerator

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.16.0
• Component/s: impl
• Labels: ThreadSafe

Environment

any computer with multiple cores, does not always happen since it is not a determinitic bug. The more threads and real cpu, the more chances it will happen.

• Hadoop Flags: Reviewed
• Release Note: ThreadSafe version of NodeIdGenerator
• Assignee: Remi Catherinot
• Reporter: Remi Catherinot
• Created: 01/Jun/15 08:24
• Updated: 08/Jun/16 20:48
• Resolved: 25/Apr/16 21:04

Description

NodeIdGenerator use an HashMap without any synchronization. In multithreaded environment it may lead to concurrent access to the HashMap which, in some case, triggers an infinite loop inside the HashMap implementation => the parsing of the pig request never ends so the job will never run + it eats a threads in the JVM that just keep burning CPU cycles.

since it is hard to reproduce and it is not a deterministic bug it is impossible to provide a viable unit tests for it to demonstrate both the bug and the fix.