[FLINK-2834] Global round-robin for temporary directories

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.9.2, 0.10.0
• Component/s: Runtime / Coordination
• Assignee: Greg Hogan
• Reporter: Greg Hogan
• Created: 08/Oct/15 13:15
• Updated: 21/Oct/15 12:42
• Resolved: 21/Oct/15 12:42

Description

Currently each FileIOChannel performs a local round-robin among temporary directories. Initially every writer will be concurrently spilling to the same first directory, then the same second directory, etc. Even when evenly dispersed this effect will naturally occur due to the bunching effect.

One solution is to use a global counter when selecting the temporary path.