[FLINK-1928] BlobManager can be a bottleneck on large deployments

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.9
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: Robert Metzger
• Created: 22/Apr/15 10:34
• Updated: 28/Feb/19 14:01
• Resolved: 26/Feb/19 15:52

Description

The BlobManager needed more than 2 minutes to distribute 220+ jar files to 50 TaskManagers in a user's setup.

Serving the files for all the TMs from one JM might will quickly make the network of the JM a bottleneck when distributing the files.
Potential solutions could be:

• Use a torrent-style distribution mode
• Use HDFS

Issue Links

• is superceded by: FLINK-6916 FLIP-19: Improved BLOB storage architecture