[FLINK-9366] Distribute Cache only works for client-accessible files

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.0
• Component/s: Command Line Client, Runtime / Task
• Assignee: Dawid Wysakowicz
• Reporter: Chesnay Schepler
• Created: 15/May/18 09:54
• Updated: 15/Jun/18 15:34
• Resolved: 15/Jun/18 15:34

Description

In FLINK-8620 the distributed cache was modified to the distribute files via the blob store, instead of downloading them from a distributed filesystem.

Previously, taskmanagers would download requested files from the DFS. Now, they retrieve it form the blob store. This requires the client to preemptively upload all files used with distributed cache.

As a result it is no longer possible to use the distributed cache for files that reside in a cluster-internal DFS, as the client cannot download it. This is a regression from the previous behavior and may break existing setups.

aljoscha dawidwys

Issue Links

• links to: GitHub Pull Request #6107