[FLINK-6916] FLIP-19: Improved BLOB storage architecture

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Component/s: Runtime / Coordination, Runtime / Network
• Assignee: Nico Kruber
• Reporter: Nico Kruber
• Created: 14/Jun/17 10:20
• Updated: 26/Feb/19 16:08
• Resolved: 30/Oct/17 15:15

Description

The current architecture around the BLOB server and cache components seems rather patched up and has some issues regarding concurrency (FLINK-6380), cleanup, API inconsistencies / currently unused API (FLINK-6329, FLINK-6008). These make future integration with FLIP-6 or extensions like offloading oversized RPC messages (FLINK-6046) difficult. We therefore propose an improvement on the current architecture as described below which tackles these issues, provides some cleanup, and enables further BLOB server use cases.

Please refer to https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A+Improved+BLOB+storage+architecture for a full overview on the proposed changes.

Issue Links

• relates to: FLINK-5908 Blob Cache can (rarely) get corrupted on failed blob downloads
• relates to: FLINK-6380 BlobService concurrency issues between delete and put/get methods
• relates to: FLINK-6046 Add support for oversized messages during deployment
• relates to: FLINK-7285 allow submission of big job graphs
• supercedes: FLINK-1928 BlobManager can be a bottleneck on large deployments
• supercedes: FLINK-2760 Avoid uploading of already existing jars from the JobClient
• supercedes: FLINK-6329 refactor name-addressable accessors in BlobService classes from returning a URL to a File or InputStream
• links to: GitHub Pull Request #4158
• links to: GitHub Pull Request #4174
• links to: GitHub Pull Request #4176

Sub-Tasks

• remove NAME_ADDRESSABLE mode: Closed
• improve code quality in some tests: Resolved
• remove LibraryCacheManager#getFile(): Resolved
• refactor BlobService#getURL() methods to return a File object: Resolved
• add API to allow job-related BLOBs to be stored: Resolved
• move BLOB ref-counting from LibraryCacheManager to BlobCache: Resolved
• change BlobService sub-classes for permanent and transient BLOBs: Resolved
• include a UUID/random name into the BlobKey: Resolved
• add a TTL to transient BLOB files: Resolved
• avoid unnecessary exceptions in the logs in non-HA cases: Resolved
• remove unused FallbackLibraryCacheManager: Resolved