The current architecture around the BLOB server and cache components seems rather patched up and has some issues regarding concurrency (
FLINK-6380), cleanup, API inconsistencies / currently unused API ( FLINK-6329, FLINK-6008). These make future integration with FLIP-6 or extensions like offloading oversized RPC messages ( FLINK-6046) difficult. We therefore propose an improvement on the current architecture as described below which tackles these issues, provides some cleanup, and enables further BLOB server use cases.
Please refer to https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A+Improved+BLOB+storage+architecture for a full overview on the proposed changes.