Description
Currently, the External Shuffle Service stores application metadata in LevelDB. This is necessary to enable the shuffle server to resume serving shuffle data for an application whose executors registered before the NodeManager restarts. However, the metadata includes the application secret, which is stored in LevelDB without encryption. This is a potential security risk, particularly for applications with high security requirements. While filesystem access control lists (ACLs) can help protect keys and certificates, they may not be sufficient for some use cases. In response, we have decided not to store metadata for these high-security applications in LevelDB. As a result, these applications may experience more failures in the event of a node restart, but we believe this trade-off is acceptable given the increased security risk.