Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-43179

Add option for applications to control saving of metadata in the External Shuffle Service LevelDB

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.5.0
    • Shuffle
    • None

    Description

      Currently, the External Shuffle Service stores application metadata in LevelDB. This is necessary to enable the shuffle server to resume serving shuffle data for an application whose executors registered before the NodeManager restarts. However, the metadata includes the application secret, which is stored in LevelDB without encryption. This is a potential security risk, particularly for applications with high security requirements. While filesystem access control lists (ACLs) can help protect keys and certificates, they may not be sufficient for some use cases. In response, we have decided not to store metadata for these high-security applications in LevelDB. As a result, these applications may experience more failures in the event of a node restart, but we believe this trade-off is acceptable given the increased security risk.

      Attachments

        Activity

          People

            csingh Chandni Singh
            csingh Chandni Singh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: