Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19520

WAL should not be encrypted

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 2.1.1, 2.2.0
    • DStreams
    • None

    Description

      If I/O encryption is enabled in Spark (SPARK-5682), the write ahead logs written by streaming applications will also be encrypted.

      That creates issues because only the driver that wrote those files knows the key to decrypt them. So basically the WAL is unusuable in that scenario.

      Instead, the WAL should not be encrypted by Spark; if the user has a need to encrypt that data, other means (such as HDFS encryption zones) should be used instead.

      Attachments

        Activity

          People

            vanzin Marcelo Masiero Vanzin
            vanzin Marcelo Masiero Vanzin
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: