Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.1, 2.2.0
    • Component/s: DStreams
    • Labels:
      None

      Description

      If I/O encryption is enabled in Spark (SPARK-5682), the write ahead logs written by streaming applications will also be encrypted.

      That creates issues because only the driver that wrote those files knows the key to decrypt them. So basically the WAL is unusuable in that scenario.

      Instead, the WAL should not be encrypted by Spark; if the user has a need to encrypt that data, other means (such as HDFS encryption zones) should be used instead.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Vanzin
              Reporter:
              vanzin Marcelo Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: