Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19520

WAL should not be encrypted

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 2.1.1, 2.2.0
    • DStreams
    • None

    Description

      If I/O encryption is enabled in Spark (SPARK-5682), the write ahead logs written by streaming applications will also be encrypted.

      That creates issues because only the driver that wrote those files knows the key to decrypt them. So basically the WAL is unusuable in that scenario.

      Instead, the WAL should not be encrypted by Spark; if the user has a need to encrypt that data, other means (such as HDFS encryption zones) should be used instead.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vanzin Marcelo Masiero Vanzin Assign to me
            vanzin Marcelo Masiero Vanzin
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment