Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3115 Improve task broadcast latency for small tasks
  3. SPARK-3132

Avoid serialization for Array[Byte] in TorrentBroadcast

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersConvert to IssueLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • None
    • None
    • Spark Core
    • None

    Description

      If the input data is a byte array, we should allow TorrentBroadcast to skip serializing and compressing the input.

      To do this, we should add a new parameter (shortCircuitByteArray) to TorrentBroadcast, and then avoid serialization in if the input is byte array and shortCircuitByteArray is true.

      We should then also do compression in task serialization itself instead of doing it in TorrentBroadcast.

      Attachments

        Issue Links

        is related to

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-7448 Implement custom bye array serializer for use in PySpark shuffle

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            rxin Reynold Xin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment