Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3151

DiskStore attempts to map any size BlockId without checking MappedByteBuffer limit

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.0.2
    • 2.3.0
    • Block Manager, Spark Core
    • None

    • IBM 64-bit JVM PPC64

    Description

      DiskStore attempts to memory map the block file in def getBytes. If the file is larger than 2GB (Integer.MAX_VALUE) as specified by FileChannel.map, then the memory map fails.

      Some(channel.map(MapMode.READ_ONLY, segment.offset, segment.length)) # line 104

      Attachments

        Issue Links

        is related to

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-1476 2GB limit in spark for blocks

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed
        relates to

        Sub-task - The sub-task of the issue SPARK-6190 create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

        • Major - Major loss of function.
        • Resolved
        links to

        Pull request #4857 [Github] Pull Request #4857 (squito)

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            eyalfa Eyal Farago
            damonab Damon Brown
            Votes:
            2 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment