Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3151

DiskStore attempts to map any size BlockId without checking MappedByteBuffer limit

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.0.2
    • 2.3.0
    • Block Manager, Spark Core
    • None

    • IBM 64-bit JVM PPC64

    Description

      DiskStore attempts to memory map the block file in def getBytes. If the file is larger than 2GB (Integer.MAX_VALUE) as specified by FileChannel.map, then the memory map fails.

      Some(channel.map(MapMode.READ_ONLY, segment.offset, segment.length)) # line 104

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-1476 2GB limit in spark for blocks

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Sub-task - The sub-task of the issue SPARK-6190 create LargeByteBuffer abstraction for eliminating 2GB limit on blocks

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #4857 [Github] Pull Request #4857 (squito)

          Activity

            People

              eyalfa Eyal Farago
              damonab Damon Brown
              Votes:
              2 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: