Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37593

Reduce default page size by LONG_ARRAY_OFFSET if G1GC and ON_HEAP are used

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • Spark Core, SQL
    • None

    Description

      Spark's tungsten memory model usually tries to allocate memory by one `page` each time and allocated by long[pageSizeBytes/8] in HeapMemoryAllocator.allocate. 

      Remember that java long array needs extra object header (usually 16 bytes in 64bit system), so the really bytes allocated is pageSize+16.

      Assume that the G1HeapRegionSize is 4M and pageSizeBytes is 4M as well. Since every time we need to allocate 4M+16byte memory, so two regions are used with one region only occupies 16byte. Then there are about 50% memory waste.
      It can happenes under different combinations of G1HeapRegionSize (varies from 1M to 32M) and pageSizeBytes (varies from 1M to 64M).

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            EdisonWang EdisonWang
            EdisonWang EdisonWang
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment