Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3634

reduce the buffer sizes in PipelinedSorter by a small amount

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9.0
    • None
    • None

    Description

      When running in large JVMs with the G1GC, the regionsize maxes out at 32MB.

      PipelinedSorter does allocations of 32MB and 256MB. The actual allocation ends up being 24 bytes over 32M/256M. As a result the 32M allocation ends up taking 2 regions (or 64M), and the 256M ends up taking 9 regions instead of 8. Also leads to fragmentation of the heap, resulting in more Full GCs than required.

      Live humongous region 2 size 33554456 start 0x00007f4918000000 length 2 with remset 0 code roots 0 is marked 0 reclaim candidate 0 type array 1
      
      Dead humongous region 111 size 268435480 start 0x00007f49f2000000 length 9 with remset 0 code roots 0 is marked 0 reclaim candidate 1 type array 1
      

      Attachments

        1. TEZ-3634.01.patch
          1 kB
          Siddharth Seth
        2. TEZ-3634.02.patch
          5 kB
          Siddharth Seth

        Activity

          People

            sseth Siddharth Seth
            sseth Siddharth Seth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: