Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-2106

BinaryComparator should avoid doing ByteBuffer.wrap in the hot-path

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.12.2
    • 1.12.3
    • parquet-mr
    • None

    Description

      Background

      While writing out large Parquet tables using Spark, we've noticed that BinaryComparator is the source of substantial churn of extremely short-lived `HeapByteBuffer` objects – It's taking up to 16% of total amount of allocations in our benchmarks, putting substantial pressure on a Garbage Collector:

      profile_48449_alloc_1638494450_sort_by.html

       

      Proposal

      We're proposing to adjust lexicographical comparison (at least) to avoid doing any allocations, since this code lies on the hot-path of every Parquet write, therefore causing substantial churn amplification.

       

       

       

      Attachments

        1. profile_48449_alloc_1638494450_sort_by.html
          488 kB
          Alexey Kudinkin
        2. Screen Shot 2021-12-03 at 3.26.31 PM.png
          635 kB
          Alexey Kudinkin

        Issue Links

          Activity

            People

              alexey.kudinkin Alexey Kudinkin
              alexey.kudinkin Alexey Kudinkin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: