Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2649 improve incremental stats scalability
  3. IMPALA-7424

Improve in-memory representation of incremental stats

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 2.13.0, Impala 3.1.0
    • Impala 3.1.0
    • Catalog
    • None
    • ghx-label-1

    Description

      Incremental stats are stored in the HMS' parameters map as plain Java Strings. This is suboptimal since Java String class internally uses UTF-16 encoding for the underlying bytes. The idea here is to switch to a byte array representation so that we can reduce the memory usage by half (8 bytes).  We can also compress the byte array using gzip compression and lazily decompress them when needed (typically during the incremental stats computation's finalization phase).

      A prototype of this patch on a real-world Catalog dump showed ~54% JVM heap usage reduction (end-to-end) and ~79% reduction in the heap footprint for the incremental stats.

      Attachments

        Issue Links

          Activity

            People

              bharathv Bharath Vissapragada
              bharathv Bharath Vissapragada
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: