Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7934

Switch to using Java 8's Base64 impl for incremental stats encoding

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 3.1.0
    • Fix Version/s: Impala 3.2.0
    • Component/s: Catalog
    • Labels:
    • Epic Color:
      ghx-label-4

      Description

      Incremental stats are compressed and Base64 encoded before they are chunked and written to the HMS' partition parameters map. When they are read back, we need to Base64 decode and decompress. 

      For certain incremental stats heavy tables, we noticed that a significant amount of time is spent in these base64 classes (see the attached image for the stack. Unfortunately, I don't have the text version of it).

      Java 8 comes with its own Base64 implementation and that has shown much better perf results [1] compared to apache codec's impl. So consider switching to Java 8's base64 impl.

       [1] http://java-performance.info/base64-encoding-and-decoding-performance/

       

        Attachments

        1. base64.png
          171 kB
          Bharath Vissapragada

          Activity

            People

            • Assignee:
              fredyw Fredy Wijaya
              Reporter:
              bharathv Bharath Vissapragada
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: