Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5500

Reduce catalog topic size when --compact_catalog_topic is enabled

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Catalog
    • Labels:
      None

      Description

      IMPALA-4029 introduced the use of flatbuffers for storing file and block metadata. The flatbuffers are stored as binary fields in the THdfsPartition thrift struct. However, by doing so, we limit the ability of Thrift to compact catalog update messages sent by the statestore to the impalad nodes when --compact_catalog_topic is set to true. In particular, if that flag is set to true upgrading to Impala 2.9 will result in the size of catalog topic updates to increase by 2X, thereby increasing network and CPU utilization both at the statestore and the impalads. We should consider compressing the catalog update messages sent from the statestore to the impalad nodes when that flag is set to true.

        Attachments

          Activity

            People

            • Assignee:
              dtsirogiannis Dimitris Tsirogiannis
              Reporter:
              dtsirogiannis Dimitris Tsirogiannis
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: