Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9405

Unify MetaStoreClientPool Usage

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Frontend
    • Labels:
      None
    • Epic Color:
      ghx-label-4

      Description

      While trying to resurrect tests/experiments/test_catalog_hms_failures.py I noticed the test TestCatalogHMSFailures::test_start_catalog_before_hms has started to fail. The reason is that when this test was written, only the catalogd was connecting to HMS, but with catalog v2 and ACID integration this is no longer the case.

      It looks like catalog v2 honors initial_hms_cnxn_timeout_s, (at least DirectMetaProvider honors the flag, and I think that is part of the metadata v2 code), but the Frontend Java class has a member variable metaStoreClientPool_ that does not use the flag. It looks like that pool was added for ACID integration.

      The flag initial_hms_cnxn_timeout_s was added in IMPALA-4278 to help with concurrent startup of Impala and HMS.

      Somewhat related to this issue, is that there seems to be multiple places where Impala creates a MetaStoreClientPool, I think it would make more sense to just have one global pool that is used across the process. Doing so would improve connection re-use and possibly decrease the number of HMS connections. There is actually a TODO in DirectMetaProvider as well that says msClientPool_ should be a process wide singleton.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stakiar Sahil Takiar
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: