[IMPALA-9609] Minimize Frontend activity in executor only Impalas

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Implemented
• Fix Version/s: Impala 4.0.0
• Component/s: Backend, Frontend
• Epic Color: ghx-label-4
• Assignee: Csaba Ringhofer
• Reporter: Csaba Ringhofer
• Created: 06/Apr/20 22:33
• Updated: 25/Apr/20 12:12
• Resolved: 25/Apr/20 12:12

Description

Currently Impala both starts a JVM and creates a Frontend in executor mode impalads:

JVM:
https://github.com/apache/impala/blob/04fd9ae268d89b07e2a692a916bf2ddcfb2e351b/be/src/service/impalad-main.cc#L61
The 3rd argument of is init_jvm. This is actually needed for some executor features, e.g. Hive UDFs and HBase.

Frontend:
https://github.com/apache/impala/blob/d877dbc572e95f086b37142e1d41231d5b6b7f9f/be/src/runtime/exec-env.cc#L250

The frontend code itself doesn't really seem to care about whether it serves as a coordinator.

It would be great to remove the Frontend object from executors completely, but this may be tricky as Java features used by the executor can have widespread dependencies.

The critical part is to avoid starting services that are both not needed and are problematic, e.g. connecting to HMS and Auth services. These can consume resource and cause errors if not configured well.

The executor only setting is only tested by custom_cluster test, so probably we'll need to extend those to ensure that HBase/Hive UDFs are not broken.

Issue Links

• relates to: IMPALA-9638 Don't create unnecessary threads in executor-only impalads
• relates to: IMPALA-9375 Remove DirectMetaProvider usage from CatalogMetaProvider