[IMPALA-1764] Change Llama default resource reservation behavior

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: Impala 2.1.1
• Labels: llama ramp-up resource-management
• Target Version: Impala 2.5.0
• Assignee: Matthew Jacobs
• Reporter: Henry Robinson
• Created: 13/Feb/15 23:26
• Updated: 04/Jan/17 23:58
• Resolved: 10/Nov/15 21:21

Description

(Filing on behalf of Yibing Shi)

According to Impala document, we should enable -rm_always_use_defaults after truning on Llama. Currently the default value for this parameter is false. After customer enables Llama in CM, this parameter will be set as "false". (We even don't have a CM configuration item to set it apart from using safety valve). This often causes problems to customer, especially when table/column statistics are missing.

I understand that setting this parameter to false is because it is very hard to choose the default initial resource reservation values, which depends on the workload and machine configuration. But is it possible for us to use some conservative values for them and then monitor the resource expansion requests in Impala (and warn users if needed). Using conservative resource reservation might increase Impala query response time, but it is better than having query failures because of resource problem.

My suggestion would be:

1. Set the default value of "-rm_always_use_defaults" to true so that its value is correct after enabling Llama. This value won't take any effect when Llama is disabled (-enable_rm=false)
2. Monitor resource expansion activities in Impala.
   • Calculate the percentage of queries that need a resource expansion. If the percentage exceeds a limit (70% for example), warn the customer that they might need to increase the initial resource reservation
   • Calculate the percentage of queries whose reserved resource have been wasted more than 50% (or any other appropriate value). Warn the customer if this percentage is too high.