[HUDI-111] HBase index QPS resource allocation

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: index
• Assignee: Venkatesh Rudraraju
• Reporter: Venkatesh Rudraraju
• Created: 07/May/19 05:58
• Updated: 28/Aug/19 17:21
• Resolved: 29/May/19 23:35

Description

Provide QPSResourceAllocator interface in hoodie that exposes two APIs:

• getQPSFraction() : Get requested QPS resource from HBase based on current utilization. Default implementation will simply return the requested HBase QPS.
• releaseQPS(): Release the acquired QPS in HBase cluster so other jobs in the cluster can use it. Default implementation does nothing.

This will help throttle inserts on HBase global index based on volume, in a scenario where multiple jobs with different priorities are running. Based on some benchmarks, heavy volume of PUTS (new inserts) into HBase bring down HBase cluster. 

Dynamic QPS resource allocator implementation can solve 2 issues:

• Throttling all inserts across all jobs, such that QPS_Fraction does not exceed 1. (Without this, there are no checks of overall utilization as qps_fraction is purely config driven)
• Set QPS_Fraction based on the job volume and priority, while having checks on total utilization.