[LENS-619] Applying Query Launching Constraints before allowing a query to be launched

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.3
• Component/s: api, docs, driver-hive, driver-jdbc, server
• Assignee: Himanshu Gahlaut
• Reporter: Himanshu Gahlaut
• Created: 18/Jun/15 20:45
• Updated: 08/Dec/15 07:23
• Resolved: 22/Jul/15 11:22

Description

Lens will always accept a new query from a user and put it in a scheduling queue for processing.

Next candidate query picked up from scheduling queue for processing will be launched only if all query constraints evaluated on candidate query and launched queries allows the candidate query to be launched, otherwise the candidate query will be added to waiting queries.

When any launched query is finished, a waiting query selector will select waiting queries for rescheduling using a list of waiting queries selection policy. Every waiting query selection policy will return a list of eligible waiting queries to be rescheduled. Query Selector will calculate intersection of multiple list of eligible waiting queries and add the result of intersection to scheduling queue for reprocessing.

At initialization, Query Constraints and Waiting Query Selection Policies will be configured using configuration values.

New Query Constraints and Waiting Query Selection Policies can be added at runtime, without rebuilding and deploying lens module. Drivers should be allowed to add more Query Constraints and Waiting Queries Selection policies.

Waiting Queries will be persisted across server restarts.

Query Constraint 1: Allow a candidate query to be launched, only if, cumulative query cost of all current queries launched by the user, who submitted candidate query, is less than a cumulative query cost ceiling for launching a new query.

Waiting Query Selection Policy 1: Select all waiting queries of the user whose query has finished.

Query Constraint 2: Allow a candidate query to be launched, only if, current concurrent queries launched on the selected driver are less than max concurrent queries allowed on the driver.

Waiting Query Selection Policy 2: Select all waiting queries of the driver whose query has finished.

Issue Links

• duplicates: LENS-192 Add submitted query throttling for lens drivers
• incorporates: LENS-687 LauchTime is intermittently set to 0 in response of getStatus API for a successful finished query
• incorporates: LENS-688 Assert on driverQuery fails intermittently
• is blocked by: LENS-630 Using Duration and Fact Weight Based Query Cost Calculator for Hive Driver