Uploaded image for project: 'Apache Lens (Retired)'
  1. Apache Lens (Retired)
  2. LENS-619

Applying Query Launching Constraints before allowing a query to be launched

    Details

      Description

      Lens will always accept a new query from a user and put it in a scheduling queue for processing.

      Next candidate query picked up from scheduling queue for processing will be launched only if all query constraints evaluated on candidate query and launched queries allows the candidate query to be launched, otherwise the candidate query will be added to waiting queries.

      When any launched query is finished, a waiting query selector will select waiting queries for rescheduling using a list of waiting queries selection policy. Every waiting query selection policy will return a list of eligible waiting queries to be rescheduled. Query Selector will calculate intersection of multiple list of eligible waiting queries and add the result of intersection to scheduling queue for reprocessing.

      At initialization, Query Constraints and Waiting Query Selection Policies will be configured using configuration values.

      New Query Constraints and Waiting Query Selection Policies can be added at runtime, without rebuilding and deploying lens module. Drivers should be allowed to add more Query Constraints and Waiting Queries Selection policies.

      Waiting Queries will be persisted across server restarts.

      Query Constraint 1: Allow a candidate query to be launched, only if, cumulative query cost of all current queries launched by the user, who submitted candidate query, is less than a cumulative query cost ceiling for launching a new query.

      Waiting Query Selection Policy 1: Select all waiting queries of the user whose query has finished.

      Query Constraint 2: Allow a candidate query to be launched, only if, current concurrent queries launched on the selected driver are less than max concurrent queries allowed on the driver.

      Waiting Query Selection Policy 2: Select all waiting queries of the driver whose query has finished.

        Attachments

        1. LENS-619.01.patch
          226 kB
          Himanshu Gahlaut

          Issue Links

            Activity

              People

              • Assignee:
                himanshu.gahlaut Himanshu Gahlaut
                Reporter:
                himanshu.gahlaut Himanshu Gahlaut
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: