[DRILL-2383] add exception and pause injections for testing drillbit stability

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Execution - Flow
• Assignee: Parth Chandra
• Reporter: Chris Westin
• Created: 04/Mar/15 19:49
• Updated: 21/Apr/15 05:44
• Resolved: 21/Apr/15 05:34

Description

Use the exception injection mechanism to add exception injections to test a variety of distributed failure scenarios.

Here are some scenarios we've worked out before:
1. Cancellation:
TC1: cancel before any result set is returned
TC2: cancel in the middle of fetching result set
TC3: cancel after all result set are produced but not all are fetched
TC4: cancel after everything is completed and fetched

As test setup, we need:

• query dataset large enough to be sent to different drillbits, e.g., TPCH 100
• queries that force multiple drillbits to work on them; e.g., count ... group by

2. Completed (in each case check all drillbits are still up and running):
TC1: success
TC2: failed query - before query is executed - while sql parsing
TC3: failed query - before query is executed - while sending fragments to other drillbits for execution
TC4: failed query - during query execution

It is currently not possible to create a scenario in which a query may hang.

To check all drillbits up and running and in a clean state, run:

select count from sys.drillbits;

select count(*) from sys.memory;

Issue Links

• contains: DRILL-2384 QueryState is not making it back to the client
• contains: DRILL-2437 enhance exception injection to support session level injections
• contains: DRILL-2382 enhance exception injection to support node-specific injections
• depends upon: DRILL-2245 Improve Drillbit resiliency
• depends upon: DRILL-2275 need implementations of sys tables for drill memory and threads profiles
• relates to: DRILL-2525 The Drill Web UI needs to be reimplemented in terms of a public client