[PHOENIX-5688] Investigate better client/server work pacing

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Lars Hofhansl
• Created: 19/Jan/20 22:27
• Updated: 20/Jan/20 21:02

Description

kozdemir shared an intriguing idea that he used for the server side index repair tool, which would equally well apply to the server side deletes and server side UPSERT/SELECT.

The main problem with the current implementation is that we basically send a predicate to the server - DELETE FROM <table> WHERE <condition>. Now the server(s) will go away per region chunk, evaluate the condition and delete whatever matched it... All in tight server loop.

The downside is that (a) a server thread is held up arbitrarily long, (b) there is no way for the server to do any fair queuing, the loop has to finish, and (c) if the server takes too long the client will just time out.

The alternative used to be to do the work on the client instead: Issue a scan with the condition to the server, retrieve the IDs to the client, and then issue nice chunks of deletes back to the server.

The downside here is the extra communication overhead between the server and client (which might be especially taxing for UPSERT/SELECTS).

Kadir's approach is a middle ground:

1. Issue a scan from the client, and send along a chunk size (N rows), when getting the scanner.
2. The server will do N rows worth of work, then return.
3. The client keeps the scanner open, and calls next.
4. Goto #2

This way we get the benefit of both approaches: (1) work close to where the data is, (2) the client can pace the work and the server gets a chance to schedule other work.

 

Issue Links

• relates to: PHOENIX-5672 Unable to find cached index metadata with large UPSERT/SELECT and local index.