[PHOENIX-2940] Remove STATS RPCs from rowlock

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.8.0

Environment

HDP 2.3 + Apache Phoenix 4.6.0

• Assignee: Josh Elser
• Reporter: Nick Dimiduk
• Created: 26/May/16 01:18
• Updated: 11/Aug/16 08:28
• Resolved: 20/Jun/16 18:34

Description

We have an unfortunate situation wherein we potentially execute many RPCs while holding a row lock. This is problem is discussed in detail on the user list thread "Write path blocked by MetaDataEndpoint acquiring region lock". During some situations, the MetaDataEndpoint coprocessor will attempt to refresh it's view of the schema definitions and statistics. This involves taking a rowlock, executing a scan against the local region, and then a scan against a potentially remote statistics table.

This issue is apparently exacerbated by the use of user-provided timestamps (in my case, the use of the ROW_TIMESTAMP feature, or perhaps as in PHOENIX-2607). When combined with other issues (PHOENIX-2939), we end up with total gridlock in our handler threads – everyone queued behind the rowlock, scanning and rescanning SYSTEM.STATS. Because this happens in the MetaDataEndpoint, the means by which all clients refresh their knowledge of schema, gridlock in that RS can effectively stop all forward progress on the cluster.

Issue Links

• is related to: PHOENIX-2910 Avoid synchronous stats lookup in PTable creation while holding row lock
• relates to: PHOENIX-2939 MetaCache is easily thrashed with default settings
• relates to: PHOENIX-2607 PhoenixMapReduceUtil Upserts with earlier ts (relative to latest data ts) slower by 25x after stats collection
• relates to: PHOENIX-2941 Alternative means of propagating schema changes