Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.8.0
    • Labels:
      None
    • Environment:

      HDP 2.3 + Apache Phoenix 4.6.0

      Description

      We have an unfortunate situation wherein we potentially execute many RPCs while holding a row lock. This is problem is discussed in detail on the user list thread "Write path blocked by MetaDataEndpoint acquiring region lock". During some situations, the MetaDataEndpoint coprocessor will attempt to refresh it's view of the schema definitions and statistics. This involves taking a rowlock, executing a scan against the local region, and then a scan against a potentially remote statistics table.

      This issue is apparently exacerbated by the use of user-provided timestamps (in my case, the use of the ROW_TIMESTAMP feature, or perhaps as in PHOENIX-2607). When combined with other issues (PHOENIX-2939), we end up with total gridlock in our handler threads – everyone queued behind the rowlock, scanning and rescanning SYSTEM.STATS. Because this happens in the MetaDataEndpoint, the means by which all clients refresh their knowledge of schema, gridlock in that RS can effectively stop all forward progress on the cluster.

        Attachments

        1. PHOENIX-2940.004.patch
          63 kB
          Josh Elser
        2. PHOENIX-2940.003.patch
          61 kB
          Josh Elser
        3. PHOENIX-2940.002.patch
          57 kB
          Josh Elser
        4. PHOENIX-2940.001.patch
          31 kB
          Josh Elser

          Issue Links

            Activity

              People

              • Assignee:
                elserj Josh Elser
                Reporter:
                ndimiduk Nick Dimiduk
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: