Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-3081

MIsleading exception on async stats update after major compaction

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9.0, 4.8.1
    • Labels:
      None

      Description

      Saw an error in some $dayjob testing where, while a RegionServer was going down to due to an exception, there was a scary looking exception about being unable to write to the stats table because an hconnection was closed. Pardon the mis-matched line numbers:

      2016-07-17 07:52:13,229 ERROR [phoenix-update-statistics-0] stats.StatisticsScanner: Failed to update statistics table!
      org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location
        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:309)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:152)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326)
        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301)
        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:161)
        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
        at org.apache.hadoop.hbase.client.HTableWrapper.getScanner(HTableWrapper.java:215)
        at org.apache.phoenix.schema.stats.StatisticsUtil.readStatistics(StatisticsUtil.java:136)
        at org.apache.phoenix.schema.stats.StatisticsWriter.deleteStats(StatisticsWriter.java:230)
        at org.apache.phoenix.schema.stats.StatisticsScanner$StatisticsScannerCallable.call(StatisticsScanner.java:117)
        at org.apache.phoenix.schema.stats.StatisticsScanner$StatisticsScannerCallable.call(StatisticsScanner.java:102)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: hconnection-0x5314972b closed
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1153)
        at org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1133)
        at org.apache.hadoop.hbase.client.CoprocessorHConnection.relocateRegion(CoprocessorHConnection.java:41)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1338)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1162)
        at org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
        ... 17 more
      

      Looking into this some more, this async task to update the stats was still running after a RegionServer already was in the process of shutting down. The RegionServer already closed all of the "userRegions", but, because this task is async, the task is still running and using the RegionServer's CoprocessorHConnection. So, the RegionServer thinks all of the user regions are closed and it is safe to close the HConnection. In reality, there is still code tied to those user regions that might be running (as we can see with the above stacktrace). The next time the StatisticsScannerCallable tries to use the HConnection, it will then error.

      I think the simple fix is to just use the CoprocessorEnvironment to access the RegionServerServices and use the isClosing() and isClosed() methods. This is all pretty minor because the RegionServer is already shutting down, but it is likely misleading to less-experienced users who would think that the last exception in the log is the problem.

      Will put up a patch shortly.

        Attachments

        1. PHOENIX-3081.001.patch
          13 kB
          Josh Elser

          Issue Links

            Activity

              People

              • Assignee:
                elserj Josh Elser
                Reporter:
                elserj Josh Elser
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: