Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-5364

Run OrderByWithServerClientSpoolingDisabledIT and OrderByWithServerMemoryLimitIT in their own mini cluster.

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.15.0, 5.1.0
    • Labels:
      None

      Description

      Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions like this:

      org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1
       at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121)
       at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87)
       at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357)
       at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182)
       at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187)
       at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251)
       at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299)
       at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904)
       at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153)
       at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
       at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
       at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
      Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1
       at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58)
       at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315)
       at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245)
       at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352)
       ... 11 more
      

      This never happens with spooling enabled - it would not use the SizeBoundQueue at all. So something must set spooling to false.

      In fact I found two more tests that set phoenix.query.server.orderBy.spooling.enabled to false:

      • OrderByWithServerClientSpoolingDisabledIT, and
      • OrderByWithServerMemoryLimitIT

      They need to run in their own cluster.

      (Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the logs of the jenkins run... Perhaps because it's a direct child of BaseTest...?))

        Attachments

        1. 5364.txt
          2 kB
          Lars Hofhansl

          Activity

            People

            • Assignee:
              larsh Lars Hofhansl
              Reporter:
              larsh Lars Hofhansl
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: