Description
Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions like this:
org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1 at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121) at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182) at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1 at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58) at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315) at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245) at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352) ... 11 more
This never happens with spooling enabled - it would not use the SizeBoundQueue at all. So something must set spooling to false.
In fact I found two more tests that set phoenix.query.server.orderBy.spooling.enabled to false:
- OrderByWithServerClientSpoolingDisabledIT, and
- OrderByWithServerMemoryLimitIT
They need to run in their own cluster.
(Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the logs of the jenkins run... Perhaps because it's a direct child of BaseTest...?))