[PHOENIX-5364] Run OrderByWithServerClientSpoolingDisabledIT and OrderByWithServerMemoryLimitIT in their own mini cluster.

• Type: Test
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.15.0, 5.1.0
• Assignee: Lars Hofhansl
• Reporter: Lars Hofhansl
• Created: 22/Jun/19 06:10
• Updated: 21/Dec/19 00:55
• Resolved: 22/Jun/19 06:27

Description

Spooling defaults to true. Yet, I see some tests failing (sometimes) with exceptions like this:

org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: N000531,00D300000000XHP00A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1560900587336.ca5a6d7b1f0951e30d15567c3aa909b1.: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1
 at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:121)
 at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:87)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:357)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getRegionScanner(NonAggregateRegionScannerFactory.java:182)
 at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:187)
 at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:251)
 at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:299)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2904)
 at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3153)
 at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
Caused by: java.lang.IllegalStateException: Queue full. Consider increasing memory threshold or spooling to disk. Max size: 1048576, Current size: 77, Number of elements:1
 at org.apache.phoenix.iterate.SizeBoundQueue.add(SizeBoundQueue.java:58)
 at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:315)
 at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:245)
 at org.apache.phoenix.iterate.NonAggregateRegionScannerFactory.getTopNScanner(NonAggregateRegionScannerFactory.java:352)
 ... 11 more

This never happens with spooling enabled - it would not use the SizeBoundQueue at all. So something must set spooling to false.

In fact I found two more tests that set phoenix.query.server.orderBy.spooling.enabled to false:

• OrderByWithServerClientSpoolingDisabledIT, and
• OrderByWithServerMemoryLimitIT

They need to run in their own cluster.

(Also, OrderByWithServerMemoryLimitIT seems to not run at all (it does not show in the logs of the jenkins run... Perhaps because it's a direct child of BaseTest...?))