Drill queries that use REST API calls have started failing(given below) after recent changes.
They were running fine earlier as the ResultSet returned was just few rows. These queries now fail for even very small resultSets( < 10rows).
Investigating the issue revealed that we introduced a check to limit the Heap usage.
The Wrapper code from exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java that throws this error, i see certain issues. It does seem we use a threshold of 85% of heap usage before throwing that warning and exiting the query.
By using above check, we unintentionally invited all those issues that happen with Java’s Heap usage. JVM does try to make maximum usage of HEAP until Minor or Major GC kicks in i.e GC kicks after there is no more space left in heap(eden or young gen).
The workarounds i can think of in order to resolve this issue are:
- Remove this check altogether so we know why it is filling up Heap.
- Advise the users to stop using REST for querying data.(We did this already). But not all users may not be happy with this suggestion. There could be few dynamic applications(dashboard, monitoring etc).
- Make the threshold high enough so that GC kicks in much better.
If not above options, we have to tune the Heap sizes of drillbit. A quick fix would be to increase the threshold from 85% to 100%(option-3 above).
New Drill configuration property - drill.exec.http.memory.heap.failure.threshold