[DRILL-6406] Apache Drill API /status unreliable - add Proper Health Check to API - /status returns "Running" even though Drill isn't usable eg. java classpath errors for flexjson-*.jar when trying to execute any query

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.10.0, 1.13.0
• Component/s: Server, Client - HTTP, Execution - Monitoring, Tools, Build & Test, Web Server

Environment

MapR Enterprise 5.2 / 6.0

• Assignee: Unassigned
• Reporter: Hari Sekhon
• Created: 11/May/18 14:12
• Updated: 18/Jul/18 18:38

Description

I've come across situations on MapR 5.2 / 6.0 where Drill 1.10 / 1.13 is up and running but there are flexjson jar classpath errors meaning that Drill is not usable so queries fail even though the API gives the impression that Drill is available. This is bad for load balancers.

This API needs to be much more robust and terms of checking if Drill is fully working. I suggest internally doing a health check real query and if that executes successfully and returns the expected result, only then updating the status to OK / Running.

This internal health check query should be periodically re-executed (eg. every 30 secs) and the status set to ERROR or something if if fails.

Issue Links

• relates to: DRILL-5990 Apache Drill /status API returns OK ('Running') even with JRE while queries will not work - make status API reflect the fact that Drill is broken on JRE or stop Drill starting up with JRE