Ran the following query from sqlline:
select a.columns, b.columns, a.columns, b.columns, a.columns from `test` a join `test` b on a.columns=b.columns and a.columns=b.columns order by a.columns limit 1000;
While the query was running, initiated a graceful shutdown from command line on the foreman node. The query failed with the following error message:
Error: RESOURCE ERROR: Hash Join failed to open spill file: /tmp/drill/spill/248a054a-ee63-e795-a44e-d9205df8e9b8_HashJoin_3-2-0/spill7_outer
Looks like somehow the spill directory gets deleted while query is still running when graceful_shutdown is initiated.