Frequently seen CHANNEL CLOSED EXCEPTION while running concurrent quries with relatively large LIMIT.
Here are the details,
- Single drillbit running on a single zookeeper node
- 4G heap size, 8G direct memory
- Storage plugins: local filesystem, hdfs, hbase
- A 50,000,000 records json file test.json, with two fields id<Int>, title<String> (approximately 3G).
- Running 4 drill shells concurrently with query:
SELECT id, title from dfs.`test.json` LIMIT 5000000.
- Queries got canceled. Channel closing between client and server were seen randomly, as an example shown below:
- 6 separate threads running the same query: SELECT id, title from dfs.`test.json` LIMIT 10000000, each maintains its own connection. ResultSet, statement and connection are closed finally.
- Throws the same channel closed exception randomly. Log file were enclosed for review.
- Memory usage was monitored, all good.
CROSS STORAGE PLUGINS:
- The same issue can be found not only in JSON on a file system (local/hdfs), but also in HBASE.
- The issue was not found in a single thread application.