Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3826

Concurrent Query Submission leads to Channel Closed Exception

    XMLWordPrintableJSON

    Details

      Description

      Frequently seen CHANNEL CLOSED EXCEPTION while running concurrent quries with relatively large LIMIT.

      Here are the details,

      SET UP:

      • Single drillbit running on a single zookeeper node
      • 4G heap size, 8G direct memory
      • Storage plugins: local filesystem, hdfs, hbase

      TEST DATA:

      • A 50,000,000 records json file test.json, with two fields id<Int>, title<String> (approximately 3G).

      SHELL TEST:

      • Running 4 drill shells concurrently with query:
        SELECT id, title from dfs.`test.json` LIMIT 5000000.
      • Queries got canceled. Channel closing between client and server were seen randomly, as an example shown below:
      java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: ChannelClosedException: Channel closed /192.168.4.201:31010 <--> /192.168.4.201:48829.
      
      Fragment 0:0
      
      [Error Id: 0bd2b500-155e-46e0-9f26-bd89fea47a25 on TEST-101:31010]
      	at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
      	at sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
      	at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
      	at sqlline.SqlLine.print(SqlLine.java:1583)
      	at sqlline.Commands.execute(Commands.java:852)
      	at sqlline.Commands.sql(Commands.java:751)
      	at sqlline.SqlLine.dispatch(SqlLine.java:738)
      	at sqlline.SqlLine.begin(SqlLine.java:612)
      	at sqlline.SqlLine.start(SqlLine.java:366)
      	at sqlline.SqlLine.main(SqlLine.java:259)
      

      JDBC TEST:

      • 6 separate threads running the same query: SELECT id, title from dfs.`test.json` LIMIT 10000000, each maintains its own connection. ResultSet, statement and connection are closed finally.
      • Throws the same channel closed exception randomly. Log file were enclosed for review.
      • Memory usage was monitored, all good.

      CROSS STORAGE PLUGINS:

      • The same issue can be found not only in JSON on a file system (local/hdfs), but also in HBASE.
      • The issue was not found in a single thread application.

        Attachments

        1. shell-test-drillbit.log
          626 kB
          Yiyi Hu
        2. shell-sqlline.log
          18 kB
          Yiyi Hu
        3. jdbc-test-client-drillbit.log
          176 kB
          Yiyi Hu

          Issue Links

            Activity

              People

              • Assignee:
                adeneche Abdel Hakim Deneche
                Reporter:
                Yiyi Hu Yiyi Hu
                Reviewer:
                Rahul Kumar Challapalli
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: