Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3826

Concurrent Query Submission leads to Channel Closed Exception

    XMLWordPrintableJSON

Details

    Description

      Frequently seen CHANNEL CLOSED EXCEPTION while running concurrent quries with relatively large LIMIT.

      Here are the details,

      SET UP:

      • Single drillbit running on a single zookeeper node
      • 4G heap size, 8G direct memory
      • Storage plugins: local filesystem, hdfs, hbase

      TEST DATA:

      • A 50,000,000 records json file test.json, with two fields id<Int>, title<String> (approximately 3G).

      SHELL TEST:

      • Running 4 drill shells concurrently with query:
        SELECT id, title from dfs.`test.json` LIMIT 5000000.
      • Queries got canceled. Channel closing between client and server were seen randomly, as an example shown below:
      java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: ChannelClosedException: Channel closed /192.168.4.201:31010 <--> /192.168.4.201:48829.
      
      Fragment 0:0
      
      [Error Id: 0bd2b500-155e-46e0-9f26-bd89fea47a25 on TEST-101:31010]
      	at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
      	at sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
      	at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
      	at sqlline.SqlLine.print(SqlLine.java:1583)
      	at sqlline.Commands.execute(Commands.java:852)
      	at sqlline.Commands.sql(Commands.java:751)
      	at sqlline.SqlLine.dispatch(SqlLine.java:738)
      	at sqlline.SqlLine.begin(SqlLine.java:612)
      	at sqlline.SqlLine.start(SqlLine.java:366)
      	at sqlline.SqlLine.main(SqlLine.java:259)
      

      JDBC TEST:

      • 6 separate threads running the same query: SELECT id, title from dfs.`test.json` LIMIT 10000000, each maintains its own connection. ResultSet, statement and connection are closed finally.
      • Throws the same channel closed exception randomly. Log file were enclosed for review.
      • Memory usage was monitored, all good.

      CROSS STORAGE PLUGINS:

      • The same issue can be found not only in JSON on a file system (local/hdfs), but also in HBASE.
      • The issue was not found in a single thread application.

      Attachments

        1. jdbc-test-client-drillbit.log
          176 kB
          Yiyi Hu
        2. shell-sqlline.log
          18 kB
          Yiyi Hu
        3. shell-test-drillbit.log
          626 kB
          Yiyi Hu

        Issue Links

          Activity

            People

              adeneche Abdel Hakim Deneche
              Yiyi Hu Yiyi Hu
              Rahul Kumar Challapalli Rahul Kumar Challapalli
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: