[DRILL-3722] LIMIT 1 query on top of a dir with 50K files takes ~150 seconds

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.0
• Fix Version/s: Future
• Component/s: Query Planning & Optimization
• Assignee: Unassigned
• Reporter: Rahul Kumar Challapalli
• Created: 28/Aug/15 17:55
• Updated: 12/Oct/15 16:10

Description

git.commit.id.abbrev=445790f

I ran the below query on top of TPCH SF100 lineitem table with 50K files. For the nature of the query, it looks like drill is very slow in handling it.

select * from lineitem limit 1;
+-------------+------------+------------+---------------+-------------+------------------+-------------+--------+---------------+---------------+--------------+---------------+----------------+-----------------+--------------+--------------+
| L_ORDERKEY  | L_PARTKEY  | L_SUPPKEY  | L_LINENUMBER  | L_QUANTITY  | L_EXTENDEDPRICE  | L_DISCOUNT  | L_TAX  | L_RETURNFLAG  | L_LINESTATUS  |  L_SHIPDATE  | L_COMMITDATE  | L_RECEIPTDATE  | L_SHIPINSTRUCT  |  L_SHIPMODE  |  L_COMMENT   |
+-------------+------------+------------+---------------+-------------+------------------+-------------+--------+---------------+---------------+--------------+---------------+----------------+-----------------+--------------+--------------+
| 456884480   | 19781678   | 781679     | 1             | 21.0        | 36932.49         | 0.1         | 0.03   | [B@44f54509   | [B@4287753d   | [B@4b2219ea  | [B@2bd3782f   | [B@48776c23    | [B@185c9300     | [B@65b6f17e  | [B@4da8bb5d  |
+-------------+------------+------------+---------------+-------------+------------------+-------------+--------+---------------+---------------+--------------+---------------+----------------+-----------------+--------------+--------------+
1 row selected (158.976 seconds)

Issue Links

• relates to: DRILL-1681 select with limit on directory with csv files takes quite long to terminate
• relates to: DRILL-1457 Limit operator optimization : push limit operator past exchange operator