[DRILL-7014] Format plugin for LTSV files

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.16.0
• Component/s: Storage - Other
• Labels: doc-complete ready-to-commit
• Assignee: Takako Shimamoto
• Reporter: Takako Shimamoto
• Reviewer: Charles Givre
• Created: 29/Jan/19 10:15
• Updated: 11/Apr/19 08:38
• Resolved: 15/Mar/19 22:02

Description

I would like to contribute this plugin to Drill.

Abstract

storage-plugins-override.conf

"storage":{
  dfs: {
    type: "file",
    connection: "file:///",
    formats: {
      "ltsv": {
        "type": "ltsv",
        "extensions": [
          "ltsv"
        ]
      }
    },
    enabled: true
  }
}

sample.ltsv

time:30/Nov/2016:00:55:08 +0900 host:xxx.xxx.xxx.xxx  forwardedfor:-  req:GET /v1/xxx HTTP/1.1  status:200  size:4968 referer:- ua:Java/1.8.0_131 reqtime:2.532 apptime:2.532 vhost:api.example.com
time:30/Nov/2016:00:56:37 +0900 host:xxx.xxx.xxx.xxx  forwardedfor:-  req:GET /v1/yyy HTTP/1.1  status:200  size:412  referer:- ua:Java/1.8.0_201 reqtime:3.580 apptime:3.580 vhost:api.example.com

Run query

root@1805183e9b65:/apache-drill-1.15.0# ./bin/drill-embedded 
Apache Drill 1.15.0
"Drill must go on."
0: jdbc:drill:zk=local> SELECT * FROM dfs.`/apache-drill-1.15.0/sample-data/sample.ltsv` WHERE reqtime > 3.0;
+-----------------------------+------------------+---------------+-----------------------+---------+-------+----------+-----------------+----------+----------+------------------+
|            time             |       host       | forwardedfor  |          req          | status  | size  | referer  |       ua        | reqtime  | apptime  |      vhost       |
+-----------------------------+------------------+---------------+-----------------------+---------+-------+----------+-----------------+----------+----------+------------------+
| 30/Nov/2016:00:56:37 +0900  | xxx.xxx.xxx.xxx  | -             | GET /v1/yyy HTTP/1.1  | 200     | 412   | -        | Java/1.8.0_201  | 3.580    | 3.580    | api.example.com  |
+-----------------------------+------------------+---------------+-----------------------+---------+-------+----------+-----------------+----------+----------+------------------+
1 row selected (6.074 seconds)
0: jdbc:drill:zk=local> 

Issue Links

• links to: GitHub Pull Request #1627