[DRILL-1146] JSON schema change cause query fail/sqline hang

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Future
• Assignee: Neeraja
• Reporter: Cliff Buchanan
• Created: 15/Jul/14 21:23
• Updated: 15/Sep/15 00:38
• Resolved: 15/Sep/15 00:38

Description

Running a simple select * against this file

{"a" : 0, "b" : 1}
{"a" : 1, "b" : "1"}

yields these results:

add_producer_consumer = true (default)

0: jdbc:drill:zk=local> explain plan for select * from dfs.home.`test.json`;
+------------+------------+
|    text    |    json    |
+------------+------------+
| 00-00    Screen
00-01      ProducerConsumer
00-02        Scan(groupscan=[EasyGroupScan [selectionRoot=/Users/rbuchanan/test.json, columns = null]])
 | {
  "head" : {
    "version" : 1,
    "generator" : {
 |
+------------+------------+
1 row selected (0.992 seconds)
0: jdbc:drill:zk=local> select * from dfs.home.`test.json`;
Exception in thread "c7f12d21-4def-461f-9d1b-ae7af669ae81:frag:0:0 - Producer Thread" java.lang.IllegalArgumentException: You tried to write a VarChar type when you are using a ValueWriter of type NullableBigIntWriterImpl.
        at org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:509)
        at org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:197)
        at org.apache.drill.exec.vector.complex.impl.NullableBigIntWriterImpl.write(NullableBigIntWriterImpl.java:83)
        at org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:140)
        at org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:65)
        at org.apache.drill.exec.vector.complex.fn.JsonReaderWithState.write(JsonReaderWithState.java:52)
        at org.apache.drill.exec.store.easy.json.JSONRecordReader2.next(JSONRecordReader2.java:82)
        at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:138)
        at org.apache.drill.exec.physical.impl.producer.ProducerConsumerBatch$Producer.run(ProducerConsumerBatch.java:122)
        at java.lang.Thread.run(Thread.java:745)

add_producer_consumer = false

0: jdbc:drill:zk=local> alter session set `planner.add_producer_consumer` = false;
+------------+------------+
|     ok     |  summary   |
+------------+------------+
| true       | planner.add_producer_consumer updated. |
+------------+------------+
1 row selected (0.7 seconds)
0: jdbc:drill:zk=local> explain plan for select * from dfs.home.`test.json`;
+------------+------------+
|    text    |    json    |
+------------+------------+
| 00-00    Screen
00-01      Scan(groupscan=[EasyGroupScan [selectionRoot=/Users/rbuchanan/test.json, columns = null]])
 | {
  "head" : {
    "version" : 1,
    "generator" : {
      "type" : "ExplainHandler |
+------------+------------+
1 row selected (0.287 seconds)
0: jdbc:drill:zk=local> select * from dfs.home.`test.json`;
Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while running query.[error_id: "65c1d03a-3641-482e-871b-4c52ea951b4d"
endpoint {
  address: "10.250.0.18"
  user_port: 31010
  control_port: 31011
  data_port: 31012
}
error_type: 0
message: "Failure while running fragment. < IllegalArgumentException:[ You tried to write a VarChar type when you are using a ValueWriter of type NullableBigIntWriterImpl. ]"
]
Error: exception while executing query (state=,code=0)
0: jdbc:drill:zk=local>