Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-1673

Flatten function can not work well with nested arrays.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 1.1.0
    • Component/s: Functions - Drill
    • Labels:
      None
    • Environment:

      0.7.0

      Description

      Flatten function failed to scan nested arrays , for example something like ""[[ ]]"".

      The only difference of JSON files between below 2 tests is
      "num":[1,2,3]
      VS
      "num":[[1,2,3]]

      ==============Test 1 (Works well):==============
      file:

      {"fixed_column":"abc", "list_column":[{"id1":"1","name":"zhu", "num": [1,2,3]}, {"id1":"2","name":"hao", "num": [4,5,6]} ]}
      

      SQL:

      0: jdbc:drill:zk=local> select t.`fixed_column` as fixed_column, flatten(t.`list_column`)  from dfs.root.`/Users/hzu/Documents/sharefolder/hp/n2.json` as t;
      +--------------+------------+
      | fixed_column |   EXPR$1   |
      +--------------+------------+
      | abc          | {"id1":"1","name":"zhu","num":[1,2,3]} |
      | abc          | {"id1":"2","name":"hao","num":[4,5,6]} |
      +--------------+------------+
      2 rows selected (0.154 seconds)
      

      ==============Test 2 (Failed):==============
      file:

      {"fixed_column":"abc", "list_column":[{"id1":"1","name":"zhu", "num": [[1,2,3]]}, {"id1":"2","name":"hao", "num": [[4,5,6]]} ]}
      

      SQL:

      0: jdbc:drill:zk=local>  select t.`fixed_column` as fixed_column, flatten(t.`list_column`)  from dfs.root.`/Users/hzu/Documents/sharefolder/hp/n3.json` as t;
      +--------------+------------+
      | fixed_column |   EXPR$1   |
      +--------------+------------+
      Query failed: Failure while running fragment.[ df28347b-fac1-497d-b9c5-a313ba77aa4d on 10.250.0.115:31010 ]
        (java.lang.UnsupportedOperationException) 
          org.apache.drill.exec.vector.complex.RepeatedListVector$RepeatedListTransferPair.splitAndTransfer():339
          org.apache.drill.exec.vector.complex.RepeatedMapVector$SingleMapTransferPair.splitAndTransfer():305
          org.apache.drill.exec.test.generated.FlattenerGen22.flattenRecords():93
          org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.doWork():152
          org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():89
          org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext():118
          org.apache.drill.exec.record.AbstractRecordBatch.next():106
          org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():124
          org.apache.drill.exec.record.AbstractRecordBatch.next():86
          org.apache.drill.exec.record.AbstractRecordBatch.next():76
          org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():52
          org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
          org.apache.drill.exec.record.AbstractRecordBatch.next():106
          org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():124
          org.apache.drill.exec.physical.impl.BaseRootExec.next():67
          org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():122
          org.apache.drill.exec.physical.impl.BaseRootExec.next():57
          org.apache.drill.exec.work.fragment.FragmentExecutor.run():105
          org.apache.drill.exec.work.WorkManager$RunnableWrapper.run():249
          .......():0
      
      
      java.lang.RuntimeException: java.sql.SQLException: Failure while executing query.
      	at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
      	at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
      	at sqlline.SqlLine.print(SqlLine.java:1809)
      	at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
      	at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
      	at sqlline.SqlLine.dispatch(SqlLine.java:889)
      	at sqlline.SqlLine.begin(SqlLine.java:763)
      	at sqlline.SqlLine.start(SqlLine.java:498)
      	at sqlline.SqlLine.main(SqlLine.java:460)
      

        Attachments

        1. DRILL-1673.patch
          5 kB
          Mehant Baid
        2. DRILL-1673-reopened.patch
          17 kB
          Jason Altekruse
        3. error.log
          6 kB
          Rahul Challapalli

          Issue Links

            Activity

              People

              • Assignee:
                jaltekruse Jason Altekruse
                Reporter:
                haozhu Hao Zhu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: