Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-7326

Support repeated lists for CTAS parquet format

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.16.0
    • Fix Version/s: 1.17.0
    • Component/s: None
    • Labels:

      Description

      STEPS TO REPRODUCE

      1. Create json file with which has double nesting array as a value e.g.
        "stringx2":[["asd","фывфы","asg"],["as","acz","gte"],["as","tdf","dsd"]] 
      2. Use CTAS to create table in drill with created json file
      3. Observe the result

      EXPECTED RESULT
      Table is created

      ACTUAL RESULT
      UnsupportedOperationException appears on attempting to create the table

      ADDITIONAL INFO
      It is possible to create table with with single nested array
      Error log

      Error: SYSTEM ERROR: UnsupportedOperationException: Unsupported type LIST
      
      Fragment 0:0
      
      Please, refer to logs for more information.
      
      [Error Id: c48c6154-30a1-49c8-ac3b-7c2f898a7f4e on node1.cluster.com:31010]
      
      (java.lang.UnsupportedOperationException) Unsupported type LIST
       org.apache.drill.exec.store.parquet.ParquetRecordWriter.getType():295
       org.apache.drill.exec.store.parquet.ParquetRecordWriter.newSchema():226
       org.apache.drill.exec.store.parquet.ParquetRecordWriter.updateSchema():211
       org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema():160
       org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext():108
       org.apache.drill.exec.record.AbstractRecordBatch.next():186
       org.apache.drill.exec.record.AbstractRecordBatch.next():126
       org.apache.drill.exec.record.AbstractRecordBatch.next():116
       org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
       org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():141
       org.apache.drill.exec.record.AbstractRecordBatch.next():186
       org.apache.drill.exec.physical.impl.BaseRootExec.next():104
       org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83
       org.apache.drill.exec.physical.impl.BaseRootExec.next():94
       org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():296
       org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():283
       java.security.AccessController.doPrivileged():-2
       javax.security.auth.Subject.doAs():422
       org.apache.hadoop.security.UserGroupInformation.doAs():1669
       org.apache.drill.exec.work.fragment.FragmentExecutor.run():283
       org.apache.drill.common.SelfCleaningRunnable.run():38
       java.util.concurrent.ThreadPoolExecutor.runWorker():1149
       java.util.concurrent.ThreadPoolExecutor$Worker.run():624
       java.lang.Thread.run():748 (state=,code=0)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                IhorHuzenko Igor Guzenko
                Reporter:
                Pavel.Semenov Pavel Semenov
                Reviewer:
                Arina Ielchiieva
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: