[DRILL-7326] Support repeated lists for CTAS parquet format

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.17.0
• Labels: ready-to-commit
• Assignee: Igor Guzenko
• Reporter: Pavel Semenov
• Reviewer: Arina Ielchiieva
• Created: 15/Jul/19 12:40
• Updated: 01/Sep/19 09:42
• Resolved: 27/Aug/19 10:20

Description

STEPS TO REPRODUCE

1. Create json file with which has double nesting array as a value e.g.

   "stringx2":[["asd","фывфы","asg"],["as","acz","gte"],["as","tdf","dsd"]] 

2. Use CTAS to create table in drill with created json file
3. Observe the result

EXPECTED RESULT
Table is created

ACTUAL RESULT
UnsupportedOperationException appears on attempting to create the table

ADDITIONAL INFO
It is possible to create table with with single nested array
Error log

Error: SYSTEM ERROR: UnsupportedOperationException: Unsupported type LIST

Fragment 0:0

Please, refer to logs for more information.

[Error Id: c48c6154-30a1-49c8-ac3b-7c2f898a7f4e on node1.cluster.com:31010]

(java.lang.UnsupportedOperationException) Unsupported type LIST
 org.apache.drill.exec.store.parquet.ParquetRecordWriter.getType():295
 org.apache.drill.exec.store.parquet.ParquetRecordWriter.newSchema():226
 org.apache.drill.exec.store.parquet.ParquetRecordWriter.updateSchema():211
 org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema():160
 org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext():108
 org.apache.drill.exec.record.AbstractRecordBatch.next():186
 org.apache.drill.exec.record.AbstractRecordBatch.next():126
 org.apache.drill.exec.record.AbstractRecordBatch.next():116
 org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
 org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():141
 org.apache.drill.exec.record.AbstractRecordBatch.next():186
 org.apache.drill.exec.physical.impl.BaseRootExec.next():104
 org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83
 org.apache.drill.exec.physical.impl.BaseRootExec.next():94
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():296
 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():283
 java.security.AccessController.doPrivileged():-2
 javax.security.auth.Subject.doAs():422
 org.apache.hadoop.security.UserGroupInformation.doAs():1669
 org.apache.drill.exec.work.fragment.FragmentExecutor.run():283
 org.apache.drill.common.SelfCleaningRunnable.run():38
 java.util.concurrent.ThreadPoolExecutor.runWorker():1149
 java.util.concurrent.ThreadPoolExecutor$Worker.run():624
 java.lang.Thread.run():748 (state=,code=0)

Issue Links

• duplicates: DRILL-6181 CTAS should support writing nested structures (nested lists) to parquet.
• duplicates: DRILL-2241 CTAS fails when writing a repeated list
• supercedes: DRILL-2768 Improve error message for CTAS fails when writing a repeated list
• links to: GitHub Pull Request #1844