Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.10.0
-
None
Description
git.commit.id.abbrev=1e0a14c
The data set used is generated from spark and contains a timestamp stored as int96
[root@qa-node190 framework]# /home/parquet-tools-1.5.1-SNAPSHOT/parquet-meta /home/framework/framework/resources/Datasources/parquet_date/spark_generated/d4/part-r-00000-08c5c621-62ea-4fee-b690-11576eddc39c.snappy.parquet creator: parquet-mr (build 32c46643845ea8a705c35d4ec8fc654cc8ff816d) extra: org.apache.spark.sql.parquet.row.metadata = {"type":"struct","fields":[{"name":"a","type":"integer","nullable":true,"metadata":{}},{"name":"b","type":"strin [more]... file schema: spark_schema ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- a: OPTIONAL INT32 R:0 D:1 b: OPTIONAL BINARY O:UTF8 R:0 D:1 c: OPTIONAL INT32 O:DATE R:0 D:1 d: OPTIONAL INT96 R:0 D:1 row group 1: RC:10000 TS:8661 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- a: INT32 SNAPPY DO:0 FPO:4 SZ:2367/2571/1.09 VC:10000 ENC:RLE,PLAIN,BIT_PACKED b: BINARY SNAPPY DO:0 FPO:2371 SZ:2329/2843/1.22 VC:10000 ENC:RLE,PLAIN_DICTIONARY,BIT_PACKED c: INT32 SNAPPY DO:0 FPO:4700 SZ:1374/1507/1.10 VC:10000 ENC:RLE,PLAIN,BIT_PACKED d: INT96 SNAPPY DO:0 FPO:6074 SZ:1597/1740/1.09 VC:10000 ENC:RLE,PLAIN_DICTIONARY,BIT_PACKED
The below query fails with an ArrayIndexOutOfBoundsException
select convert_from(d, 'TIMESTAMP_IMPALA') from dfs.`/drill/testdata/resource-manager/d4`;
Fails with below error after displaying a bunch of records
Error: SYSTEM ERROR: ArrayIndexOutOfBoundsException: 0
Fragment 1:0
[Error Id: f963f6c0-3306-49a6-9d98-a193c5e7cfee on qa-node190.qa.lab:31010] (state=,code=0)
Attached the logs, profiles and data files
Attachments
Attachments
Issue Links
- relates to
-
DRILL-4337 Drill fails to read INT96 fields from hive generated parquet files
- Resolved
-
DRILL-4742 Using convert_from timestamp_impala gives a random error
- Resolved
- links to