[DRILL-7275] CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0] - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 1.15.0
Fix Version/s: None
Component/s: Storage - Parquet
Labels:
None

Description

CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]

Precondition check fails on line 47 in VarLenFixedEntryReader.java

44 final int expectedDataLen = columnPrecInfo.precision;
45 final int entrySz = 4 + columnPrecInfo.precision;
46 final int readBatch = getFixedLengthMaxRecordsToRead(valuesToRead, entrySz);
47 Preconditions.checkState(readBatch > 0, "Read batch count [%d] should be greater than zero", readBatch);

Stack trace from drillbit.log, also has the failing query.

2019-05-13 14:40:14,090 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 23268c40-ef3a-6349-5901-5762f6888971 issued by scoop_stc: CREATE TABLE TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityProxyDailyReturn AS
WITH
che AS (
 SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_c_CompositeHierarchyEntry_TimeVarying
 WHERE CompositeHierarchyName = 'AxiomaRegion/AxiomaSector/VectorUniverse'
 AND state = 'DupesRemoved'
 AND CompositeLevel = 'AxiomaRegion_1/AxiomaSector_1/VectorUniverse_0'
),
ef AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityDailyReturn_FXAdjusted WHERE Status = 'PresentInRawData'),
d AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_r_BusinessDate WHERE IsWeekday),
x AS
(
 SELECT
 che.CompositeHierarchyName,
 che.State,
 che.CompositeNodeName,
 d.`Date` AS RecordDate,
 COUNT(che.CompositeNodeName) AS countDistinctConstituents,
 COUNT(ef.VectorListingId) AS countDataPoints,
 AVG(ef.DailyReturn) AS AvgReturn, 
 AVG(ef.DailyReturnUSD) AS AvgReturnUSD,
 AVG(ef.NotionalReturnUSD) AS AvgNotionalReturnUSD
 FROM d
 INNER JOIN che ON d.`Date` BETWEEN che.CompositeUltimateChildStartDate AND che.CompositeUltimateChildEndDate
 LEFT OUTER JOIN ef ON d.`Date` = ef.RecordDate AND 'VectorListingId_' || CAST(ef.VectorListingId AS VARCHAR(100)) = che.UltimateChild
 GROUP BY che.CompositeHierarchyName, che.State, che.CompositeNodeName, d.`Date`, d.IsWeekday, d.IsHoliday
)
SELECT * FROM x
2019-05-13 14:40:16,971 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table [tbl_c_EquityProxyDailyReturn].
...
...
2019-05-13 14:40:20,036 [23268c40-ef3a-6349-5901-5762f6888971:frag:6:10] INFO o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
 optional int64 VectorListingId;
 optional int32 RecordDate (DATE);
 required binary Status (UTF8);
 required binary CurrencyISO (UTF8);
 optional double DailyReturn;
 optional double DailyReturnUSD;
 optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status] required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050}, ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8) [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn] optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715}, ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN], 178749}]}]} (Error in parquet record reader.
...
...
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
 optional int64 VectorListingId;
 optional int32 RecordDate (DATE);
 required binary Status (UTF8);
 required binary CurrencyISO (UTF8);
 optional double DailyReturn;
 optional double DailyReturnUSD;
 optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status] required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050}, ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8) [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn] optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715}, ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN], 178749}]}]})
org.apache.drill.common.exceptions.UserException: INTERNAL_ERROR ERROR: Error in parquet record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
 optional int64 VectorListingId;
 optional int32 RecordDate (DATE);
 required binary Status (UTF8);
 required binary CurrencyISO (UTF8);
 optional double DailyReturn;
 optional double DailyReturnUSD;
 optional double NotionalReturnUSD;
}
...
...
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Error in parquet record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
 optional int64 VectorListingId;
 optional int32 RecordDate (DATE);
 required binary Status (UTF8);
 required binary CurrencyISO (UTF8);
 optional double DailyReturn;
 optional double DailyReturnUSD;
 optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status] required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050}, ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8) [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn] optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715}, ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN], 178749}]}]}
 at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleException(ParquetRecordReader.java:271) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:290) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.physical.impl.ScanBatch.internalNext(ScanBatch.java:223) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:271) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 ... 44 common frames omitted
Caused by: java.lang.IllegalStateException: Read batch count [%d] should be greater than zero [0]
 at org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkState(Preconditions.java:509) ~[drill-shaded-guava-23.0.jar:23.0]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLenFixedEntryReader.getEntry(VarLenFixedEntryReader.java:47) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getFixedEntry(VarLenBulkPageReader.java:169) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getEntry(VarLenBulkPageReader.java:132) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next(VarLenColumnBulkInput.java:154) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next(VarLenColumnBulkInput.java:38) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe(VarCharVector.java:624) ~[vector-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe(VarCharVector.java:608) ~[vector-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLengthColumnReaders$VarCharColumn.setSafe(VarLengthColumnReaders.java:168) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLengthValuesColumn.readRecordsInBulk(VarLengthValuesColumn.java:98) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readRecordsInBulk(VarLenBinaryReader.java:114) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readFields(VarLenBinaryReader.java:92) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.BatchReader$VariableWidthReader.readRecords(BatchReader.java:156) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.BatchReader.readBatch(BatchReader.java:43) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:288) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
 ... 46 common frames omitted

Attachments

Activity

People

Assignee:: Salim Achouche

Reporter:: Khurram Faraaz

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 21/May/19 23:01

Updated:: 22/May/19 08:04