[TRAFODION-2535] Create index succeeds but resulting index is empty, if dop > 32

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: any
• Fix Version/s: 2.2.0
• Component/s: sql-exe
• Assignee: Suresh Subbiah
• Reporter: Suresh Subbiah
• Created: 13/Mar/17 20:30
• Updated: 15/Sep/18 07:31
• Resolved: 16/Mar/17 23:30

Description

If more than 32 HFiles are created for a single region of an index or table during bulkload, then LOAD COMPLETE step will fail. However if snapshot recovery is not enabled the error is silently ignored and LOAD (or CREATE INDEX which uses LOAD) statement appears to succeed. But the table or index will be empty leading to wrong results for subsequent queries.

To reproduce

create table test1 (a int not null primary key, b int not null) salt using 40 partitions;
insert into test1 values (1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(7,7),(8,8),(9,9),(10,10) ;
upsert using load into test1 select a+10, b+10 from test1;
upsert using load into test1 select a+20, b+20 from test1;
upsert using load into test1 select a+40, b+40 from test1;
upsert using load into test1 select a+80, b+80 from test1;
upsert using load into test1 select a+160, b+160 from test1;
upsert using load into test1 select a+320, b+320 from test1;
upsert using load into test1 select a+640, b+640 from test1;
upsert using load into test1 select a+1280, b+1280 from test1;
upsert using load into test1 select a+2560, b+2560 from test1;
upsert using load into test1 select a+5120, b+5120 from test1;
upsert using load into test1 select a+10240, b+10240 from test1;
upsert using load into test1 select a+20480, b+20480 from test1;
upsert using load into test1 select a+40960, b+40960 from test1;
upsert using load into test1 select a+81920, b+81920 from test1;
exit ;

from HBase shell flush 'TRAFODION.SCH.TEST1'
– new sqlci

cqd HBASE_MIN_BYTES_PER_ESP_PARTITION '10' ;
cqd parallel_num_esps '40' ;
create index ix1 on test1(b) ;
set parserflags 1 ;
select * from table(index_table ix1) ;
– will have 0 rows when this problem is present.

in logs directory (cdl), trafodion.hdfs.log will have a similar message.
2017-03-14 01:28:29,242 ,ERROR ,mapreduce.LoadIncrementalHFiles ,Trying to load more than 32 hfiles to family #1 of region with start key

With the fix we get this message in sqlci and index is not created.
>>create index ix1 on test1(b) ;

• • • ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::doBulkLoad returned error HBASE_DOBULK_LOAD_ERROR(-715). Cause: java.io.IOException: Trying to load more than 32 hfiles to one family of one region
      org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:377)
      org.trafodion.sql.HBulkLoadClient.doSnapshotNBulkLoad(HBulkLoadClient.java:381)
      org.trafodion.sql.HBulkLoadClient.doBulkLoad(HBulkLoadClient.java:454).

• • • ERROR[1081] Loading of index TRAFODION.SCH.IX1 failed unexpectedly.

• • • ERROR[8811] Trying to close a statement that is either not in the open state or has not reached EOF.

— SQL operation failed with errors.
>>showddl test1 ;

CREATE TABLE TRAFODION.SCH.TEST1
(
A INT NO DEFAULT NOT NULL NOT DROPPABLE NOT
SERIALIZED
, B INT NO DEFAULT NOT NULL NOT DROPPABLE NOT
SERIALIZED
, PRIMARY KEY (A ASC)
)
SALT USING 40 PARTITIONS
ATTRIBUTES ALIGNED FORMAT
;

– GRANT SELECT, INSERT, DELETE, UPDATE, REFERENCES ON TRAFODION.SCH.TEST1 TO DB__ROOT WITH GRANT OPTION;

— SQL operation complete.
>>exit ;

Issue Links

• links to: GitHub Pull Request #1008