[TRAFODION-678] LP Bug: 1378127 - ExpHbaseInterface::nextRow sees OutOfOrderScannerNextException on hbase 0.98

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Labels: hbase
• External issue URL: https://bugs.launchpad.net/bugs/1378127
• Assignee: Atanu Mishra
• Reporter: Weishiun Tsai
• Created: 22/Jul/15 18:17
• Updated: 30/Aug/15 23:23
• Resolved: 30/Aug/15 23:23

Description

When running QA regression tests, ExpHbaseInterface::nextRow() would often run into hbase OutOfOrderScannerNextException on hbase 0.98. The following shows an example of this exception:

SQL>Insert Into BTA1P001
Values (
'ABAA', 0, 0, 'CAAAAAAA', – (0)
68, 'AAAA', 1, 0,
2, 'AA', 2,
11, 3, 'BCAAHAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA',
11,
.0, .2, 'ABAA', .6, .0,
0, 8, 8, 'AA', 68, – (5)
6, 'CBAAAAAA', 6,
1, .1,
'BIAAAAABAAAAAAAA', 1.1, 11, 626,
'BCAAHAAAAAAAAAAA', .0,
'BIAAAAAB', 8, 8, 10,
'ABAA', 10, .00011, .00011 , 11,
'CA', 626,
'AB', 6, 12,
'ABAA', 1968, 468, 69, 1, 9, 9,
'CB', 1, .03,
'ABAA', 11, .01, .06, 6.26,
'BIAAAAAB', 1968, 8,
'CBAAAAAAAAAAAAAA', 68, 2369,
'CBAAAAAAAAAAAAAA', 18, 18, 3,
'AAAAAAAA', 1, 1.1, 1.1,20,
'ABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
)
;

• • • ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-705). Cause:
      java.util.concurrent.ExecutionException: org.apache.hadoop.hbase.DoNotRetryIOException: Failed after retry of OutOfOrderScannerNextException: was there a rpc timeout?
      java.util.concurrent.FutureTask.report(FutureTask.java:122)
      java.util.concurrent.FutureTask.get(FutureTask.java:188)
      org.trafodion.sql.HBaseAccess.HTableClient.fetchRows(HTableClient.java:451)
      . [2014-10-05 21:47:01]
    • ERROR[4082] Object TRAFODION.ARKCASE_ARKT1114.BTA1P001 does not exist or is inaccessible. [2014-10-05 21:47:01]

This is seen on the v1004 build installed on clusters. It may require some new hbase settings so that Trafodion would work more smoothly on hbase 0.98. At this moment, more research is needed to figure out what these settings are.

This is failing our tests quite frequently and quite randomly, so a critical bug report is created to track this problem.