[HBASE-25598] TestFromClientSide5.testScanMetrics is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.3.4, 2.4.1
• Fix Version/s: 3.0.0-alpha-1, 2.2.7, 2.3.5, 2.4.2
• Assignee: Sun Xin
• Reporter: Sun Xin
• Created: 23/Feb/21 08:17
• Updated: 24/Feb/21 22:57
• Resolved: 24/Feb/21 06:50

Description

In some PRs, I got the following errors in UT results.

[ERROR] Errors: 
[ERROR] org.apache.hadoop.hbase.client.TestFromClientSide5.testScanMetrics[0]
[ERROR]   Run 1: TestFromClientSide5.testScanMetrics:1018 Did not count the result bytes expected:<60> but was:<120>
[ERROR]   Run 2: TestFromClientSide5.testScanMetrics:1036 Did not count the result bytes expected:<60> but was:<180>
[ERROR]   Run 3: TestFromClientSide5.testScanMetrics:951 » MasterRegistryFetch Exception making...
[INFO] 
[ERROR] org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor5.testScanMetrics[1]
[ERROR]   Run 1: TestFromClientSideWithCoprocessor5>TestFromClientSide5.testScanMetrics:1036 Did not count the result bytes expected:<60> but was:<120>
[ERROR]   Run 2: TestFromClientSideWithCoprocessor5>TestFromClientSide5.testScanMetrics:951 » IO
[ERROR]   Run 3: TestFromClientSideWithCoprocessor5>TestFromClientSide5.testScanMetrics:951 » IO
[INFO] 

I read the code further and found that this UT is flaky.

// check byte counters
scan2 = new Scan();
scan2.setScanMetricsEnabled(true);
scan2.setCaching(1);
try (ResultScanner scanner = ht.getScanner(scan2)) {
  int numBytes = 0;
  for (Result result : scanner.next(1)) {
    for (Cell cell : result.listCells()) {
      numBytes += PrivateCellUtil.estimatedSerializedSizeOf(cell);
    }
  }
  scanner.close();
  ScanMetrics scanMetrics = scanner.getScanMetrics();
  assertEquals("Did not count the result bytes", numBytes,
          scanMetrics.countOfBytesInResults.get());
}

In the code above, it is to check scanMetrics.countOfBytesInResults, but just get only ONE row by scanner.next(1) . A total of 3 rows are inserted into the table, and scanner prefetch from server in advance until maxCacheSize is exceeded, see here.

So if scanner prefetch more than one row before closing scanner, the UT fails. we can reproduce this problem steadily by sleeping before scanner.close().

Issue Links

• links to: GitHub Pull Request #2977