[HDDS-6395] TestReconScmHASnapshot.testScmHASnapshot often fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: Ozone Recon, SCM
• Labels: pull-request-available
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 01/Mar/22 15:53
• Updated: 02/Mar/22 15:01
• Resolved: 02/Mar/22 15:01

Description

I've seen this test fail quite a few times on various PRs. I don't think there is an existing Jira for it:

 Error:  Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 59.752 s <<< FAILURE! - in org.apache.hadoop.ozone.recon.TestReconScmHASnapshot
Error:  testScmHASnapshot  Time elapsed: 59.732 s  <<< FAILURE!
java.lang.AssertionError: expected:<9> but was:<8>
    at org.junit.Assert.fail(Assert.java:89)
    at org.junit.Assert.failNotEquals(Assert.java:835)
    at org.junit.Assert.assertEquals(Assert.java:647)
    at org.junit.Assert.assertEquals(Assert.java:633)
    at org.apache.hadoop.ozone.recon.TestReconScmSnapshot.testSnapshot(TestReconScmSnapshot.java:124)
    at org.apache.hadoop.ozone.recon.TestReconScmHASnapshot.testScmHASnapshot(TestReconScmHASnapshot.java:61)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:750)

The point in the test that it is failing is running this check:

nodeManager = (ReconNodeManager) cluster.getReconServer()
        .getReconStorageContainerManager().getScmNodeManager();
    long keyCountAfter = nodeManager.getNodeDBKeyCount();
    assertEquals(keyCountAfter, keyCountBefore);

`nodeManager.getNodeDBCount` calls:

@VisibleForTesting
  public long getNodeDBKeyCount() throws IOException {
    return nodeDB.getEstimatedKeyCount();
  }

Note it is calling getEstimatedKeyCount, which has this Java doc:

  /**
   * Returns the key count of this Table.  Note the result can be inaccurate.
   * @return Estimated key count of this Table
   * @throws IOException on failure
   */
  long getEstimatedKeyCount() throws IOException;

I don't think this can be used with certainty to check the count, and to get the exact count, we must iterate the table.

Issue Links

• links to: GitHub Pull Request #3149