[HDFS-17278] Detect order dependent flakiness in TestViewfsWithNfs3.java under hadoop-hdfs-nfs module

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: nfs, test
• Labels: pull-request-available

Environment

openjdk version "17.0.9"
Apache Maven 3.9.5

• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Ruby
• Reporter: Ruby
• Created: 06/Dec/23 23:36
• Updated: 28/Jan/24 01:09
• Resolved: 12/Dec/23 07:13

Description

The order dependent flakiness was detected if the test class TestDFSClientCache.java runs before TestRpcProgramNfs3.java.

The error message looks like below:

[ERROR] Failures: 
[ERROR]   TestRpcProgramNfs3.testAccess:279 Incorrect return code expected:<0> but was:<13>
[ERROR]   TestRpcProgramNfs3.testCommit:764 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testCreate:493 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testEncryptedReadWrite:359->createFileUsingNfs:393 Incorrect response:  expected:<null> but was:<org.apache.hadoop.nfs.nfs3.response.WRITE3Response@42752a9b>
[ERROR]   TestRpcProgramNfs3.testFsinfo:714 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testFsstat:696 Incorrect return code: expected:<0> but was:<13>
[ERROR]   TestRpcProgramNfs3.testGetattr:205 Incorrect return code expected:<0> but was:<13>
[ERROR]   TestRpcProgramNfs3.testLookup:249 Incorrect return code expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testMkdir:517 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testPathconf:738 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testRead:341 Incorrect return code: expected:<0> but was:<13>
[ERROR]   TestRpcProgramNfs3.testReaddir:642 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testReaddirplus:666 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testReadlink:297 Incorrect return code: expected:<0> but was:<5>
[ERROR]   TestRpcProgramNfs3.testRemove:570 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testRename:618 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testRmdir:594 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testSetattr:225 Incorrect return code expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testSymlink:546 Incorrect return code: expected:<13> but was:<5>
[ERROR]   TestRpcProgramNfs3.testWrite:468 Incorrect return code: expected:<13> but was:<5>
[INFO] 
[ERROR] Tests run: 25, Failures: 20, Errors: 0, Skipped: 0
[INFO] 
[ERROR] There are test failures. 

The polluter that led to this flakiness was the test method
testGetUserGroupInformationSecure() in TestDFSClientCache.java. There was a line 

UserGroupInformation.setLoginUser(currentUserUgi);

which modifies some shared state and resource, something like pre-setup the config. To fix this issue, I added the cleanup methods in TestDFSClientCache.java to reset the UserGroupInformation to ensure the isolation among each test class.

@AfterClass
public static void cleanup() {
    UserGroupInformation.reset();
}

Including setting

authenticationMethod = null;
conf = null; // set configuration to null
setLoginUser(null); // reset login user to default null

..., and so on. The reset() methods can be referred to hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java.

After the fix, the error was no longer exist and the succeed message was:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
[INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.457 s - in org.apache.hadoop.hdfs.nfs.nfs3.CustomTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------ 

Here is the CustomTest.java file that I used to run these two tests in order, the error can be reproduce by running this CustomTest.java. 

package org.apache.hadoop.hdfs.nfs.nfs3;

import org.junit.runner.RunWith;import org.junit.runners.Suite;



@RunWith(Suite.class)
@Suite.SuiteClasses({
    TestDFSClientCache.class,
    TestRpcProgramNfs3.class
})
public class CustomTest {} 

Issue Links

• links to: GitHub Pull Request #6329