[HBASE-21216] TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 20/Sep/18 22:49
• Updated: 21/Jul/21 14:25
• Resolved: 25/Oct/18 20:33

Description

From https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/ :

java.lang.AssertionError: Archived hfiles [] and table hfiles [9ca09392705f425f9c916beedc10d63c] is missing snapshot file:6739a09747e54189a4112a6d8f37e894
	at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)

The file appeared in archive dir before hfile cleaners were run:

2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771): |-archive/
2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771): |----data/
2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771): |-------default/
2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771): |----------test/
2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771): |-------------1237d57b63a7bdf067a930441a02514a/
2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771): |----------------recovered.edits/
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774): |-------------------4.seqid
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771): |-------------29e1700e09b51223ad2f5811105a4d51/
2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771): |----------------fam/
2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774): |-------------------2c66a18f6c1a4074b84ffbb3245268c4
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774): |-------------------45bb396c6a5e49629e45a4d56f1e9b14
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774): |-------------------6739a09747e54189a4112a6d8f37e894

However, the archive dir became empty after hfile cleaners were run:

2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771): |-archive/
2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771): |-corrupt/

Leading to the assertion failure.

This test is one of the top flaky tests.