[HBASE-14420] Zombie Stomping Session

• Type: Umbrella
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0, 1.3.0, 2.0.0
• Component/s: test
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 12/Sep/15 18:53
• Updated: 23/Jun/22 18:52
• Resolved: 18/Dec/15 18:33

Description

Patch build are now failing most of the time because we are dropping zombies. I confirm we are doing this on non-apache build boxes too.

Left-over zombies consume resources on build boxes (OOME cannot create native threads). Having to do multiple test runs in the hope that we can get a non-zombie-making build or making (arbitrary) rulings that the zombies are 'not related' is a productivity sink. And so on...

This is an umbrella issue for a zombie stomping session that started earlier this week. Will hang sub-issues of this one. Am running builds back-to-back on little cluster to turn out the monsters.

Issue Links

• is related to: HBASE-14600 Make #testWalRollOnLowReplication looser still
• is related to: HBASE-8939 Hanging unit tests
• is related to: HBASE-15168 Zombie stomping branch-1.1 edition
• relates to: HBASE-14378 Get TestAccessController* passing again on branch-1
• relates to: HBASE-14349 pre-commit zombie finder is overly broad
• relates to: HBASE-14506 Reenable TestDistributedLogSplitting#testWorkerAbort
• relates to: HBASE-14486 Disable TestRegionPlacement, a flakey test for an unfinished feature
• relates to: HBASE-14846 IT test suite fails when run standalone: "mvn -Dtest=NoUnitTests package verify"
• relates to: HBASE-15010 Hangers: Umbrellas issue to hang current set of hanging tests from
• relates to: HBASE-15012 Flakies: Umbrella issue to hang current set of flakies from

Sub-Tasks

• TestFastFail* are flakey: Closed
• TestStochasticBalancerJmxMetrics.testJmxMetrics_PerTableMode:183 NullPointer: Closed
• Upgrade our surefire-plugin from 2.18 to 2.18.1: Closed
• TestHttpServerLifecycle#testStartedServerIsAlive times out: Closed
• Set down the client executor core thread count from 256 in tests: Closed
• thrift tests don't have test-specific hbase-site.xml so 'BindException: Address already in use' because info port is not turned off: Closed
• Spark tests failing: bind exception when putting up info server: Closed
• TestFailedAppendAndSync fail: Closed
• TestHCM and TestRegionServerNoMaster fixes: Closed
• Follow-on from HBASE-14421, just disable TestFastFail* until someone digs in and fixes it: Closed
• TestHRegion#testFlushCacheWhileScanning goes zombie: Closed
• Disable TestDistributedLogSplitting#testWorkerAbort Its flakey with tenuous chance of success: Closed
• TestBucketCache runs obnoxious 1k threads in a unit test: Closed
• Purge TestFavoredNodeAssignmentHelper, a test for an abandoned feature that can hang: Closed
• Remove TestVisibilityLabelsWithDistributedLogReplay, a test for an unsupported feature: Closed
• Have findHangingTests.py dump more info: Closed
• branch-1 test tweeks; disable assert explicit region lands post-restart and up a few handlers: Closed
• Disable zombie TestReplicationShell: Closed
• Disable zombie TestHFileOutputFormat2: Closed
• Tuneup hanging test TestMobCompactor and TestMobSweeper: Closed
• Disable hanging test TestStochasticLoadBalancer2: Closed
• Disable hanging test TestNamespaceAuditor: Closed
• Split TestHBaseFsck in order to help with hanging tests: Closed
• Purge TestProcessBasedCluster; it does nothing and then fails: Closed
• TestImportExport#testImport94Table can't find its src data file: Closed
• Clean up TestSnapshotCloneIndependence: Closed
• Looking for the surefire-killer; builds being killed...ExecutionException: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?: Closed
• Add more info on zombies to test-patch.sh: Closed
• TestCellACLs failing... on1.2 builds: Closed
• Purge TestZkLess* tests from branch-1: Closed
• Loosen TestChoreService assert AND have TestDataBlockEncoders do less work (and add timeouts): Closed
• Disable flakey TestMultiParallel#testActiveThreadsCount: Closed
• Move TestCellACLs from medium to large category: Closed
• Move TestAssignmentManager from medium to large category: Closed
• Set category timeouts on TestScanner and TestNamespaceAuditor: Closed
• TestZKProcedureControllers.testZKCoordinatorControllerWithSingleMemberCohort is a flakey: Closed
• Add category-based timeouts to MR tests: Closed
• Make TestHCM and TestMetaWithReplicas large tests rather than mediums: Closed
• Vet categorization of tests so they for sure go into the right small/medium/large buckets: Closed
• Run less client threads in tests: Closed
• Improve zombie detector; be more discerning: Closed
• TestProcedureAdmin hangs: Closed
• Cleanup TestAtomicOperation, TestImportExport, and TestMetaWithReplicas: Closed
• NPE reporting server load causes regionserver abort; causes TestAcidGuarantee to fail: Closed
• hbase-it tests failing with OOME; permgen: Closed
• TestSplitTransactionOnCluster#testFailedSplit flakey: Closed
• TestSplitTransactionOnCluster.testSSHCleanupDaugtherRegionsOfAbortedSplit is flakey: Closed
• TestRowCounter flakey especially on branch-1: Closed
• NPE testing for RIT: Closed
• Hanging test : org.apache.hadoop.hbase.mapreduce.TestImportExport: Closed
• OOME: cannot create native thread is back: Closed