[SOLR-13330] Improve HDFS tests

• Type: Test
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.1, 9.0
• Component/s: Hadoop Integration, hdfs, Tests
• Assignee: Kevin Risden
• Reporter: Kevin Risden
• Created: 18/Mar/19 19:24
• Updated: 12/May/22 00:26
• Resolved: 19/Mar/19 09:56

Description

Currently there are failures with HDFS tests intermittently. Some of these are due to resource constraints on the Apache Jenkins machines others are due to poor test assumptions.

Related JIRAs:

• SOLR-11010
• SOLR-11381
• SOLR-12040
• SOLR-13297

Changes:

• Consolidate hdfs configuration into HdfsTestUtil
• Ensure socketTimeout long enough for HDFS tests
• Ensure HdfsTestUtil.getClientConfiguration used in tests
• Replace deprecated HDFS calls
• Use try-with-resources to ensure closing of HDFS resources

Issue Links

• is related to: SOLR-13338 HdfsAutoAddReplicasIntegrationTest failures
• relates to: SOLR-11010 OutOfMemoryError in tests when using HDFS BlockCache
• relates to: SOLR-12040 HdfsBasicDistributedZkTest & HdfsBasicDistributedZk2 fail on virtually every jenkins run
• relates to: SOLR-11381 HdfsDirectoryFactory throws NPE on cleanup because file system has been closed
• relates to: SOLR-9961 RestoreCore needs the option to download files in parallel.
• relates to: SOLR-13297 StressHdfsTest and HdfsUnloadDistributedZkTest fail more often after Hadoop 3
• supercedes: SOLR-6008 HDFS tests are using local filesystem because solr.hdfs.home is set to a local filesystem path.
• links to: GitHub Pull Request #609