[SOLR-14179] BasicDistributedZkTest reproducible jenkins failure due to exceeding file handle limit

• Type: Test
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Chris M. Hostetter
• Created: 08/Jan/20 18:29
• Updated: 21/Jan/20 16:58

Description

jenkins recently found a reproducing seed for BasicDistributedZkTest - the exact failures can vary in each run, but the common root issue appears to be that with this seed it's very easy to hit "Too many open files"...

   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.method=test -Dtests.seed=DE4EDB45974CB32A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=zh-TW -Dtests.timezone=Canada/Pacific -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   41.8s | BasicDistributedZkTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:44105/collection1: /home/hossman/lucene/alt_dev/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_DE4EDB45974CB32A-001/index-SimpleFSDirectory-003/_x_MockRandom_0.tix: Too many open files
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DE4EDB45974CB32A:561AE49F39B0DED2]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:665)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZkTest.testUpdateProcessorsRunOnlyOnce(BasicDistributedZkTest.java:852)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:438)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
   [junit4]    > 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/hossman/lucene/alt_dev/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest_DE4EDB45974CB32A-001
   [junit4]   2> Jan 08, 2020 11:27:32 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84): {regex_dup_A_s=PostingsFormat(name=LuceneVarGapFixedInterval), regex_dup_B_s=Lucene84, SubjectTerms_mfacet=PostingsFormat(name=MockRandom), multiDefault=Lucene84, _root_=PostingsFormat(name=MockRandom), genre_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), author_t=PostingsFormat(name=MockRandom), series_t=PostingsFormat(name=MockRandom), rnd_b=PostingsFormat(name=LuceneVarGapDocFreqInterval), oddField_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_t=Lucene84, cat=Lucene84, foo_b=PostingsFormat(name=LuceneVarGapFixedInterval), name=PostingsFormat(name=LuceneVarGapDocFreqInterval), inStock=PostingsFormat(name=LuceneVarGapFixedInterval), id=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=PostingsFormat(name=LuceneVarGapFixedInterval), t_sortable=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{other_tl1=DocValuesFormat(name=Lucene80), range_facet_l_dv=DocValuesFormat(name=Direct), n_l1=DocValuesFormat(name=Asserting), intDefault=DocValuesFormat(name=Lucene80), n_td1=DocValuesFormat(name=Lucene80), n_d1=DocValuesFormat(name=Asserting), range_facet_l=DocValuesFormat(name=Asserting), n_f1=DocValuesFormat(name=Lucene80), n_tl1=DocValuesFormat(name=Lucene80), n_tf1=DocValuesFormat(name=Asserting), price=DocValuesFormat(name=Lucene80), sequence_i=DocValuesFormat(name=Direct), intDvoDefault=DocValuesFormat(name=Direct), t_sortable=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Asserting), foo_i=DocValuesFormat(name=Lucene80), val_i=DocValuesFormat(name=Direct), n_dt1=DocValuesFormat(name=Lucene80), a_i1=DocValuesFormat(name=Asserting), n_ti1=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Lucene80), n_tdt1=DocValuesFormat(name=Lucene80), id_i1=DocValuesFormat(name=Lucene80), foo_d=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Asserting), foo_f=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=1214, maxMBSortInHeap=7.926887523783416, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4ead536d), locale=zh-TW, timezone=Canada/Pacific
   [junit4]   2> NOTE: Linux 5.0.0-32-generic amd64/AdoptOpenJDK 11.0.4 (64-bit)/cpus=8,threads=2,free=135495624,total=522190848
   [junit4]   2> NOTE: All tests run in this JVM: [BasicDistributedZkTest]

Issue Links

• is related to: SOLR-14180 RequestHandlerBase.java:227 assertion failing: not all "Tragic" exceptions have 500 status, assert masks real failure