[TEZ-4388] Asynchttpclient can cause stuck TezChild processes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.3, 0.10.2
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 15/Feb/22 13:19
• Updated: 22/Feb/22 09:33
• Resolved: 22/Feb/22 09:33

Description

laszlobodor      96935   0.0  1.6  8398064 553604 s001  S     2:14PM   0:06.21 /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java -Xmx819m -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/Users/laszlobodor/apache/tez/tez-tests/target/tmp/org.apache.tez.test.TestSecureShuffle-withoutssl/yarn-466119913/org.apache.tez.test.TestSecureShuffle-withoutssl-logDir-nm-0_0/application_1644930832286_0002/container_1644930832286_0002_01_000002 -Dtez.root.logger=INFO,CLA -Djava.io.tmpdir=/Users/laszlobodor/apache/tez/tez-tests/target/tmp/org.apache.tez.test.TestSecureShuffle-withoutssl/yarn-466119913/org.apache.tez.test.TestSecureShuffle-withoutssl-localDir-nm-0_0/usercache/laszlobodor/appcache/application_1644930832286_0002/container_1644930832286_0002_01_000002/tmp org.apache.tez.runtime.task.TezChild 192.168.0.52 55541 container_1644930832286_0002_01_000002 application_1644930832286_0002 1

laszlobodor      96789   0.0  1.5  8136944 487980 s001  S     2:13PM   0:06.06 /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java -Xmx819m -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/Users/laszlobodor/apache/tez/tez-tests/target/tmp/org.apache.tez.test.TestSecureShuffle-withoutssl/yarn-466119913/org.apache.tez.test.TestSecureShuffle-withoutssl-logDir-nm-0_0/application_1644930832286_0001/container_1644930832286_0001_01_000002 -Dtez.root.logger=INFO,CLA -Djava.io.tmpdir=/Users/laszlobodor/apache/tez/tez-tests/target/tmp/org.apache.tez.test.TestSecureShuffle-withoutssl/yarn-466119913/org.apache.tez.test.TestSecureShuffle-withoutssl-localDir-nm-0_0/usercache/laszlobodor/appcache/application_1644930832286_0001/container_1644930832286_0001_01_000002/tmp org.apache.tez.runtime.task.TezChild 192.168.0.52 55519 container_1644930832286_0001_01_000002 application_1644930832286_0001 1

laszlobodor      96282   0.0  1.4  8258788 474648 s001  S     2:12PM   0:06.69 /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java -Xmx819m -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/Users/laszlobodor/apache/tez/tez-tests/target/tmp/org.apache.tez.test.TestSecureShuffle-withssl/yarn-466021610/org.apache.tez.test.TestSecureShuffle-withssl-logDir-nm-0_0/application_1644930734009_0002/container_1644930734009_0002_01_000002 -Dtez.root.logger=INFO,CLA -Djava.io.tmpdir=/Users/laszlobodor/apache/tez/tez-tests/target/tmp/org.apache.tez.test.TestSecureShuffle-withssl/yarn-466021610/org.apache.tez.test.TestSecureShuffle-withssl-localDir-nm-0_0/usercache/laszlobodor/appcache/application_1644930734009_0002/container_1644930734009_0002_01_000002/tmp org.apache.tez.runtime.task.TezChild 192.168.0.52 55452 container_1644930734009_0002_01_000002 application_1644930734009_0002 1

laszlobodor      96129   0.0  1.5  8402248 500904 s001  S     2:12PM   0:06.89 /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java -Xmx819m -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/Users/laszlobodor/apache/tez/tez-tests/target/tmp/org.apache.tez.test.TestSecureShuffle-withssl/yarn-466021610/org.apache.tez.test.TestSecureShuffle-withssl-logDir-nm-0_0/application_1644930734009_0001/container_1644930734009_0001_01_000002 -Dtez.root.logger=INFO,CLA -Djava.io.tmpdir=/Users/laszlobodor/apache/tez/tez-tests/target/tmp/org.apache.tez.test.TestSecureShuffle-withssl/yarn-466021610/org.apache.tez.test.TestSecureShuffle-withssl-localDir-nm-0_0/usercache/laszlobodor/appcache/application_1644930734009_0001/container_1644930734009_0001_01_000002/tmp org.apache.tez.runtime.task.TezChild 192.168.0.52 55410 container_1644930734009_0001_01_000002 application_1644930734009_0001 1

after some investigation, I can see that the issue comes in case of async-http client cases, regardless of ssl/non-ssl or positive/negative testcases

Issue Links

• causes: TEZ-4390 Find a way to reuse async http clients when they are closed properly
• links to: GitHub Pull Request #189