[YARN-9336] JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0
• Fix Version/s: 3.3.0
• Component/s: log-aggregation
• Hadoop Flags: Reviewed
• Assignee: Tarun Parimi
• Reporter: Tarun Parimi
• Created: 28/Feb/19 12:11
• Updated: 16/Apr/19 09:17
• Resolved: 16/Apr/19 09:17

Description

The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever viewing a huge log file in JobhistoryServer. This happens only when the below is configured.

yarn.log-aggregation.file-formats=IndexedFormat
yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs

On investigation, I found that the FSDataInputStream is not closed in org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock . Since this block is called every time the Jobhistory page displays the logs, CLOSE_WAIT connections to DataNodes keep on increasing in JobHistoryServer.