Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.9.0
-
None
-
Reviewed
Description
The JobHistoryServer is leaking CLOSE_WAIT connections to DataNodes whenever viewing a huge log file in JobhistoryServer. This happens only when the below is configured.
yarn.log-aggregation.file-formats=IndexedFormat
yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
yarn.log.server.url=http://jobhistory-host:19888/jobhistory/logs
On investigation, I found that the FSDataInputStream is not closed in org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.IndexedFileAggregatedLogsBlock . Since this block is called every time the Jobhistory page displays the logs, CLOSE_WAIT connections to DataNodes keep on increasing in JobHistoryServer.