Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
2.3.0, 2.4.0
-
None
-
None
Description
Typically for long running applications with large number of tasks it is common to disable UI to minimize overhead at driver.
Earlier, with spark ui disabled, only stage/job information was kept as part of JobProgressListener.
As part of history server scalability fixes, particularly SPARK-20643, inspite of disabling UI - task information continues to be maintained in memory.
In our long running tests against spark thrift server, this eventually results in OOM.