Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21460

Spark dynamic allocation breaks when ListenerBus event queue runs full

    Details

      Description

      When ListenerBus event queue runs full, spark dynamic allocation stops working - Spark fails to shrink number of executors when there are no active jobs (Spark driver "thinks" there are active jobs since it didn't capture when they finished) .

      ps. What's worse it also makes Spark flood YARN RM with reservation requests, so YARN preemption doesn't function properly too (we're on Spark 2.1 / Hadoop 2.6).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Tagar Ruslan Dautkhanov
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: