[SPARK-4403] Elastic allocation(spark.dynamicAllocation.enabled) results in task never being execued.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.1.1
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Egor Pahomov
• Created: 14/Nov/14 15:04
• Updated: 15/Nov/14 13:40
• Resolved: 15/Nov/14 13:40

Description

I execute ipython notebook + pyspark with spark.dynamicAllocation.enabled = true. Task never ends.

Code:

import sys
from random import random
from operator import add
partitions = 10
n = 100000 * partitions

def f(_):
    x = random() * 2 - 1
    y = random() * 2 - 1
    return 1 if x ** 2 + y ** 2 < 1 else 0

count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add)
print "Pi is roughly %f" % (4.0 * count / n)

IPYTHON_ARGS="notebook --profile=ydf --port $IPYTHON_PORT --port-retries=0 --ip='*' --no-browser"
pyspark \
        --verbose \
        --master yarn-client \
        --conf spark.driver.port=$((RANDOM_PORT + 2)) \
        --conf spark.broadcast.port=$((RANDOM_PORT + 3)) \
        --conf spark.replClassServer.port=$((RANDOM_PORT + 4)) \
        --conf spark.blockManager.port=$((RANDOM_PORT + 5)) \
        --conf spark.executor.port=$((RANDOM_PORT + 6)) \
        --conf spark.fileserver.port=$((RANDOM_PORT + 7)) \
        --conf spark.shuffle.service.enabled=true \
        --conf spark.dynamicAllocation.enabled=true \
        --conf spark.dynamicAllocation.minExecutors=1 \
        --conf spark.dynamicAllocation.maxExecutors=10 \
        --conf spark.ui.port=$SPARK_UI_PORT

Issue Links

• is related to: SPARK-3174 Provide elastic scaling within a Spark application