Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6680

Be able to specifie IP for spark-shell(spark driver) blocker for Docker integration

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 1.3.0
    • Fix Version/s: None
    • Component/s: Deploy
    • Labels:
    • Environment:

      Docker.

      Description

      Suppose I have 3 docker containers - spark_master, spark_worker and spark_shell. In docker for public IP of this container there is an alias like "fgsdfg454534". It only visible in this container. When spark use it for communication other containers receive this alias and don't know what to do with it. Thats why I used SPARK_LOCAL_IP for master and worker. But it doesn't work for spark driver(for spark shell - other types of drivers I haven't try). Spark driver sent everyone "fgsdfg454534" alias about itself and then nobody can address it. I've overcome it in https://github.com/epahomov/docker-spark, but it would be better if it would be solved on spark code level.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                epakhomov Egor Pakhomov
              • Votes:
                2 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: