Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11638

Run Spark on Mesos with bridge networking

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Later
    • Affects Version/s: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0
    • Fix Version/s: None
    • Component/s: Mesos, Spark Core
    • Labels:
      None
    • Flags:
      Patch

      Description

      Summary

      Provides spark.driver.advertisedPort, spark.fileserver.advertisedPort, spark.broadcast.advertisedPort and spark.replClassServer.advertisedPort settings to enable running Spark in Mesos on Docker with Bridge networking. Provides patches for Akka Remote to enable Spark driver advertisement using alternative host and port.

      With these settings, it is possible to run Spark Master in a Docker container and have the executors running on Mesos talk back correctly to such Master.

      The problem is discussed on the Mesos mailing list here: https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bFOtj5LJZFRH5u7ix-ghppFqKnVg9mkKctjg@mail.gmail.com%3E

      Running Spark on Mesos - LIBPROCESS_ADVERTISE_IP opens the door

      In order for the framework to receive orders in the bridged container, Mesos in the container has to register for offers using the IP address of the Agent. Offers are sent by Mesos Master to the Docker container running on a different host, an Agent. Normally, prior to Mesos 0.24.0, libprocess would advertise itself using the IP address of the container, something like 172.x.x.x. Obviously, Mesos Master can't reach that address, it's a different host, it's a different machine. Mesos 0.24.0 introduced two new properties for libprocess - LIBPROCESS_ADVERTISE_IP and LIBPROCESS_ADVERTISE_PORT. This allows the container to use the Agent's address to register for offers. This was provided mainly for running Mesos in Docker on Mesos.

      Spark - how does the above relate and what is being addressed here?

      Similar to Mesos, out of the box, Spark does not allow to advertise its services on ports different than bind ports. Consider following scenario:

      Spark is running inside a Docker container on Mesos, it's a bridge networking mode. Assuming a port 6666 for the spark.driver.port, 6677 for the spark.fileserver.port, 6688 for the spark.broadcast.port and 23456 for the spark.replClassServer.port. If such task is posted to Marathon, Mesos will give 4 ports in range 31000-32000 mapping to the container ports. Starting the executors from such container results in executors not being able to communicate back to the Spark Master.

      This happens because of 2 things:

      Spark driver is effectively an akka-remote system with akka.tcp transport. akka-remote prior to version 2.4 can't advertise a port different to what it bound to. The settings discussed are here: https://github.com/akka/akka/blob/f8c1671903923837f22d0726a955e0893add5e9f/akka-remote/src/main/resources/reference.conf#L345-L376. These do not exist in Akka 2.3.x. Spark driver will always advertise port 6666 as this is the one akka-remote is bound to.
      Any URIs the executors contact the Spark Master on, are prepared by Spark Master and handed over to executors. These always contain the port number used by the Master to find the service on. The services are:

      • spark.broadcast.port
      • spark.fileserver.port
      • spark.replClassServer.port

      all above ports are by default 0 (random assignment) but can be specified using Spark configuration ( -Dspark...port ). However, they are limited in the same way as the spark.driver.port; in the above example, an executor should not contact the file server on port 6677 but rather on the respective 31xxx assigned by Mesos.

      Spark currently does not allow any of that.

      Taking on the problem, step 1: Spark Driver

      As mentioned above, Spark Driver is based on akka-remote. In order to take on the problem, the akka.remote.net.tcp.bind-hostname and akka.remote.net.tcp.bind-port settings are a must. Spark does not compile with Akka 2.4.x yet.

      What we want is the back port of mentioned akka-remote settings to 2.3.x versions. These patches are attached to this ticket - 2.3.4.patch and 2.3.11.patch files provide patches for respective akka versions. These add mentioned settings and ensure they work as documented for Akka 2.4. In other words, these are future compatible.

      A part of that patch also exists in the patch for Spark, in the org.apache.spark.util.AkkaUtils class. This is where Spark is creating the driver and compiling the Akka configuration. That part of the patch tells Akka to use bind-hostname instead of hostname, if spark.driver.advertisedHost is given and use bind-port instead of port, if spark.driver.advertisedPort is given. In such cases, hostname and port are set to the advertised values, respectively.

      Worth mentioning: if spark.driver.advertisedHost or spark.driver.advertisedPort isn't given, patched Spark reverts to using the settings as they would be in case of non-patched akka-remote; exactly for that purpose: if there is no patched akka-remote in use. Even if it is in use, akka-remote will correctly handle undefined bind-hostname and bind-port, as specified by Akka 2.4.x.

      Akka versions in Spark (attached patches only)
      • Akka 2.3.4
      • Spark 1.4.0
      • Spark 1.4.1
      • Akka 2.3.11
      • spark 1.5.0
      • spark 1.5.1
      • spark-1.6.0-SNAPSHOT

      Taking on the problem, step 2: Spark services

      The fortunate thing is that every other Spark service is running over HTTP, using an org.apache.spark.HttpServer class. This is where the second part of the Spark patch comes into play. All other changes in the patch files provide alternative advertised... ports for each of the following services:

      • spark.broadcast.port -> spark.broadcast.advertisedPort
      • spark.fileserver.port -> spark.fileserver.advertisedPort
      • spark.replClassServer.port -> spark.replClassServer.advertisedPort

      What we are telling Spark here, is the following: if there is an alternative advertisedPort setting given to this server instance, use that setting for advertising the port.

      Patches

      These patches are cleared by the Technicolor IP&L Team to be contributed back under the Apache 2.0 License to Spark.
      All patches for versions from 1.4.0 to 1.5.2 can be applied directly to the respective tag from Spark git repository. The 1.6.0-master.patch applies to git sha 18350a57004eb87cafa9504ff73affab4b818e06.

      Building Akka

      To build the required akka version:

      AKKA_VERSION=2.3.4
      git clone https://github.com/akka/akka.git .
      git fetch origin
      git checkout v${AKKA_VERSION}
      git apply ...2.3.4.patch
      sbt package -Dakka.scaladoc.diagrams=false
      

      What is not supplied

      At the moment of contribution, we do not supply any unit tests. We would like to contribute those but we may require some assistance.

      =====

      Happy to answer any questions and looking forward to any guidance which would lead to have these included in the master Spark version.

      1. 1.4.0.patch
        14 kB
        Radoslaw Gruchalski
      2. 1.4.1.patch
        14 kB
        Radoslaw Gruchalski
      3. 1.5.0.patch
        14 kB
        Radoslaw Gruchalski
      4. 1.5.1.patch
        14 kB
        Radoslaw Gruchalski
      5. 1.5.2.patch
        14 kB
        Radoslaw Gruchalski
      6. 1.6.0.patch
        15 kB
        Radoslaw Gruchalski
      7. 2.3.11.patch
        7 kB
        Radoslaw Gruchalski
      8. 2.3.4.patch
        7 kB
        Radoslaw Gruchalski

        Issue Links

          Activity

          Hide
          radekg Radoslaw Gruchalski added a comment -

          The 2.3.4.patch and 2.3.11.patch are akka-remote patches. The rest of the files are Apache Spark patches for the respective versions.

          Show
          radekg Radoslaw Gruchalski added a comment - The 2.3.4.patch and 2.3.11.patch are akka-remote patches. The rest of the files are Apache Spark patches for the respective versions.
          Hide
          radekg Radoslaw Gruchalski added a comment -

          Working on providing full description. I will update the JIRA ticket shortly giving full context and what the attached patches bring to Spark.

          Show
          radekg Radoslaw Gruchalski added a comment - Working on providing full description. I will update the JIRA ticket shortly giving full context and what the attached patches bring to Spark.
          Hide
          srowen Sean Owen added a comment -

          We don't use patches in Spark. Why not just get the text ready before opening a JIRA?
          I'm not sure what this is but I think you might want to discuss on user@ first.

          Show
          srowen Sean Owen added a comment - We don't use patches in Spark. Why not just get the text ready before opening a JIRA? I'm not sure what this is but I think you might want to discuss on user@ first.
          Hide
          radekg Radoslaw Gruchalski added a comment -

          Provided full description of the patches are for. About to notify the user list and open a PR to master for 1.6.0 (however, knowing that it will not go directly in because of the Akka dependency, I started here).

          Show
          radekg Radoslaw Gruchalski added a comment - Provided full description of the patches are for. About to notify the user list and open a PR to master for 1.6.0 (however, knowing that it will not go directly in because of the Akka dependency, I started here).
          Hide
          apachespark Apache Spark added a comment -

          User 'radekg' has created a pull request for this issue:
          https://github.com/apache/spark/pull/9608

          Show
          apachespark Apache Spark added a comment - User 'radekg' has created a pull request for this issue: https://github.com/apache/spark/pull/9608
          Hide
          ChrisHeller Chris Heller added a comment -

          Can't wait to try this out. +1

          Show
          ChrisHeller Chris Heller added a comment - Can't wait to try this out. +1
          Hide
          radekg Radoslaw Gruchalski added a comment - - edited

          I suppose I should give some baseline for how this can be tested. Here is the method:

          In the container:

          The container has to be aware of the ip address of an agent it is running on - pass it as an env variable or prepare correct /etc/hosts file and add it as a docker volume to the container; for the purpose of the example below, make sure it is called AGENT_ADVERTISE_IP. Follow these guidelines - in the container:

          export AGENT_ADVERTISE_IP=${AGENT_ADVERTISE_IP-127.0.0.1}
          export LIBPROCESS_ADVERTISE_IP=$AGENT_ADVERTISE_IP
          export LIBPROCESS_ADVERTISE_PORT=$PORT_9050 # choose your own port
          export SPARK_LOCAL_IP=$(ip -o -4 addr list eth0 | perl -n -e 'if (m{inet\s([\d\.]+)\/\d+\s}xms) { print $1 }')
          export SPARK_PUBLIC_DNS=$HOST
          export SPARK_LOCAL_HOSTNAME=$HOST
          
          # this is required so Spark can bind to the $SPARK_PUBLIC_DNS
          my_own_ip=$(cat /etc/hosts | grep $HOSTNAME | awk '{print $1}')
          echo "$my_own_ip    $SPARK_PUBLIC_DNS" >> /etc/hosts
          
          export SPARK_EXECUTOR_URI=... the location the tgz, can be stock download from Spark website
          
          # Make sure the spark-core, spark-repl and akka-remote jars are in the sandbox (add them with uris).
          # This is important now, these 3 jars have to placed on the class path first, before any other jar goes in, Data Fellas Spark notebook provides CLASSPATH_OVERRIDES env variable especially for that:
          
          export CLASSPATH_OVERRIDES=$MESOS_SANDBOX/akka-remote_2.10-x.x.x.jar:$MESOS_SANDBOX/spark-core_2.10-x.x.x.jar:$MESOS_SANDBOX/spark-repl_2.10-x.x.x.jar
          
          # Spark notebook settings:
          cd /opt/docker && ./bin/spark-notebook -Dapplication.context=$NOTEBOOK_APPLICATION_CONTEXT \
              -Dmanager.notebooks.custom.args.0="-Dspark.driver.host=$SPARK_PUBLIC_DNS" \
              -Dmanager.notebooks.custom.args.1="-Dspark.driver.port=6666" \
              -Dmanager.notebooks.custom.args.2="-Dspark.driver.advertisedPort=$PORT_6666" \
              -Dmanager.notebooks.custom.args.3="-Dspark.replClassServer.port=23456" \
              -Dmanager.notebooks.custom.args.4="-Dspark.replClassServer.advertisedPort=$PORT_23456" \
              -Dmanager.notebooks.custom.sparkConf.spark.executor.uri=${SPARK_EXECUTOR_URI} \
              -Dmanager.notebooks.custom.sparkConf.spark.master=mesos://zk://... your zookeepers .../mesos \
              -Dmanager.notebooks.custom.sparkConf.spark.mesos.executor.home=/opt/spark/dist \
              -Dmanager.notebooks.custom.sparkConf.spark.fileserver.port=6677 \
              -Dmanager.notebooks.custom.sparkConf.spark.fileserver.advertisedPort=$PORT_6677 \
              -Dmanager.notebooks.custom.sparkConf.spark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory \
              -Dmanager.notebooks.custom.sparkConf.spark.broadcast.port=6688 \
              -Dmanager.notebooks.custom.sparkConf.spark.broadcast.advertisedPort=$PORT_6688
          

          Zeppelin notebook is soon to support a similar variable - ZEPPELIN_CLASSPATH_OVERRIDES: https://github.com/apache/incubator-zeppelin/pull/412.

          Show
          radekg Radoslaw Gruchalski added a comment - - edited I suppose I should give some baseline for how this can be tested. Here is the method: build spark as described in the official guide: http://spark.apache.org/docs/latest/building-spark.html - there is no need to have patched akka during the build , copy spark-core_2.10-x.x.x.jar and spark-repl_2.10-x.x.x.jar ; the executors do not need the custom build, only the master build corresponding akka version, copy the akka-remote_2.10-x.x.x.jar somewhere set it all up (using Data Fellas Spark notebook as an example: https://github.com/andypetrella/spark-notebook ) In the container: The container has to be aware of the ip address of an agent it is running on - pass it as an env variable or prepare correct /etc/hosts file and add it as a docker volume to the container; for the purpose of the example below, make sure it is called AGENT_ADVERTISE_IP. Follow these guidelines - in the container: export AGENT_ADVERTISE_IP=${AGENT_ADVERTISE_IP-127.0.0.1} export LIBPROCESS_ADVERTISE_IP=$AGENT_ADVERTISE_IP export LIBPROCESS_ADVERTISE_PORT=$PORT_9050 # choose your own port export SPARK_LOCAL_IP=$(ip -o -4 addr list eth0 | perl -n -e 'if (m{inet\s([\d\.]+)\/\d+\s}xms) { print $1 }') export SPARK_PUBLIC_DNS=$HOST export SPARK_LOCAL_HOSTNAME=$HOST # this is required so Spark can bind to the $SPARK_PUBLIC_DNS my_own_ip=$(cat /etc/hosts | grep $HOSTNAME | awk '{print $1}') echo "$my_own_ip $SPARK_PUBLIC_DNS" >> /etc/hosts export SPARK_EXECUTOR_URI=... the location the tgz, can be stock download from Spark website # Make sure the spark-core, spark-repl and akka-remote jars are in the sandbox (add them with uris). # This is important now, these 3 jars have to placed on the class path first, before any other jar goes in, Data Fellas Spark notebook provides CLASSPATH_OVERRIDES env variable especially for that: export CLASSPATH_OVERRIDES=$MESOS_SANDBOX/akka-remote_2.10-x.x.x.jar:$MESOS_SANDBOX/spark-core_2.10-x.x.x.jar:$MESOS_SANDBOX/spark-repl_2.10-x.x.x.jar # Spark notebook settings: cd /opt/docker && ./bin/spark-notebook -Dapplication.context=$NOTEBOOK_APPLICATION_CONTEXT \ -Dmanager.notebooks.custom.args.0="-Dspark.driver.host=$SPARK_PUBLIC_DNS" \ -Dmanager.notebooks.custom.args.1="-Dspark.driver.port=6666" \ -Dmanager.notebooks.custom.args.2="-Dspark.driver.advertisedPort=$PORT_6666" \ -Dmanager.notebooks.custom.args.3="-Dspark.replClassServer.port=23456" \ -Dmanager.notebooks.custom.args.4="-Dspark.replClassServer.advertisedPort=$PORT_23456" \ -Dmanager.notebooks.custom.sparkConf.spark.executor.uri=${SPARK_EXECUTOR_URI} \ -Dmanager.notebooks.custom.sparkConf.spark.master=mesos://zk://... your zookeepers .../mesos \ -Dmanager.notebooks.custom.sparkConf.spark.mesos.executor.home=/opt/spark/dist \ -Dmanager.notebooks.custom.sparkConf.spark.fileserver.port=6677 \ -Dmanager.notebooks.custom.sparkConf.spark.fileserver.advertisedPort=$PORT_6677 \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.port=6688 \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.advertisedPort=$PORT_6688 Zeppelin notebook is soon to support a similar variable - ZEPPELIN_CLASSPATH_OVERRIDES : https://github.com/apache/incubator-zeppelin/pull/412 .
          Hide
          skonto Stavros Kontopoulos added a comment - - edited

          Does the test method assume one container (the one having spark-notebook) running everything in it (mesos master, executors), started with marathon?

          Show
          skonto Stavros Kontopoulos added a comment - - edited Does the test method assume one container (the one having spark-notebook) running everything in it (mesos master, executors), started with marathon?
          Hide
          radekg Radoslaw Gruchalski added a comment - - edited

          Stavros Kontopoulos: No, this assumes spark master running in a docker container and executors running outside of the container, scheduled with Mesos. spark-notebook is just an example usage. For Spark without notebook, the settings would be along the lines:

              -Dspark.driver.host=$SPARK_PUBLIC_DNS \
              -Dspark.driver.port=6666 \
              -Dspark.driver.advertisedPort=$PORT_6666 \
              -Dspark.replClassServer.port=23456 \
              -Dspark.replClassServer.advertisedPort=$PORT_23456 \
              -Dspark.executor.uri=${SPARK_EXECUTOR_URI} \
              -Dspark.master=mesos://zk://... your zookeepers .../mesos \
              -Dspark.mesos.executor.home=/opt/spark/dist \
              -Dspark.fileserver.port=6677 \
              -Dspark.fileserver.advertisedPort=$PORT_6677 \
              -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory \
              -Dspark.broadcast.port=6688 \
              -Dspark.broadcast.advertisedPort=$PORT_6688
          

          Because the master is running in a container, it's safe to assume the ports. Executors will run outside of the container, harnessing the full capacity of your Mesos cluster.

          Regarding the Marathon part, no Marathon required, just Docker on Mesos. Again, using Marathon to simplify things.

          Show
          radekg Radoslaw Gruchalski added a comment - - edited Stavros Kontopoulos : No, this assumes spark master running in a docker container and executors running outside of the container, scheduled with Mesos. spark-notebook is just an example usage. For Spark without notebook, the settings would be along the lines: -Dspark.driver.host=$SPARK_PUBLIC_DNS \ -Dspark.driver.port=6666 \ -Dspark.driver.advertisedPort=$PORT_6666 \ -Dspark.replClassServer.port=23456 \ -Dspark.replClassServer.advertisedPort=$PORT_23456 \ -Dspark.executor.uri=${SPARK_EXECUTOR_URI} \ -Dspark.master=mesos://zk://... your zookeepers .../mesos \ -Dspark.mesos.executor.home=/opt/spark/dist \ -Dspark.fileserver.port=6677 \ -Dspark.fileserver.advertisedPort=$PORT_6677 \ -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory \ -Dspark.broadcast.port=6688 \ -Dspark.broadcast.advertisedPort=$PORT_6688 Because the master is running in a container, it's safe to assume the ports. Executors will run outside of the container, harnessing the full capacity of your Mesos cluster. Regarding the Marathon part, no Marathon required, just Docker on Mesos. Again, using Marathon to simplify things.
          Hide
          skonto Stavros Kontopoulos added a comment -

          Ok it makes sense....so executors could run in containers too as a predefined setup or use spark.mesos.executor.docker.image at the driver side so they are launched simply on mesos, each choice affects the test method in the same way? I am trying to cover test cases...

          Show
          skonto Stavros Kontopoulos added a comment - Ok it makes sense....so executors could run in containers too as a predefined setup or use spark.mesos.executor.docker.image at the driver side so they are launched simply on mesos, each choice affects the test method in the same way? I am trying to cover test cases...
          Hide
          radekg Radoslaw Gruchalski added a comment -

          Indeed, executors can run in docker containers as well. There's a caveat though. The executor container must be able to resolve the agent. For this, some way of injecting IP addresses into the container is required. This can be done but may not be trivial. I am looking at providing minimum viable example to be added to this ticket.

          Show
          radekg Radoslaw Gruchalski added a comment - Indeed, executors can run in docker containers as well. There's a caveat though. The executor container must be able to resolve the agent. For this, some way of injecting IP addresses into the container is required. This can be done but may not be trivial. I am looking at providing minimum viable example to be added to this ticket.
          Hide
          skonto Stavros Kontopoulos added a comment -

          Ok so the question from my side is that if ips (to setup advertisement ips) are injected somehow at the start-up then applying the fix makes this common scenario work? I understand completely the intention of something limited as a proof of concept, what i try to find out is the extend of use cases covered and as a result test scenarios and/or future work if any....

          Show
          skonto Stavros Kontopoulos added a comment - Ok so the question from my side is that if ips (to setup advertisement ips) are injected somehow at the start-up then applying the fix makes this common scenario work? I understand completely the intention of something limited as a proof of concept, what i try to find out is the extend of use cases covered and as a result test scenarios and/or future work if any....
          Hide
          radekg Radoslaw Gruchalski added a comment - - edited

          Exactly, the only "problematic" thing is how to get the ips into the container. When submitting a task to mesos/marathon, you submit the task to the mesos master, so at the time of submission you don't know where the task is going to run. When submitting a task to Marathon, this is what we do at Virdata (pseudo code):

          • have a file called /etc/agent.sh, this file contains something like:
          #!/bin/bash
          AGENT_PRIVATE_IP=$(ifconfig ...)
          

          When we submit the task to Marathon (we use Marathon), we do:

          {
           ...
            "container": {
              "type": "docker",
              "docker": ...
            },
            "volumes": {
              "containerPath": "/etc/agent.sh",
              "hostPath": "/etc/agent.sh",
              "mode": "RO"
            }
          }
          

          In the container, source /etc/agent.sh.

          The /etc/agent.sh file needs to exist on every agent node.

          In case of the executors having to know the addresses of every agent (so they can resolve back to the master), the simplest way would be to generate a file like this:

          # /etc/mesos-hosts
          10.100.1.10    mesos-agent1
          10.100.1.11    mesos-agent2
          ...
          

          And store it on hdfs. As long as the executor container can read from hdfs, you'll be sorted. Again, I think an MVE would be much clearer than this write up. Happy to provide such code but it may be difficult today.

          Show
          radekg Radoslaw Gruchalski added a comment - - edited Exactly, the only "problematic" thing is how to get the ips into the container. When submitting a task to mesos/marathon, you submit the task to the mesos master, so at the time of submission you don't know where the task is going to run. When submitting a task to Marathon, this is what we do at Virdata (pseudo code): have a file called /etc/agent.sh, this file contains something like: #!/bin/bash AGENT_PRIVATE_IP=$(ifconfig ...) When we submit the task to Marathon (we use Marathon), we do: { ... "container": { "type": "docker", "docker": ... }, "volumes": { "containerPath": "/etc/agent.sh", "hostPath": "/etc/agent.sh", "mode": "RO" } } In the container, source /etc/agent.sh . The /etc/agent.sh file needs to exist on every agent node. In case of the executors having to know the addresses of every agent (so they can resolve back to the master), the simplest way would be to generate a file like this: # /etc/mesos-hosts 10.100.1.10 mesos-agent1 10.100.1.11 mesos-agent2 ... And store it on hdfs. As long as the executor container can read from hdfs, you'll be sorted. Again, I think an MVE would be much clearer than this write up. Happy to provide such code but it may be difficult today.
          Hide
          skonto Stavros Kontopoulos added a comment - - edited

          Ok i tried to veirfy that but no result:

          On my host machine:

          echo 'docker,mesos' > /etc/mesos-slave/containerizers
          echo '5mins' > /etc/mesos-slave/executor_registration_timeout
          sudo service mesos-slave restart

          curl -X POST -H "Content-Type: application/json" http://localhost:8080/v2/apps -d@Docker.json

          #copy patched jars
          export CONT="$(docker ps | grep mesos | awk '

          {print $15; }

          ')"
          docker cp spark-core_2.10-1.5.1.jar $CONT:/mnt/mesos/sandbox
          docker cp spark-repl_2.10-1.5.1.jar $CONT:/mnt/mesos/sandbox
          docker cp akka-remote_2.10-2.3.4.jar $CONT:/mnt/mesos/sandbox

          Docker.json:

          {
          "container": {
          "type": "DOCKER",
          "name" : "test",
          "docker": {
          "image": "andypetrella/spark-notebook:0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0",
          "network": "BRIDGE",
          "portMappings": [

          { "containerPort": 9000, "hostPort": 0, "protocol": "tcp" }

          ]
          },

          "volumes": [

          { "containerPath": "/var/spark/spark-1.5.1-bin-custom-spark.tgz", "hostPath": "/xxxx/spark-1.5.1-bin-custom-spark.tgz", "mode": "RO" }

          ]

          },
          "id": "ubuntu2",
          "instances": 1,
          "cpus": 2,
          "mem": 1024,
          "cmd" : "while sleep 1000; do date -u +%T; done"
          }

          On my host Ubuntu machine i run mesos-master, mesos-slave, and zookeeper. zookeeper listens on *:2181.
          My hostname points to 172.17.0.1 in /etc/hosts.

          I have used this image "andypetrella/spark-notebook:0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0

          The docker0 interface has ip 172.17.0.1

          The containers private ip is 172.17.0.2

          You can ping to each other...

          Next i run inside the container:

          sudo apt-get --only-upgrade install mesos #upgrade to 0.25.0

          export AGENT_ADVERTISE_IP=172.17.0.1
          export LIBPROCESS_ADVERTISE_IP=$AGENT_ADVERTISE_IP
          export LIBPROCESS_ADVERTISE_PORT=9050 # choose your own port
          export SPARK_LOCAL_IP=$(ip -o -4 addr list eth0 | perl -n -e 'if (m

          {inet\s([\d\.]+)\/\d+\s}

          xms)

          { print $1 }

          ') #172.17.0.1
          export SPARK_PUBLIC_DNS=$HOST
          export SPARK_LOCAL_HOSTNAME=$HOST //thinkpad->172.17.0.2

          1. this is required so Spark can bind to the $SPARK_PUBLIC_DNS
            my_own_ip=$(cat /etc/hosts | grep $HOSTNAME | awk ' {print $1}

            ') #172.17.0.1

          echo "$my_own_ip $SPARK_PUBLIC_DNS" >> /etc/hosts

          export SPARK_EXECUTOR_URI=/var/spark/spark-1.5.1-bin-custom-spark.tgz

          1. Make sure the spark-core, spark-repl and akka-remote jars are in the sandbox (add them with uris).
          2. This is important now, these 3 jars have to placed on the class path first, before any other jar goes in, Data Fellas Spark notebook provides CLASSPATH_OVERRIDES env variable especially for that:

          export CLASSPATH_OVERRIDES=$MESOS_SANDBOX/akka-remote_2.10-2.3.4.jar:$MESOS_SANDBOX/spark-core_2.10-1.5.1.jar:$MESOS_SANDBOX/spark-repl_2.10-1.5.1.jar

          export NOTEBOOK_APPLICATION_CONTEXT=/

          1. Spark notebook settings:
            cd /opt/docker && ./bin/spark-notebook -Dapplication.context="$NOTEBOOK_APPLICATION_CONTEXT" \
            -Dmanager.notebooks.custom.args.0="-Dspark.driver.host=$SPARK_PUBLIC_DNS" \
            -Dmanager.notebooks.custom.args.1="-Dspark.driver.port=6666" \
            -Dmanager.notebooks.custom.args.2="-Dspark.driver.advertisedPort=6666" \
            -Dmanager.notebooks.custom.args.3="-Dspark.replClassServer.port=23456" \
            -Dmanager.notebooks.custom.args.4="-Dspark.replClassServer.advertisedPort=23456" \
            -Dmanager.notebooks.custom.sparkConf.spark.executor.uri=$ {SPARK_EXECUTOR_URI}

            \
            -Dmanager.notebooks.custom.sparkConf.spark.master=mesos://zk://172.17.0.1:2181/mesos \
            -Dmanager.notebooks.custom.sparkConf.spark.fileserver.port=6677 \
            -Dmanager.notebooks.custom.sparkConf.spark.fileserver.advertisedPort=6677 \
            -Dmanager.notebooks.custom.sparkConf.spark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory \
            -Dmanager.notebooks.custom.sparkConf.spark.broadcast.port=6688 \
            -Dmanager.notebooks.custom.sparkConf.spark.broadcast.advertisedPort=6688

          These settings are not passed when i open the notebook i had to edit the metadata for the notebook i wanted to use for example i tried to use http://localhost:31590/notebooks/core/Simple%20Spark.snb#:

          {
          "name": "Simple Spark",
          "user_save_timestamp": "2014-10-11T17:33:45.703Z",
          "auto_save_timestamp": "2015-01-10T00:02:12.659Z",
          "language_info":

          { "name": "scala", "file_extension": "scala", "codemirror_mode": "text/x-scala" }

          ,
          "trusted": true,
          "customLocalRepo": null,
          "customRepos": null,
          "customDeps": null,
          "customImports": null,
          "customArgs": [
          "-Dspark.driver.host=172.17.0.1",
          "-Dspark.driver.port=6666",
          "-Dspark.driver.advertisedPort=6666",
          "-Dspark.replClassServer.port=23456",
          "-Dspark.replClassServer.advertisedPort=23456"
          ]
          ,
          "customSparkConf":

          { "spark.master": "mesos://zk://172.17.0.1:2181/mesos", "spark.executor.uri": "/var/spark/spark-1.5.1-bin-custom-spark.tgz", "spark.fileserver.port": "6677", "spark.fileserver.advertisedPort": "6677", "spark.broadcast.factory": "org.apache.spark.broadcast.HttpBroadcastFactory", "spark.broadcast.port": "6688", "spark.broadcast.advertisedPort": "6688" }

          ,
          "kernelspec":

          { "name": "spark", "display_name": "Scala [2.10.4] Spark [1.5.1] Hadoop [2.6.0] " }

          }

          This is the log outuput at the the sparknotebook side:

          [info] application - customSparkConf >>

          {"spark.master":"mesos://zk://172.17.0.1:2181/mesos","spark.executor.uri":"/var/spark/spark-1.5.1-bin-custom-spark.tgz","spark.fileserver.port":"6677","spark.fileserver.advertisedPort":"6677","spark.broadcast.factory":"org.apache.spark.broadcast.HttpBroadcastFactory","spark.broadcast.port":"6688","spark.broadcast.advertisedPort":"6688"}

          [info] application - Spawning [/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java, -Xmx1834483712, -XX:MaxPermSize=1073741824, -server, -Dspark.driver.host=thinkpad, -Dspark.driver.port=6666, -Dspark.driver.advertisedPort=6666, -Dspark.replClassServer.port=23456, -Dspark.replClassServer.advertisedPort=23456, notebook.kernel.pfork.ChildProcessMain, notebook.kernel.remote.RemoteActorProcess, 44225, info, 1953f7c7-0119-4326-9ad3-d72dbeec846a, "core/Simple Spark.snb", kernel, ]
          [info] application - With Env Map(SHLVL > 1, JAVA_HOME -> /usr/lib/jvm/java-7-openjdk-amd64, GREP_OPTIONS -> --color=auto, PORT_9000 -> 31590, SPARK_LOCAL_HOSTNAME -> thinkpad, PWD -> /opt/docker, HOST -> thinkpad, HOSTNAME -> 0c4fba1664ac, MESOS_JAVA_NATIVE_LIBRARY -> /usr/local/lib/libmesos-0.22.0.so, PORT -> 31590, CLASSPATH_OVERRIDES -> /mnt/mesos/sandbox/akka-remote_2.10-2.3.4.jar:/mnt/mesos/sandbox/spark-core_2.10-1.5.1.jar:/mnt/mesos/sandbox/spark-repl_2.10-1.5.1.jar, GIT_PS1_SHOWDIRTYSTATE -> 1, MESOS_TASK_ID -> ubuntu2.f13323a6-8e01-11e5-a3ef-361f883cecc0, MESOS_SANDBOX -> /mnt/mesos/sandbox, GREP_COLOR -> 1;31, MARATHON_APP_VERSION -> 2015-11-18T14:37:49.682Z, SPARK_PUBLIC_DNS -> thinkpad, PORT0 -> 31590, MARATHON_APP_ID -> /ubuntu2, ADD_JARS -> ,/opt/docker/lib/common.common-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar, MESOS_CONTAINER_NAME -> mesos-031543d6-8344-4b3d-b294-c6c226287d92-S0.95338bd0-e9f2-4c6c-ba62-f8e5650f1ed7, PORTS -> 31590, NOTEBOOK_APPLICATION_CONTEXT -> /, MESOS_LOG_DIR -> /var/log/mesos, SPARK_EXECUTOR_URI -> /var/spark/spark-1.5.1-bin-custom-spark.tgz, MARATHON_APP_DOCKER_IMAGE -> andypetrella/spark-notebook:0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0, CLASSPATH -> /mnt/mesos/sandbox/akka-remote_2.10-2.3.4.jar:/mnt/mesos/sandbox/spark-core_2.10-1.5.1.jar:/mnt/mesos/sandbox/spark-repl_2.10-1.5.1.jar:/opt/docker:/opt/docker/lib/nooostab.spark-notebook-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/tachyon.tachyon-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/subprocess.subprocess-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/observable.observable-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/common.common-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/spark.spark-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/kernel.kernel-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/wisp_2.10-0.0.5.jar:/opt/docker/lib/org.scala-lang.scala-compiler-2.10.4.jar:/opt/docker/lib/org.scala-lang.scala-library-2.10.4.jar:/opt/docker/lib/org.scala-lang.scala-reflect-2.10.4.jar:/opt/docker/lib/com.google.guava.guava-14.0.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-common-0.7.1.jar:/opt/docker/lib/commons-io.commons-io-2.4.jar:/opt/docker/lib/log4j.log4j-1.2.16.jar:/opt/docker/lib/org.apache.thrift.libthrift-0.9.1.jar:/opt/docker/lib/org.apache.httpcomponents.httpclient-4.2.5.jar:/opt/docker/lib/org.apache.httpcomponents.httpcore-4.2.4.jar:/opt/docker/lib/org.tachyonproject.tachyon-client-0.7.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-underfs-hdfs-0.7.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-underfs-local-0.7.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-servers-0.7.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-client-unshaded-0.7.1.jar:/opt/docker/lib/org.eclipse.jetty.jetty-jsp-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.orbit.javax.servlet.jsp-2.1.0.v201105211820.jar:/opt/docker/lib/org.eclipse.jetty.orbit.org.apache.jasper.glassfish-2.1.0.v201110031002.jar:/opt/docker/lib/org.eclipse.jetty.orbit.javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/opt/docker/lib/org.eclipse.jetty.orbit.org.apache.taglibs.standard.glassfish-1.2.0.v201112081803.jar:/opt/docker/lib/org.eclipse.jetty.orbit.javax.el-2.1.0.v201105211819.jar:/opt/docker/lib/org.eclipse.jetty.orbit.com.sun.el-1.0.0.v201105211818.jar:/opt/docker/lib/org.eclipse.jetty.orbit.org.eclipse.jdt.core-3.7.1.jar:/opt/docker/lib/org.eclipse.jetty.jetty-webapp-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-xml-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-util-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-servlet-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-security-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-server-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-continuation-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-http-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-io-7.6.15.v20140411.jar:/opt/docker/lib/org.tachyonproject.tachyon-minicluster-0.7.1.jar:/opt/docker/lib/org.apache.curator.curator-test-2.1.0-incubating.jar:/opt/docker/lib/org.apache.commons.commons-math-2.2.jar:/opt/docker/lib/com.typesafe.play.play_2.10-2.3.7.jar:/opt/docker/lib/com.typesafe.play.build-link-2.3.7.jar:/opt/docker/lib/com.typesafe.play.play-exceptions-2.3.7.jar:/opt/docker/lib/org.javassist.javassist-3.18.2-GA.jar:/opt/docker/lib/org.scala-stm.scala-stm_2.10-0.7.jar:/opt/docker/lib/com.typesafe.config-1.2.1.jar:/opt/docker/lib/org.joda.joda-convert-1.6.jar:/opt/docker/lib/com.typesafe.play.twirl-api_2.10-1.0.2.jar:/opt/docker/lib/io.netty.netty-3.9.3.Final.jar:/opt/docker/lib/com.typesafe.netty.netty-http-pipelining-1.1.2.jar:/opt/docker/lib/ch.qos.logback.logback-core-1.1.1.jar:/opt/docker/lib/commons-codec.commons-codec-1.10.jar:/opt/docker/lib/xerces.xercesImpl-2.11.0.jar:/opt/docker/lib/xml-apis.xml-apis-1.4.01.jar:/opt/docker/lib/javax.transaction.jta-1.1.jar:/opt/docker/lib/com.typesafe.akka.akka-actor_2.10-2.3.11.jar:/opt/docker/lib/com.typesafe.akka.akka-remote_2.10-2.3.11.jar:/opt/docker/lib/com.google.protobuf.protobuf-java-2.5.0.jar:/opt/docker/lib/org.uncommons.maths.uncommons-maths-1.2.2a.jar:/opt/docker/lib/com.typesafe.akka.akka-slf4j_2.10-2.3.11.jar:/opt/docker/lib/org.apache.commons.commons-exec-1.3.jar:/opt/docker/lib/com.github.fommil.netlib.core-1.1.2.jar:/opt/docker/lib/net.sourceforge.f2j.arpack_combined_all-0.1.jar:/opt/docker/lib/net.sourceforge.f2j.arpack_combined_all-0.1-javadoc.jar:/opt/docker/lib/net.sf.opencsv.opencsv-2.3.jar:/opt/docker/lib/com.github.rwl.jtransforms-2.4.0.jar:/opt/docker/lib/org.spire-math.spire_2.10-0.7.4.jar:/opt/docker/lib/org.spire-math.spire-macros_2.10-0.7.4.jar:/opt/docker/lib/org.apache.spark.spark-core_2.10-1.5.1.jar:/opt/docker/lib/org.apache.avro.avro-mapred-1.7.7-hadoop2.jar:/opt/docker/lib/org.apache.avro.avro-ipc-1.7.7-tests.jar:/opt/docker/lib/org.apache.avro.avro-ipc-1.7.7.jar:/opt/docker/lib/org.apache.avro.avro-1.7.7.jar:/opt/docker/lib/org.codehaus.jackson.jackson-core-asl-1.9.13.jar:/opt/docker/lib/org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:/opt/docker/lib/org.apache.commons.commons-compress-1.4.1.jar:/opt/docker/lib/org.tukaani.xz-1.0.jar:/opt/docker/lib/com.twitter.chill_2.10-0.5.0.jar:/opt/docker/lib/com.twitter.chill-java-0.5.0.jar:/opt/docker/lib/com.esotericsoftware.kryo.kryo-2.21.jar:/opt/docker/lib/com.esotericsoftware.reflectasm.reflectasm-1.07-shaded.jar:/opt/docker/lib/com.esotericsoftware.minlog.minlog-1.2.jar:/opt/docker/lib/org.objenesis.objenesis-1.2.jar:/opt/docker/lib/org.apache.spark.spark-launcher_2.10-1.5.1.jar:/opt/docker/lib/org.spark-project.spark.unused-1.0.0.jar:/opt/docker/lib/org.apache.spark.spark-network-common_2.10-1.5.1.jar:/opt/docker/lib/io.netty.netty-all-4.0.29.Final.jar:/opt/docker/lib/org.apache.spark.spark-network-shuffle_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-unsafe_2.10-1.5.1.jar:/opt/docker/lib/com.google.code.findbugs.jsr305-1.3.9.jar:/opt/docker/lib/net.java.dev.jets3t.jets3t-0.9.0.jar:/opt/docker/lib/org.eclipse.jetty.orbit.javax.servlet-3.0.0.v201112011016.jar:/opt/docker/lib/org.apache.commons.commons-lang3-3.3.2.jar:/opt/docker/lib/org.apache.commons.commons-math3-3.4.1.jar:/opt/docker/lib/org.slf4j.slf4j-api-1.7.10.jar:/opt/docker/lib/org.slf4j.jul-to-slf4j-1.7.10.jar:/opt/docker/lib/org.slf4j.jcl-over-slf4j-1.7.10.jar:/opt/docker/lib/org.slf4j.slf4j-log4j12-1.7.10.jar:/opt/docker/lib/com.ning.compress-lzf-1.0.3.jar:/opt/docker/lib/org.xerial.snappy.snappy-java-1.1.1.7.jar:/opt/docker/lib/net.jpountz.lz4.lz4-1.3.0.jar:/opt/docker/lib/org.roaringbitmap.RoaringBitmap-0.4.5.jar:/opt/docker/lib/org.json4s.json4s-jackson_2.10-3.2.10.jar:/opt/docker/lib/org.json4s.json4s-core_2.10-3.2.10.jar:/opt/docker/lib/org.json4s.json4s-ast_2.10-3.2.10.jar:/opt/docker/lib/com.thoughtworks.paranamer.paranamer-2.6.jar:/opt/docker/lib/org.scala-lang.scalap-2.10.0.jar:/opt/docker/lib/com.fasterxml.jackson.core.jackson-databind-2.4.4.jar:/opt/docker/lib/com.fasterxml.jackson.core.jackson-core-2.4.4.jar:/opt/docker/lib/com.sun.jersey.jersey-server-1.9.jar:/opt/docker/lib/com.sun.jersey.jersey-core-1.9.jar:/opt/docker/lib/org.apache.mesos.mesos-0.21.1-shaded-protobuf.jar:/opt/docker/lib/com.clearspring.analytics.stream-2.7.0.jar:/opt/docker/lib/io.dropwizard.metrics.metrics-core-3.1.2.jar:/opt/docker/lib/io.dropwizard.metrics.metrics-jvm-3.1.2.jar:/opt/docker/lib/io.dropwizard.metrics.metrics-json-3.1.2.jar:/opt/docker/lib/io.dropwizard.metrics.metrics-graphite-3.1.2.jar:/opt/docker/lib/com.fasterxml.jackson.module.jackson-module-scala_2.10-2.4.4.jar:/opt/docker/lib/com.fasterxml.jackson.core.jackson-annotations-2.4.4.jar:/opt/docker/lib/oro.oro-2.0.8.jar:/opt/docker/lib/net.razorvine.pyrolite-4.4.jar:/opt/docker/lib/net.sf.py4j.py4j-0.8.2.1.jar:/opt/docker/lib/com.jamesmurty.utils.java-xmlbuilder-0.4.jar:/opt/docker/lib/org.apache.spark.spark-yarn_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-sql_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-catalyst_2.10-1.5.1.jar:/opt/docker/lib/org.codehaus.janino.janino-2.7.8.jar:/opt/docker/lib/org.codehaus.janino.commons-compiler-2.7.8.jar:/opt/docker/lib/org.apache.parquet.parquet-column-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-common-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-encoding-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-generator-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-hadoop-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-format-2.3.0-incubating.jar:/opt/docker/lib/org.apache.parquet.parquet-jackson-1.7.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-client-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-common-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-annotations-2.6.0.jar:/opt/docker/lib/commons-cli.commons-cli-1.2.jar:/opt/docker/lib/xmlenc.xmlenc-0.52.jar:/opt/docker/lib/commons-httpclient.commons-httpclient-3.1.jar:/opt/docker/lib/commons-logging.commons-logging-1.1.3.jar:/opt/docker/lib/commons-net.commons-net-3.1.jar:/opt/docker/lib/commons-collections.commons-collections-3.2.1.jar:/opt/docker/lib/commons-lang.commons-lang-2.6.jar:/opt/docker/lib/commons-configuration.commons-configuration-1.6.jar:/opt/docker/lib/commons-digester.commons-digester-1.8.jar:/opt/docker/lib/commons-beanutils.commons-beanutils-1.7.0.jar:/opt/docker/lib/commons-beanutils.commons-beanutils-core-1.8.0.jar:/opt/docker/lib/com.google.code.gson.gson-2.2.4.jar:/opt/docker/lib/org.apache.hadoop.hadoop-auth-2.6.0.jar:/opt/docker/lib/org.apache.directory.server.apacheds-kerberos-codec-2.0.0-M15.jar:/opt/docker/lib/org.apache.directory.server.apacheds-i18n-2.0.0-M15.jar:/opt/docker/lib/org.apache.directory.api.api-asn1-api-1.0.0-M20.jar:/opt/docker/lib/org.apache.directory.api.api-util-1.0.0-M20.jar:/opt/docker/lib/org.apache.curator.curator-framework-2.6.0.jar:/opt/docker/lib/org.apache.curator.curator-client-2.6.0.jar:/opt/docker/lib/org.apache.zookeeper.zookeeper-3.4.6.jar:/opt/docker/lib/org.apache.curator.curator-recipes-2.6.0.jar:/opt/docker/lib/org.htrace.htrace-core-3.0.4.jar:/opt/docker/lib/org.apache.hadoop.hadoop-hdfs-2.6.0.jar:/opt/docker/lib/org.mortbay.jetty.jetty-util-6.1.26.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-app-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-common-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-common-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-api-2.6.0.jar:/opt/docker/lib/com.sun.jersey.jersey-client-1.9.jar:/opt/docker/lib/org.codehaus.jackson.jackson-jaxrs-1.9.13.jar:/opt/docker/lib/org.codehaus.jackson.jackson-xc-1.9.13.jar:/opt/docker/lib/com.google.inject.guice-3.0.jar:/opt/docker/lib/javax.inject.javax.inject-1.jar:/opt/docker/lib/aopalliance.aopalliance-1.0.jar:/opt/docker/lib/org.sonatype.sisu.inject.cglib-2.2.1-v20090111.jar:/opt/docker/lib/asm.asm-3.2.jar:/opt/docker/lib/com.sun.jersey.jersey-json-1.9.jar:/opt/docker/lib/org.codehaus.jettison.jettison-1.1.jar:/opt/docker/lib/com.sun.jersey.contribs.jersey-guice-1.9.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-client-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-core-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-server-common-2.6.0.jar:/opt/docker/lib/org.fusesource.leveldbjni.leveldbjni-all-1.8.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-shuffle-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-jobclient-2.6.0.jar:/opt/docker/lib/org.apache.spark.spark-repl_2.10-1.5.1.jar:/opt/docker/lib/org.scala-lang.jline-2.10.4.jar:/opt/docker/lib/org.fusesource.jansi.jansi-1.4.jar:/opt/docker/lib/org.apache.spark.spark-bagel_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-mllib_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-streaming_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-graphx_2.10-1.5.1.jar:/opt/docker/lib/org.scalanlp.breeze_2.10-0.11.2.jar:/opt/docker/lib/org.scalanlp.breeze-macros_2.10-0.11.2.jar:/opt/docker/lib/org.jpmml.pmml-model-1.1.15.jar:/opt/docker/lib/org.jpmml.pmml-agent-1.1.15.jar:/opt/docker/lib/org.jpmml.pmml-schema-1.1.15.jar:/opt/docker/lib/com.sun.xml.bind.jaxb-impl-2.2.7.jar:/opt/docker/lib/com.sun.xml.bind.jaxb-core-2.2.7.jar:/opt/docker/lib/javax.xml.bind.jaxb-api-2.2.7.jar:/opt/docker/lib/com.sun.istack.istack-commons-runtime-2.16.jar:/opt/docker/lib/com.sun.xml.fastinfoset.FastInfoset-1.2.12.jar:/opt/docker/lib/javax.xml.bind.jsr173_api-1.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-server-web-proxy-2.6.0.jar:/opt/docker/lib/org.mortbay.jetty.jetty-6.1.26.jar:/opt/docker/lib/io.reactivex.rxscala_2.10-0.22.0.jar:/opt/docker/lib/io.reactivex.rxjava-1.0.0-rc.5.jar:/opt/docker/lib/org.scalaz.scalaz-core_2.10-7.0.6.jar:/opt/docker/lib/org.scala-sbt.sbt-0.13.8.jar:/opt/docker/lib/org.scala-sbt.main-0.13.8.jar:/opt/docker/lib/org.scala-sbt.actions-0.13.8.jar:/opt/docker/lib/org.scala-sbt.classpath-0.13.8.jar:/opt/docker/lib/org.scala-sbt.launcher-interface-0.13.8.jar:/opt/docker/lib/org.scala-sbt.interface-0.13.8.jar:/opt/docker/lib/org.scala-sbt.io-0.13.8.jar:/opt/docker/lib/org.scala-sbt.control-0.13.8.jar:/opt/docker/lib/org.scala-sbt.completion-0.13.8.jar:/opt/docker/lib/org.scala-sbt.collections-0.13.8.jar:/opt/docker/lib/jline.jline-2.11.jar:/opt/docker/lib/org.scala-sbt.api-0.13.8.jar:/opt/docker/lib/org.scala-sbt.compiler-integration-0.13.8.jar:/opt/docker/lib/org.scala-sbt.incremental-compiler-0.13.8.jar:/opt/docker/lib/org.scala-sbt.logging-0.13.8.jar:/opt/docker/lib/org.scala-sbt.process-0.13.8.jar:/opt/docker/lib/org.scala-sbt.relation-0.13.8.jar:/opt/docker/lib/org.scala-sbt.compile-0.13.8.jar:/opt/docker/lib/org.scala-sbt.classfile-0.13.8.jar:/opt/docker/lib/org.scala-sbt.persist-0.13.8.jar:/opt/docker/lib/org.scala-tools.sbinary.sbinary_2.10-0.4.2.jar:/opt/docker/lib/org.scala-sbt.compiler-ivy-integration-0.13.8.jar:/opt/docker/lib/org.scala-sbt.ivy-0.13.8.jar:/opt/docker/lib/org.scala-sbt.cross-0.13.8.jar:/opt/docker/lib/org.scala-sbt.ivy.ivy-2.3.0-sbt-fccfbd44c9f64523b61398a0155784dcbaeae28f.jar:/opt/docker/lib/com.jcraft.jsch-0.1.46.jar:/opt/docker/lib/org.scala-sbt.serialization_2.10-0.1.1.jar:/opt/docker/lib/org.scala-lang.modules.scala-pickling_2.10-0.10.0.jar:/opt/docker/lib/org.scalamacros.quasiquotes_2.10-2.0.1.jar:/opt/docker/lib/org.spire-math.jawn-parser_2.10-0.6.0.jar:/opt/docker/lib/org.spire-math.json4s-support_2.10-0.6.0.jar:/opt/docker/lib/org.scala-sbt.run-0.13.8.jar:/opt/docker/lib/org.scala-sbt.task-system-0.13.8.jar:/opt/docker/lib/org.scala-sbt.tasks-0.13.8.jar:/opt/docker/lib/org.scala-sbt.tracking-0.13.8.jar:/opt/docker/lib/org.scala-sbt.cache-0.13.8.jar:/opt/docker/lib/org.scala-sbt.testing-0.13.8.jar:/opt/docker/lib/org.scala-sbt.test-agent-0.13.8.jar:/opt/docker/lib/org.scala-sbt.test-interface-1.0.jar:/opt/docker/lib/org.scala-sbt.main-settings-0.13.8.jar:/opt/docker/lib/org.scala-sbt.apply-macro-0.13.8.jar:/opt/docker/lib/org.scala-sbt.command-0.13.8.jar:/opt/docker/lib/org.scala-sbt.logic-0.13.8.jar:/opt/docker/lib/org.scala-sbt.compiler-interfacebin-0.13.8.jar:/opt/docker/lib/org.scala-sbt.compiler-interface-src-0.13.8.jar:/opt/docker/lib/org.scala-sbt.precompiled-2_8_2-compiler-interface-bin-0.13.8.jar:/opt/docker/lib/org.scala-sbt.precompiled-2_9_2-compiler-interface-bin-0.13.8.jar:/opt/docker/lib/org.scala-sbt.precompiled-2_9_3-compiler-interface-bin-0.13.8.jar:/opt/docker/lib/com.frugalmechanic.fm-sbt-s3-resolver-0.5.0.jar:/opt/docker/lib/com.amazonaws.aws-java-sdk-s3-1.9.0.jar:/opt/docker/lib/com.amazonaws.aws-java-sdk-core-1.9.0.jar:/opt/docker/lib/joda-time.joda-time-2.9.1.jar:/opt/docker/lib/io.continuum.bokeh.bokeh_2.10-0.2.jar:/opt/docker/lib/io.continuum.bokeh.core_2.10-0.2.jar:/opt/docker/lib/com.typesafe.play.play-json_2.10-2.4.0-M1.jar:/opt/docker/lib/com.typesafe.play.play-iteratees_2.10-2.4.0-M1.jar:/opt/docker/lib/com.typesafe.play.play-functional_2.10-2.4.0-M1.jar:/opt/docker/lib/com.typesafe.play.play-datacommons_2.10-2.4.0-M1.jar:/opt/docker/lib/io.continuum.bokeh.bokehjs_2.10-0.2.jar:/opt/docker/lib/com.github.scala-incubator.io.scala-io-core_2.10-0.4.3.jar:/opt/docker/lib/com.jsuereth.scala-arm_2.10-1.3.jar:/opt/docker/lib/com.github.scala-incubator.io.scala-io-file_2.10-0.4.3.jar:/opt/docker/lib/com.quantifind.sumac_2.10-0.3.0.jar:/opt/docker/lib/com.typesafe.play.play-cache_2.10-2.3.7.jar:/opt/docker/lib/net.sf.ehcache.ehcache-core-2.6.8.jar:/opt/docker/lib/nooostab.spark-notebook-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0-assets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/icedtea-sound.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar, PATH -> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, CLICOLOR -> 1, HOME -> /root, SPARK_LOCAL_IP -> 172.17.0.2, AGENT_ADVERTISE_IP -> 172.17.0.1, LIBPROCESS_ADVERTISE_IP -> 172.17.0.1, LS_COLORS -> di=34:ln=35:so=32:pi=33:ex=1;40:bd=34;40:cd=34;40:su=0;40:sg=0;40:tw=0;40:ow=0;40:, LIBPROCESS_ADVERTISE_PORT -> 9050)
          [info] application - In working directory .
          [info] application - Registering first web-socket (notebook.server.WebSockWrapperImpl@57ae2ceb) in service notebook.server.CalcWebSocketService$CalcActor@6c072545
          [info] application - Spawning calculator in service notebook.server.CalcWebSocketService$CalcActor@6c072545
          [DEBUG] [11/18/2015 14:57:19.073] [main] [EventStream] StandardOutLogger started
          [DEBUG] [11/18/2015 14:57:19.263] [main] [EventStream(akka://Remote)] logger log1-Slf4jLogger started
          [DEBUG] [11/18/2015 14:57:19.265] [main] [EventStream(akka://Remote)] Default Loggers started
          I1118 14:57:28.938076 3879 logging.cpp:172] INFO level logging started!
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@716: Client environment:host.name=0c4fba1664ac
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@724: Client environment:os.arch=3.19.0-30-generic
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@725: Client environment:os.version=#34~14.04.1-Ubuntu SMP Fri Oct 2 22:09:39 UTC 2015
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@741: Client environment:user.home=/root
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@753: Client environment:user.dir=/opt/docker
          2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=172.17.0.1:2181 sessionTimeout=10000 watcher=0x7f3d1f4ad600 sessionId=0 sessionPasswd=<null> context=0x7f3d80001e10 flags=0
          2015-11-18 14:57:28,940:3857(0x7f3d14de2700):ZOO_INFO@check_events@1703: initiated connection to server [172.17.0.1:2181]
          I1118 14:57:28.940862 3956 sched.cpp:164] Version: 0.25.0
          2015-11-18 14:57:28,944:3857(0x7f3d14de2700):ZOO_INFO@check_events@1750: session establishment complete on server [172.17.0.1:2181], sessionId=0x1511a4b0ab0000b, negotiated timeout=10000
          I1118 14:57:28.944319 3950 group.cpp:331] Group process (group(1)@172.17.0.1:9050) connected to ZooKeeper
          I1118 14:57:28.944707 3950 group.cpp:805] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
          I1118 14:57:28.944722 3950 group.cpp:403] Trying to create path '/mesos' in ZooKeeper
          I1118 14:57:28.948410 3950 detector.cpp:156] Detected a new leader: (id='1')
          I1118 14:57:28.948495 3952 group.cpp:674] Trying to get '/mesos/json.info_0000000001' in ZooKeeper
          I1118 14:57:28.948930 3951 detector.cpp:481] A new leading master (UPID=master@172.17.0.1:5050) is detected
          I1118 14:57:28.948982 3951 sched.cpp:262] New master detected at master@172.17.0.1:5050
          I1118 14:57:28.949141 3951 sched.cpp:272] No credentials provided. Attempting to register without authentication

          At the mesos master i got:
          I1118 16:32:04.114322 11968 master.cpp:2179] Received SUBSCRIBE call for framework 'Simple Spark' at scheduler-c860460e-8878-4e92-a28e-871890ef2227@172.17.0.1:9050
          I1118 16:32:04.114537 11968 master.cpp:2250] Subscribing framework Simple Spark with checkpointing disabled and capabilities [ ]
          E1118 16:32:04.114892 11975 socket.hpp:174] Shutdown failed on fd=28: Transport endpoint is not connected [107]
          I1118 16:32:04.115171 11968 master.cpp:1119] Framework 8236605c-0abd-4562-8061-0520b9448f41-0235 (Simple Spark) at scheduler-

          Info for spark notebook:
          Build: | buildTime-Mon Nov 16 14:38:21 UTC 2015 | formattedShaVersion-0.6.2-SNAPSHOT-dcacd17902968c767f98fbc659d6115e25a5f753 | sbtVersion-0.13.8 | scalaVersion-2.10.4 | sparkNotebookVersion-0.6.2-SNAPSHOT | hadoopVersion-2.6.0 | jets3tVersion-0.7.1 | jlineDef-(org.scala-lang,2.10.4) | sparkVersion-1.5.1 | withHive-false | withParquet-false |.

          So it seems that something is wrong with the connection from scheduler to the mesos master... whats the best setup to test it?

          Show
          skonto Stavros Kontopoulos added a comment - - edited Ok i tried to veirfy that but no result: On my host machine: echo 'docker,mesos' > /etc/mesos-slave/containerizers echo '5mins' > /etc/mesos-slave/executor_registration_timeout sudo service mesos-slave restart curl -X POST -H "Content-Type: application/json" http://localhost:8080/v2/apps -d@Docker.json #copy patched jars export CONT="$(docker ps | grep mesos | awk ' {print $15; } ')" docker cp spark-core_2.10-1.5.1.jar $CONT:/mnt/mesos/sandbox docker cp spark-repl_2.10-1.5.1.jar $CONT:/mnt/mesos/sandbox docker cp akka-remote_2.10-2.3.4.jar $CONT:/mnt/mesos/sandbox Docker.json: { "container": { "type": "DOCKER", "name" : "test", "docker": { "image": "andypetrella/spark-notebook:0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0", "network": "BRIDGE", "portMappings": [ { "containerPort": 9000, "hostPort": 0, "protocol": "tcp" } ] }, "volumes": [ { "containerPath": "/var/spark/spark-1.5.1-bin-custom-spark.tgz", "hostPath": "/xxxx/spark-1.5.1-bin-custom-spark.tgz", "mode": "RO" } ] }, "id": "ubuntu2", "instances": 1, "cpus": 2, "mem": 1024, "cmd" : "while sleep 1000; do date -u +%T; done" } On my host Ubuntu machine i run mesos-master, mesos-slave, and zookeeper. zookeeper listens on *:2181. My hostname points to 172.17.0.1 in /etc/hosts. I have used this image "andypetrella/spark-notebook:0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0 The docker0 interface has ip 172.17.0.1 The containers private ip is 172.17.0.2 You can ping to each other... Next i run inside the container: sudo apt-get --only-upgrade install mesos #upgrade to 0.25.0 export AGENT_ADVERTISE_IP=172.17.0.1 export LIBPROCESS_ADVERTISE_IP=$AGENT_ADVERTISE_IP export LIBPROCESS_ADVERTISE_PORT=9050 # choose your own port export SPARK_LOCAL_IP=$(ip -o -4 addr list eth0 | perl -n -e 'if (m {inet\s([\d\.]+)\/\d+\s} xms) { print $1 } ') #172.17.0.1 export SPARK_PUBLIC_DNS=$HOST export SPARK_LOCAL_HOSTNAME=$HOST //thinkpad->172.17.0.2 this is required so Spark can bind to the $SPARK_PUBLIC_DNS my_own_ip=$(cat /etc/hosts | grep $HOSTNAME | awk ' {print $1} ') #172.17.0.1 echo "$my_own_ip $SPARK_PUBLIC_DNS" >> /etc/hosts export SPARK_EXECUTOR_URI=/var/spark/spark-1.5.1-bin-custom-spark.tgz Make sure the spark-core, spark-repl and akka-remote jars are in the sandbox (add them with uris). This is important now, these 3 jars have to placed on the class path first, before any other jar goes in, Data Fellas Spark notebook provides CLASSPATH_OVERRIDES env variable especially for that: export CLASSPATH_OVERRIDES=$MESOS_SANDBOX/akka-remote_2.10-2.3.4.jar:$MESOS_SANDBOX/spark-core_2.10-1.5.1.jar:$MESOS_SANDBOX/spark-repl_2.10-1.5.1.jar export NOTEBOOK_APPLICATION_CONTEXT=/ Spark notebook settings: cd /opt/docker && ./bin/spark-notebook -Dapplication.context="$NOTEBOOK_APPLICATION_CONTEXT" \ -Dmanager.notebooks.custom.args.0="-Dspark.driver.host=$SPARK_PUBLIC_DNS" \ -Dmanager.notebooks.custom.args.1="-Dspark.driver.port=6666" \ -Dmanager.notebooks.custom.args.2="-Dspark.driver.advertisedPort=6666" \ -Dmanager.notebooks.custom.args.3="-Dspark.replClassServer.port=23456" \ -Dmanager.notebooks.custom.args.4="-Dspark.replClassServer.advertisedPort=23456" \ -Dmanager.notebooks.custom.sparkConf.spark.executor.uri=$ {SPARK_EXECUTOR_URI} \ -Dmanager.notebooks.custom.sparkConf.spark.master=mesos://zk://172.17.0.1:2181/mesos \ -Dmanager.notebooks.custom.sparkConf.spark.fileserver.port=6677 \ -Dmanager.notebooks.custom.sparkConf.spark.fileserver.advertisedPort=6677 \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.port=6688 \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.advertisedPort=6688 These settings are not passed when i open the notebook i had to edit the metadata for the notebook i wanted to use for example i tried to use http://localhost:31590/notebooks/core/Simple%20Spark.snb#: { "name": "Simple Spark", "user_save_timestamp": "2014-10-11T17:33:45.703Z", "auto_save_timestamp": "2015-01-10T00:02:12.659Z", "language_info": { "name": "scala", "file_extension": "scala", "codemirror_mode": "text/x-scala" } , "trusted": true, "customLocalRepo": null, "customRepos": null, "customDeps": null, "customImports": null, "customArgs": [ "-Dspark.driver.host=172.17.0.1", "-Dspark.driver.port=6666", "-Dspark.driver.advertisedPort=6666", "-Dspark.replClassServer.port=23456", "-Dspark.replClassServer.advertisedPort=23456" ] , "customSparkConf": { "spark.master": "mesos://zk://172.17.0.1:2181/mesos", "spark.executor.uri": "/var/spark/spark-1.5.1-bin-custom-spark.tgz", "spark.fileserver.port": "6677", "spark.fileserver.advertisedPort": "6677", "spark.broadcast.factory": "org.apache.spark.broadcast.HttpBroadcastFactory", "spark.broadcast.port": "6688", "spark.broadcast.advertisedPort": "6688" } , "kernelspec": { "name": "spark", "display_name": "Scala [2.10.4] Spark [1.5.1] Hadoop [2.6.0] " } } This is the log outuput at the the sparknotebook side: [info] application - customSparkConf >> {"spark.master":"mesos://zk://172.17.0.1:2181/mesos","spark.executor.uri":"/var/spark/spark-1.5.1-bin-custom-spark.tgz","spark.fileserver.port":"6677","spark.fileserver.advertisedPort":"6677","spark.broadcast.factory":"org.apache.spark.broadcast.HttpBroadcastFactory","spark.broadcast.port":"6688","spark.broadcast.advertisedPort":"6688"} [info] application - Spawning [/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java, -Xmx1834483712, -XX:MaxPermSize=1073741824, -server, -Dspark.driver.host=thinkpad, -Dspark.driver.port=6666, -Dspark.driver.advertisedPort=6666, -Dspark.replClassServer.port=23456, -Dspark.replClassServer.advertisedPort=23456, notebook.kernel.pfork.ChildProcessMain, notebook.kernel.remote.RemoteActorProcess, 44225, info, 1953f7c7-0119-4326-9ad3-d72dbeec846a, "core/Simple Spark.snb", kernel, ] [info] application - With Env Map(SHLVL > 1, JAVA_HOME -> /usr/lib/jvm/java-7-openjdk-amd64, GREP_OPTIONS -> --color=auto, PORT_9000 -> 31590, SPARK_LOCAL_HOSTNAME -> thinkpad, PWD -> /opt/docker, HOST -> thinkpad, HOSTNAME -> 0c4fba1664ac, MESOS_JAVA_NATIVE_LIBRARY -> /usr/local/lib/libmesos-0.22.0.so, PORT -> 31590, CLASSPATH_OVERRIDES -> /mnt/mesos/sandbox/akka-remote_2.10-2.3.4.jar:/mnt/mesos/sandbox/spark-core_2.10-1.5.1.jar:/mnt/mesos/sandbox/spark-repl_2.10-1.5.1.jar, GIT_PS1_SHOWDIRTYSTATE -> 1, MESOS_TASK_ID -> ubuntu2.f13323a6-8e01-11e5-a3ef-361f883cecc0, MESOS_SANDBOX -> /mnt/mesos/sandbox, GREP_COLOR -> 1;31, MARATHON_APP_VERSION -> 2015-11-18T14:37:49.682Z, SPARK_PUBLIC_DNS -> thinkpad, PORT0 -> 31590, MARATHON_APP_ID -> /ubuntu2, ADD_JARS -> ,/opt/docker/lib/common.common-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar, MESOS_CONTAINER_NAME -> mesos-031543d6-8344-4b3d-b294-c6c226287d92-S0.95338bd0-e9f2-4c6c-ba62-f8e5650f1ed7, PORTS -> 31590, NOTEBOOK_APPLICATION_CONTEXT -> /, MESOS_LOG_DIR -> /var/log/mesos, SPARK_EXECUTOR_URI -> /var/spark/spark-1.5.1-bin-custom-spark.tgz, MARATHON_APP_DOCKER_IMAGE -> andypetrella/spark-notebook:0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0, CLASSPATH -> /mnt/mesos/sandbox/akka-remote_2.10-2.3.4.jar:/mnt/mesos/sandbox/spark-core_2.10-1.5.1.jar:/mnt/mesos/sandbox/spark-repl_2.10-1.5.1.jar:/opt/docker:/opt/docker/lib/nooostab.spark-notebook-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/tachyon.tachyon-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/subprocess.subprocess-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/observable.observable-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/common.common-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/spark.spark-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/kernel.kernel-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0.jar:/opt/docker/lib/wisp_2.10-0.0.5.jar:/opt/docker/lib/org.scala-lang.scala-compiler-2.10.4.jar:/opt/docker/lib/org.scala-lang.scala-library-2.10.4.jar:/opt/docker/lib/org.scala-lang.scala-reflect-2.10.4.jar:/opt/docker/lib/com.google.guava.guava-14.0.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-common-0.7.1.jar:/opt/docker/lib/commons-io.commons-io-2.4.jar:/opt/docker/lib/log4j.log4j-1.2.16.jar:/opt/docker/lib/org.apache.thrift.libthrift-0.9.1.jar:/opt/docker/lib/org.apache.httpcomponents.httpclient-4.2.5.jar:/opt/docker/lib/org.apache.httpcomponents.httpcore-4.2.4.jar:/opt/docker/lib/org.tachyonproject.tachyon-client-0.7.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-underfs-hdfs-0.7.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-underfs-local-0.7.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-servers-0.7.1.jar:/opt/docker/lib/org.tachyonproject.tachyon-client-unshaded-0.7.1.jar:/opt/docker/lib/org.eclipse.jetty.jetty-jsp-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.orbit.javax.servlet.jsp-2.1.0.v201105211820.jar:/opt/docker/lib/org.eclipse.jetty.orbit.org.apache.jasper.glassfish-2.1.0.v201110031002.jar:/opt/docker/lib/org.eclipse.jetty.orbit.javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/opt/docker/lib/org.eclipse.jetty.orbit.org.apache.taglibs.standard.glassfish-1.2.0.v201112081803.jar:/opt/docker/lib/org.eclipse.jetty.orbit.javax.el-2.1.0.v201105211819.jar:/opt/docker/lib/org.eclipse.jetty.orbit.com.sun.el-1.0.0.v201105211818.jar:/opt/docker/lib/org.eclipse.jetty.orbit.org.eclipse.jdt.core-3.7.1.jar:/opt/docker/lib/org.eclipse.jetty.jetty-webapp-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-xml-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-util-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-servlet-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-security-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-server-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-continuation-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-http-7.6.15.v20140411.jar:/opt/docker/lib/org.eclipse.jetty.jetty-io-7.6.15.v20140411.jar:/opt/docker/lib/org.tachyonproject.tachyon-minicluster-0.7.1.jar:/opt/docker/lib/org.apache.curator.curator-test-2.1.0-incubating.jar:/opt/docker/lib/org.apache.commons.commons-math-2.2.jar:/opt/docker/lib/com.typesafe.play.play_2.10-2.3.7.jar:/opt/docker/lib/com.typesafe.play.build-link-2.3.7.jar:/opt/docker/lib/com.typesafe.play.play-exceptions-2.3.7.jar:/opt/docker/lib/org.javassist.javassist-3.18.2-GA.jar:/opt/docker/lib/org.scala-stm.scala-stm_2.10-0.7.jar:/opt/docker/lib/com.typesafe.config-1.2.1.jar:/opt/docker/lib/org.joda.joda-convert-1.6.jar:/opt/docker/lib/com.typesafe.play.twirl-api_2.10-1.0.2.jar:/opt/docker/lib/io.netty.netty-3.9.3.Final.jar:/opt/docker/lib/com.typesafe.netty.netty-http-pipelining-1.1.2.jar:/opt/docker/lib/ch.qos.logback.logback-core-1.1.1.jar:/opt/docker/lib/commons-codec.commons-codec-1.10.jar:/opt/docker/lib/xerces.xercesImpl-2.11.0.jar:/opt/docker/lib/xml-apis.xml-apis-1.4.01.jar:/opt/docker/lib/javax.transaction.jta-1.1.jar:/opt/docker/lib/com.typesafe.akka.akka-actor_2.10-2.3.11.jar:/opt/docker/lib/com.typesafe.akka.akka-remote_2.10-2.3.11.jar:/opt/docker/lib/com.google.protobuf.protobuf-java-2.5.0.jar:/opt/docker/lib/org.uncommons.maths.uncommons-maths-1.2.2a.jar:/opt/docker/lib/com.typesafe.akka.akka-slf4j_2.10-2.3.11.jar:/opt/docker/lib/org.apache.commons.commons-exec-1.3.jar:/opt/docker/lib/com.github.fommil.netlib.core-1.1.2.jar:/opt/docker/lib/net.sourceforge.f2j.arpack_combined_all-0.1.jar:/opt/docker/lib/net.sourceforge.f2j.arpack_combined_all-0.1-javadoc.jar:/opt/docker/lib/net.sf.opencsv.opencsv-2.3.jar:/opt/docker/lib/com.github.rwl.jtransforms-2.4.0.jar:/opt/docker/lib/org.spire-math.spire_2.10-0.7.4.jar:/opt/docker/lib/org.spire-math.spire-macros_2.10-0.7.4.jar:/opt/docker/lib/org.apache.spark.spark-core_2.10-1.5.1.jar:/opt/docker/lib/org.apache.avro.avro-mapred-1.7.7-hadoop2.jar:/opt/docker/lib/org.apache.avro.avro-ipc-1.7.7-tests.jar:/opt/docker/lib/org.apache.avro.avro-ipc-1.7.7.jar:/opt/docker/lib/org.apache.avro.avro-1.7.7.jar:/opt/docker/lib/org.codehaus.jackson.jackson-core-asl-1.9.13.jar:/opt/docker/lib/org.codehaus.jackson.jackson-mapper-asl-1.9.13.jar:/opt/docker/lib/org.apache.commons.commons-compress-1.4.1.jar:/opt/docker/lib/org.tukaani.xz-1.0.jar:/opt/docker/lib/com.twitter.chill_2.10-0.5.0.jar:/opt/docker/lib/com.twitter.chill-java-0.5.0.jar:/opt/docker/lib/com.esotericsoftware.kryo.kryo-2.21.jar:/opt/docker/lib/com.esotericsoftware.reflectasm.reflectasm-1.07-shaded.jar:/opt/docker/lib/com.esotericsoftware.minlog.minlog-1.2.jar:/opt/docker/lib/org.objenesis.objenesis-1.2.jar:/opt/docker/lib/org.apache.spark.spark-launcher_2.10-1.5.1.jar:/opt/docker/lib/org.spark-project.spark.unused-1.0.0.jar:/opt/docker/lib/org.apache.spark.spark-network-common_2.10-1.5.1.jar:/opt/docker/lib/io.netty.netty-all-4.0.29.Final.jar:/opt/docker/lib/org.apache.spark.spark-network-shuffle_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-unsafe_2.10-1.5.1.jar:/opt/docker/lib/com.google.code.findbugs.jsr305-1.3.9.jar:/opt/docker/lib/net.java.dev.jets3t.jets3t-0.9.0.jar:/opt/docker/lib/org.eclipse.jetty.orbit.javax.servlet-3.0.0.v201112011016.jar:/opt/docker/lib/org.apache.commons.commons-lang3-3.3.2.jar:/opt/docker/lib/org.apache.commons.commons-math3-3.4.1.jar:/opt/docker/lib/org.slf4j.slf4j-api-1.7.10.jar:/opt/docker/lib/org.slf4j.jul-to-slf4j-1.7.10.jar:/opt/docker/lib/org.slf4j.jcl-over-slf4j-1.7.10.jar:/opt/docker/lib/org.slf4j.slf4j-log4j12-1.7.10.jar:/opt/docker/lib/com.ning.compress-lzf-1.0.3.jar:/opt/docker/lib/org.xerial.snappy.snappy-java-1.1.1.7.jar:/opt/docker/lib/net.jpountz.lz4.lz4-1.3.0.jar:/opt/docker/lib/org.roaringbitmap.RoaringBitmap-0.4.5.jar:/opt/docker/lib/org.json4s.json4s-jackson_2.10-3.2.10.jar:/opt/docker/lib/org.json4s.json4s-core_2.10-3.2.10.jar:/opt/docker/lib/org.json4s.json4s-ast_2.10-3.2.10.jar:/opt/docker/lib/com.thoughtworks.paranamer.paranamer-2.6.jar:/opt/docker/lib/org.scala-lang.scalap-2.10.0.jar:/opt/docker/lib/com.fasterxml.jackson.core.jackson-databind-2.4.4.jar:/opt/docker/lib/com.fasterxml.jackson.core.jackson-core-2.4.4.jar:/opt/docker/lib/com.sun.jersey.jersey-server-1.9.jar:/opt/docker/lib/com.sun.jersey.jersey-core-1.9.jar:/opt/docker/lib/org.apache.mesos.mesos-0.21.1-shaded-protobuf.jar:/opt/docker/lib/com.clearspring.analytics.stream-2.7.0.jar:/opt/docker/lib/io.dropwizard.metrics.metrics-core-3.1.2.jar:/opt/docker/lib/io.dropwizard.metrics.metrics-jvm-3.1.2.jar:/opt/docker/lib/io.dropwizard.metrics.metrics-json-3.1.2.jar:/opt/docker/lib/io.dropwizard.metrics.metrics-graphite-3.1.2.jar:/opt/docker/lib/com.fasterxml.jackson.module.jackson-module-scala_2.10-2.4.4.jar:/opt/docker/lib/com.fasterxml.jackson.core.jackson-annotations-2.4.4.jar:/opt/docker/lib/oro.oro-2.0.8.jar:/opt/docker/lib/net.razorvine.pyrolite-4.4.jar:/opt/docker/lib/net.sf.py4j.py4j-0.8.2.1.jar:/opt/docker/lib/com.jamesmurty.utils.java-xmlbuilder-0.4.jar:/opt/docker/lib/org.apache.spark.spark-yarn_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-sql_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-catalyst_2.10-1.5.1.jar:/opt/docker/lib/org.codehaus.janino.janino-2.7.8.jar:/opt/docker/lib/org.codehaus.janino.commons-compiler-2.7.8.jar:/opt/docker/lib/org.apache.parquet.parquet-column-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-common-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-encoding-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-generator-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-hadoop-1.7.0.jar:/opt/docker/lib/org.apache.parquet.parquet-format-2.3.0-incubating.jar:/opt/docker/lib/org.apache.parquet.parquet-jackson-1.7.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-client-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-common-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-annotations-2.6.0.jar:/opt/docker/lib/commons-cli.commons-cli-1.2.jar:/opt/docker/lib/xmlenc.xmlenc-0.52.jar:/opt/docker/lib/commons-httpclient.commons-httpclient-3.1.jar:/opt/docker/lib/commons-logging.commons-logging-1.1.3.jar:/opt/docker/lib/commons-net.commons-net-3.1.jar:/opt/docker/lib/commons-collections.commons-collections-3.2.1.jar:/opt/docker/lib/commons-lang.commons-lang-2.6.jar:/opt/docker/lib/commons-configuration.commons-configuration-1.6.jar:/opt/docker/lib/commons-digester.commons-digester-1.8.jar:/opt/docker/lib/commons-beanutils.commons-beanutils-1.7.0.jar:/opt/docker/lib/commons-beanutils.commons-beanutils-core-1.8.0.jar:/opt/docker/lib/com.google.code.gson.gson-2.2.4.jar:/opt/docker/lib/org.apache.hadoop.hadoop-auth-2.6.0.jar:/opt/docker/lib/org.apache.directory.server.apacheds-kerberos-codec-2.0.0-M15.jar:/opt/docker/lib/org.apache.directory.server.apacheds-i18n-2.0.0-M15.jar:/opt/docker/lib/org.apache.directory.api.api-asn1-api-1.0.0-M20.jar:/opt/docker/lib/org.apache.directory.api.api-util-1.0.0-M20.jar:/opt/docker/lib/org.apache.curator.curator-framework-2.6.0.jar:/opt/docker/lib/org.apache.curator.curator-client-2.6.0.jar:/opt/docker/lib/org.apache.zookeeper.zookeeper-3.4.6.jar:/opt/docker/lib/org.apache.curator.curator-recipes-2.6.0.jar:/opt/docker/lib/org.htrace.htrace-core-3.0.4.jar:/opt/docker/lib/org.apache.hadoop.hadoop-hdfs-2.6.0.jar:/opt/docker/lib/org.mortbay.jetty.jetty-util-6.1.26.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-app-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-common-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-common-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-api-2.6.0.jar:/opt/docker/lib/com.sun.jersey.jersey-client-1.9.jar:/opt/docker/lib/org.codehaus.jackson.jackson-jaxrs-1.9.13.jar:/opt/docker/lib/org.codehaus.jackson.jackson-xc-1.9.13.jar:/opt/docker/lib/com.google.inject.guice-3.0.jar:/opt/docker/lib/javax.inject.javax.inject-1.jar:/opt/docker/lib/aopalliance.aopalliance-1.0.jar:/opt/docker/lib/org.sonatype.sisu.inject.cglib-2.2.1-v20090111.jar:/opt/docker/lib/asm.asm-3.2.jar:/opt/docker/lib/com.sun.jersey.jersey-json-1.9.jar:/opt/docker/lib/org.codehaus.jettison.jettison-1.1.jar:/opt/docker/lib/com.sun.jersey.contribs.jersey-guice-1.9.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-client-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-core-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-server-common-2.6.0.jar:/opt/docker/lib/org.fusesource.leveldbjni.leveldbjni-all-1.8.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-shuffle-2.6.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-mapreduce-client-jobclient-2.6.0.jar:/opt/docker/lib/org.apache.spark.spark-repl_2.10-1.5.1.jar:/opt/docker/lib/org.scala-lang.jline-2.10.4.jar:/opt/docker/lib/org.fusesource.jansi.jansi-1.4.jar:/opt/docker/lib/org.apache.spark.spark-bagel_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-mllib_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-streaming_2.10-1.5.1.jar:/opt/docker/lib/org.apache.spark.spark-graphx_2.10-1.5.1.jar:/opt/docker/lib/org.scalanlp.breeze_2.10-0.11.2.jar:/opt/docker/lib/org.scalanlp.breeze-macros_2.10-0.11.2.jar:/opt/docker/lib/org.jpmml.pmml-model-1.1.15.jar:/opt/docker/lib/org.jpmml.pmml-agent-1.1.15.jar:/opt/docker/lib/org.jpmml.pmml-schema-1.1.15.jar:/opt/docker/lib/com.sun.xml.bind.jaxb-impl-2.2.7.jar:/opt/docker/lib/com.sun.xml.bind.jaxb-core-2.2.7.jar:/opt/docker/lib/javax.xml.bind.jaxb-api-2.2.7.jar:/opt/docker/lib/com.sun.istack.istack-commons-runtime-2.16.jar:/opt/docker/lib/com.sun.xml.fastinfoset.FastInfoset-1.2.12.jar:/opt/docker/lib/javax.xml.bind.jsr173_api-1.0.jar:/opt/docker/lib/org.apache.hadoop.hadoop-yarn-server-web-proxy-2.6.0.jar:/opt/docker/lib/org.mortbay.jetty.jetty-6.1.26.jar:/opt/docker/lib/io.reactivex.rxscala_2.10-0.22.0.jar:/opt/docker/lib/io.reactivex.rxjava-1.0.0-rc.5.jar:/opt/docker/lib/org.scalaz.scalaz-core_2.10-7.0.6.jar:/opt/docker/lib/org.scala-sbt.sbt-0.13.8.jar:/opt/docker/lib/org.scala-sbt.main-0.13.8.jar:/opt/docker/lib/org.scala-sbt.actions-0.13.8.jar:/opt/docker/lib/org.scala-sbt.classpath-0.13.8.jar:/opt/docker/lib/org.scala-sbt.launcher-interface-0.13.8.jar:/opt/docker/lib/org.scala-sbt.interface-0.13.8.jar:/opt/docker/lib/org.scala-sbt.io-0.13.8.jar:/opt/docker/lib/org.scala-sbt.control-0.13.8.jar:/opt/docker/lib/org.scala-sbt.completion-0.13.8.jar:/opt/docker/lib/org.scala-sbt.collections-0.13.8.jar:/opt/docker/lib/jline.jline-2.11.jar:/opt/docker/lib/org.scala-sbt.api-0.13.8.jar:/opt/docker/lib/org.scala-sbt.compiler-integration-0.13.8.jar:/opt/docker/lib/org.scala-sbt.incremental-compiler-0.13.8.jar:/opt/docker/lib/org.scala-sbt.logging-0.13.8.jar:/opt/docker/lib/org.scala-sbt.process-0.13.8.jar:/opt/docker/lib/org.scala-sbt.relation-0.13.8.jar:/opt/docker/lib/org.scala-sbt.compile-0.13.8.jar:/opt/docker/lib/org.scala-sbt.classfile-0.13.8.jar:/opt/docker/lib/org.scala-sbt.persist-0.13.8.jar:/opt/docker/lib/org.scala-tools.sbinary.sbinary_2.10-0.4.2.jar:/opt/docker/lib/org.scala-sbt.compiler-ivy-integration-0.13.8.jar:/opt/docker/lib/org.scala-sbt.ivy-0.13.8.jar:/opt/docker/lib/org.scala-sbt.cross-0.13.8.jar:/opt/docker/lib/org.scala-sbt.ivy.ivy-2.3.0-sbt-fccfbd44c9f64523b61398a0155784dcbaeae28f.jar:/opt/docker/lib/com.jcraft.jsch-0.1.46.jar:/opt/docker/lib/org.scala-sbt.serialization_2.10-0.1.1.jar:/opt/docker/lib/org.scala-lang.modules.scala-pickling_2.10-0.10.0.jar:/opt/docker/lib/org.scalamacros.quasiquotes_2.10-2.0.1.jar:/opt/docker/lib/org.spire-math.jawn-parser_2.10-0.6.0.jar:/opt/docker/lib/org.spire-math.json4s-support_2.10-0.6.0.jar:/opt/docker/lib/org.scala-sbt.run-0.13.8.jar:/opt/docker/lib/org.scala-sbt.task-system-0.13.8.jar:/opt/docker/lib/org.scala-sbt.tasks-0.13.8.jar:/opt/docker/lib/org.scala-sbt.tracking-0.13.8.jar:/opt/docker/lib/org.scala-sbt.cache-0.13.8.jar:/opt/docker/lib/org.scala-sbt.testing-0.13.8.jar:/opt/docker/lib/org.scala-sbt.test-agent-0.13.8.jar:/opt/docker/lib/org.scala-sbt.test-interface-1.0.jar:/opt/docker/lib/org.scala-sbt.main-settings-0.13.8.jar:/opt/docker/lib/org.scala-sbt.apply-macro-0.13.8.jar:/opt/docker/lib/org.scala-sbt.command-0.13.8.jar:/opt/docker/lib/org.scala-sbt.logic-0.13.8.jar:/opt/docker/lib/org.scala-sbt.compiler-interface bin-0.13.8.jar:/opt/docker/lib/org.scala-sbt.compiler-interface -src-0.13.8.jar:/opt/docker/lib/org.scala-sbt.precompiled-2_8_2-compiler-interface-bin-0.13.8.jar:/opt/docker/lib/org.scala-sbt.precompiled-2_9_2-compiler-interface-bin-0.13.8.jar:/opt/docker/lib/org.scala-sbt.precompiled-2_9_3-compiler-interface-bin-0.13.8.jar:/opt/docker/lib/com.frugalmechanic.fm-sbt-s3-resolver-0.5.0.jar:/opt/docker/lib/com.amazonaws.aws-java-sdk-s3-1.9.0.jar:/opt/docker/lib/com.amazonaws.aws-java-sdk-core-1.9.0.jar:/opt/docker/lib/joda-time.joda-time-2.9.1.jar:/opt/docker/lib/io.continuum.bokeh.bokeh_2.10-0.2.jar:/opt/docker/lib/io.continuum.bokeh.core_2.10-0.2.jar:/opt/docker/lib/com.typesafe.play.play-json_2.10-2.4.0-M1.jar:/opt/docker/lib/com.typesafe.play.play-iteratees_2.10-2.4.0-M1.jar:/opt/docker/lib/com.typesafe.play.play-functional_2.10-2.4.0-M1.jar:/opt/docker/lib/com.typesafe.play.play-datacommons_2.10-2.4.0-M1.jar:/opt/docker/lib/io.continuum.bokeh.bokehjs_2.10-0.2.jar:/opt/docker/lib/com.github.scala-incubator.io.scala-io-core_2.10-0.4.3.jar:/opt/docker/lib/com.jsuereth.scala-arm_2.10-1.3.jar:/opt/docker/lib/com.github.scala-incubator.io.scala-io-file_2.10-0.4.3.jar:/opt/docker/lib/com.quantifind.sumac_2.10-0.3.0.jar:/opt/docker/lib/com.typesafe.play.play-cache_2.10-2.3.7.jar:/opt/docker/lib/net.sf.ehcache.ehcache-core-2.6.8.jar:/opt/docker/lib/nooostab.spark-notebook-0.6.2-SNAPSHOT-scala-2.10.4-spark-1.5.1-hadoop-2.6.0-assets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/zipfs.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/icedtea-sound.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar, PATH -> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, CLICOLOR -> 1, HOME -> /root, SPARK_LOCAL_IP -> 172.17.0.2, AGENT_ADVERTISE_IP -> 172.17.0.1, LIBPROCESS_ADVERTISE_IP -> 172.17.0.1, LS_COLORS -> di=34:ln=35:so=32:pi=33:ex=1;40:bd=34;40:cd=34;40:su=0;40:sg=0;40:tw=0;40:ow=0;40:, LIBPROCESS_ADVERTISE_PORT -> 9050) [info] application - In working directory . [info] application - Registering first web-socket (notebook.server.WebSockWrapperImpl@57ae2ceb) in service notebook.server.CalcWebSocketService$CalcActor@6c072545 [info] application - Spawning calculator in service notebook.server.CalcWebSocketService$CalcActor@6c072545 [DEBUG] [11/18/2015 14:57:19.073] [main] [EventStream] StandardOutLogger started [DEBUG] [11/18/2015 14:57:19.263] [main] [EventStream(akka://Remote)] logger log1-Slf4jLogger started [DEBUG] [11/18/2015 14:57:19.265] [main] [EventStream(akka://Remote)] Default Loggers started I1118 14:57:28.938076 3879 logging.cpp:172] INFO level logging started! 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@716: Client environment:host.name=0c4fba1664ac 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@723: Client environment:os.name=Linux 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@724: Client environment:os.arch=3.19.0-30-generic 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@725: Client environment:os.version=#34~14.04.1-Ubuntu SMP Fri Oct 2 22:09:39 UTC 2015 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@733: Client environment:user.name=(null) 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@741: Client environment:user.home=/root 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@log_env@753: Client environment:user.dir=/opt/docker 2015-11-18 14:57:28,940:3857(0x7f3d18eeb700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=172.17.0.1:2181 sessionTimeout=10000 watcher=0x7f3d1f4ad600 sessionId=0 sessionPasswd=<null> context=0x7f3d80001e10 flags=0 2015-11-18 14:57:28,940:3857(0x7f3d14de2700):ZOO_INFO@check_events@1703: initiated connection to server [172.17.0.1:2181] I1118 14:57:28.940862 3956 sched.cpp:164] Version: 0.25.0 2015-11-18 14:57:28,944:3857(0x7f3d14de2700):ZOO_INFO@check_events@1750: session establishment complete on server [172.17.0.1:2181] , sessionId=0x1511a4b0ab0000b, negotiated timeout=10000 I1118 14:57:28.944319 3950 group.cpp:331] Group process (group(1)@172.17.0.1:9050) connected to ZooKeeper I1118 14:57:28.944707 3950 group.cpp:805] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I1118 14:57:28.944722 3950 group.cpp:403] Trying to create path '/mesos' in ZooKeeper I1118 14:57:28.948410 3950 detector.cpp:156] Detected a new leader: (id='1') I1118 14:57:28.948495 3952 group.cpp:674] Trying to get '/mesos/json.info_0000000001' in ZooKeeper I1118 14:57:28.948930 3951 detector.cpp:481] A new leading master (UPID=master@172.17.0.1:5050) is detected I1118 14:57:28.948982 3951 sched.cpp:262] New master detected at master@172.17.0.1:5050 I1118 14:57:28.949141 3951 sched.cpp:272] No credentials provided. Attempting to register without authentication At the mesos master i got: I1118 16:32:04.114322 11968 master.cpp:2179] Received SUBSCRIBE call for framework 'Simple Spark' at scheduler-c860460e-8878-4e92-a28e-871890ef2227@172.17.0.1:9050 I1118 16:32:04.114537 11968 master.cpp:2250] Subscribing framework Simple Spark with checkpointing disabled and capabilities [ ] E1118 16:32:04.114892 11975 socket.hpp:174] Shutdown failed on fd=28: Transport endpoint is not connected [107] I1118 16:32:04.115171 11968 master.cpp:1119] Framework 8236605c-0abd-4562-8061-0520b9448f41-0235 (Simple Spark) at scheduler- Info for spark notebook: Build: | buildTime-Mon Nov 16 14:38:21 UTC 2015 | formattedShaVersion-0.6.2-SNAPSHOT-dcacd17902968c767f98fbc659d6115e25a5f753 | sbtVersion-0.13.8 | scalaVersion-2.10.4 | sparkNotebookVersion-0.6.2-SNAPSHOT | hadoopVersion-2.6.0 | jets3tVersion-0.7.1 | jlineDef-(org.scala-lang,2.10.4) | sparkVersion-1.5.1 | withHive-false | withParquet-false |. So it seems that something is wrong with the connection from scheduler to the mesos master... whats the best setup to test it?
          Hide
          radekg Radoslaw Gruchalski added a comment -

          Stavros,

          These are the immediate I can see:

          1. Container port mappings should be:

          [
            { "containerPort": 9000, "hostPort": 0, "protocol": "tcp" },
            { "containerPort": 9050, "hostPort": 0, "protocol": "tcp" },
            { "containerPort": 6666, "hostPort": 0, "protocol": "tcp" },
            { "containerPort": 6677, "hostPort": 0, "protocol": "tcp" },
            { "containerPort": 6688, "hostPort": 0, "protocol": "tcp" },
            { "containerPort": 23456, "hostPort": 0, "protocol": "tcp" }
          ]
          

          2. Akka version - Spark 1.5.1 uses 2.3.11 (and any version of Data Fellas notebook requires 2.3.11, Andy is excluding Akka from Spark jars)

          3. LIBPROCESS_PORT & LIBPROCESS_ADVERTISE_PORT

          export LIBPROCESS_PORT=9050
          export LIBPROCESS_ADVERTISE_PORT=$PORT_9050 # to use the port given to you by Mesos
          

          4. Notebook settings:

          cd /opt/docker && ./bin/spark-notebook -Dapplication.context="$NOTEBOOK_APPLICATION_CONTEXT" \
            -Dmanager.notebooks.custom.args.0="-Dspark.driver.host=$SPARK_PUBLIC_DNS" \
            -Dmanager.notebooks.custom.args.1="-Dspark.driver.port=6666" \
            -Dmanager.notebooks.custom.args.2="-Dspark.driver.advertisedPort=$PORT_6666" \
            -Dmanager.notebooks.custom.args.3="-Dspark.replClassServer.port=23456" \
            -Dmanager.notebooks.custom.args.4="-Dspark.replClassServer.advertisedPort=$PORT_23456" \
            -Dmanager.notebooks.custom.sparkConf.spark.executor.uri=${SPARK_EXECUTOR_URI} \
            -Dmanager.notebooks.custom.sparkConf.spark.master=mesos://zk://172.17.0.1:2181/mesos \
            -Dmanager.notebooks.custom.sparkConf.spark.fileserver.port=6677 \
            -Dmanager.notebooks.custom.sparkConf.spark.fileserver.advertisedPort=$PORT_6677 \
            -Dmanager.notebooks.custom.sparkConf.spark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory \
            -Dmanager.notebooks.custom.sparkConf.spark.broadcast.port=6688 \
            -Dmanager.notebooks.custom.sparkConf.spark.broadcast.advertisedPort=$PORT_6688
          

          5. Do not add settings to the notebook metadata. No setting == good (picks up from environment).

          6. Your MESOS_JAVA_NATIVE_LIBRARY is wrong:

          MESOS_JAVA_NATIVE_LIBRARY -> /usr/local/lib/libmesos-0.22.0.so

          It should be /usr/local/lib/libmesos-0.25.0.so (Mesos was upgraded)

          It appears that most of the things are fine, your mesos master can't reach your framework. This is because, most likely (at least getting you closer):

          • wrong version of MESOS_JAVA_NATIVE_LIBRARY
          • LIBPROCESS_ADVERTISE_PORT is wrong - this should be the value of the port assigned by Mesos, which is routed to 9050 in the container

          7. Notebook context can be /, you'd then access the UI of the notebook as http://<container-ip>:<the value of $PORT_9000>, if you set the context to /mycontext, the URL becomes http://<container-ip>:<the value of $PORT_9000>/mycontext

          Finally,

          As I already mentioned, I am working on the full example. It's been quite busy for the last few days (including weekend) but it's coming shortly.
          I am also aware of AkkaRpcEnv not being the default for Spark 1.6.0, working on addressing that as well.

          Show
          radekg Radoslaw Gruchalski added a comment - Stavros, These are the immediate I can see: 1. Container port mappings should be: [ { "containerPort": 9000, "hostPort": 0, "protocol": "tcp" }, { "containerPort": 9050, "hostPort": 0, "protocol": "tcp" }, { "containerPort": 6666, "hostPort": 0, "protocol": "tcp" }, { "containerPort": 6677, "hostPort": 0, "protocol": "tcp" }, { "containerPort": 6688, "hostPort": 0, "protocol": "tcp" }, { "containerPort": 23456, "hostPort": 0, "protocol": "tcp" } ] 2. Akka version - Spark 1.5.1 uses 2.3.11 (and any version of Data Fellas notebook requires 2.3.11, Andy is excluding Akka from Spark jars) 3. LIBPROCESS_PORT & LIBPROCESS_ADVERTISE_PORT export LIBPROCESS_PORT=9050 export LIBPROCESS_ADVERTISE_PORT=$PORT_9050 # to use the port given to you by Mesos 4. Notebook settings: cd /opt/docker && ./bin/spark-notebook -Dapplication.context="$NOTEBOOK_APPLICATION_CONTEXT" \ -Dmanager.notebooks.custom.args.0="-Dspark.driver.host=$SPARK_PUBLIC_DNS" \ -Dmanager.notebooks.custom.args.1="-Dspark.driver.port=6666" \ -Dmanager.notebooks.custom.args.2="-Dspark.driver.advertisedPort=$PORT_6666" \ -Dmanager.notebooks.custom.args.3="-Dspark.replClassServer.port=23456" \ -Dmanager.notebooks.custom.args.4="-Dspark.replClassServer.advertisedPort=$PORT_23456" \ -Dmanager.notebooks.custom.sparkConf.spark.executor.uri=${SPARK_EXECUTOR_URI} \ -Dmanager.notebooks.custom.sparkConf.spark.master=mesos://zk://172.17.0.1:2181/mesos \ -Dmanager.notebooks.custom.sparkConf.spark.fileserver.port=6677 \ -Dmanager.notebooks.custom.sparkConf.spark.fileserver.advertisedPort=$PORT_6677 \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.port=6688 \ -Dmanager.notebooks.custom.sparkConf.spark.broadcast.advertisedPort=$PORT_6688 5. Do not add settings to the notebook metadata. No setting == good (picks up from environment). 6. Your MESOS_JAVA_NATIVE_LIBRARY is wrong: MESOS_JAVA_NATIVE_LIBRARY -> /usr/local/lib/libmesos-0.22.0.so It should be /usr/local/lib/libmesos-0.25.0.so (Mesos was upgraded) It appears that most of the things are fine, your mesos master can't reach your framework. This is because, most likely (at least getting you closer): wrong version of MESOS_JAVA_NATIVE_LIBRARY LIBPROCESS_ADVERTISE_PORT is wrong - this should be the value of the port assigned by Mesos, which is routed to 9050 in the container 7. Notebook context can be /, you'd then access the UI of the notebook as http://<container-ip>:<the value of $PORT_9000> , if you set the context to /mycontext , the URL becomes http://<container-ip>:<the value of $PORT_9000>/mycontext Finally, As I already mentioned, I am working on the full example. It's been quite busy for the last few days (including weekend) but it's coming shortly. I am also aware of AkkaRpcEnv not being the default for Spark 1.6.0 , working on addressing that as well.
          Hide
          skonto Stavros Kontopoulos added a comment - - edited

          My new trial...

          I have them all fixed...1 to 6 i think i have much better understanding now.

          I am not sure about the correct values of the following though:
          export SPARK_PUBLIC_DNS=$HOST
          export SPARK_LOCAL_HOSTNAME=$HOST

          Right now they get the hostname ($HOST) of the container which resolves to 172.17.0.2 is this the intention of the setup or it should be some ip of my hostmachine (this ip it gets
          its still visible from my host machine as i mentioned before)?

          Again i see the same error with no connection at the transport layer.

          This time i see the correct advertised port for the scheduler as it should appear at the mesos master logs.
          I1119 14:19:00.599664 3098 master.cpp:2179] Received SUBSCRIBE call for framework 'Simple Spark' at scheduler-5f7621d4-f7e3-42d9-a34d-91c1bfdcb632@172.17.0.1:31705
          I1119 14:19:00.599786 3098 master.cpp:2250] Subscribing framework Simple Spark with checkpointing disabled and capabilities [ ]
          E1119 14:19:00.599987 3101 socket.hpp:174] Shutdown failed on fd=32: Transport endpoint is not connected [107]

          Unfortunately from netstat: tcp6 0 0 [::]:31705 [::]:* LISTEN -

          And while i can do: telnet 0.0.0.0 31705 I cannot do telnet 172.17.0.1 31705 from my host machine.

          Some binds are on tcp6 i could tried to use export _JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true", but not so sure i should.

          Also i cannot pass the command line options for notebook metadata. I tried to set metadata to empty from the ui and it keeps creating its own on the fly and runs things with Local[*]. I had to add the metadata myself to make it run with mesos master where of course i had to put the actual values of the ports like: "-Dspark.replClassServer.advertisedPort=31709".

          Show
          skonto Stavros Kontopoulos added a comment - - edited My new trial... I have them all fixed...1 to 6 i think i have much better understanding now. I am not sure about the correct values of the following though: export SPARK_PUBLIC_DNS=$HOST export SPARK_LOCAL_HOSTNAME=$HOST Right now they get the hostname ($HOST) of the container which resolves to 172.17.0.2 is this the intention of the setup or it should be some ip of my hostmachine (this ip it gets its still visible from my host machine as i mentioned before)? Again i see the same error with no connection at the transport layer. This time i see the correct advertised port for the scheduler as it should appear at the mesos master logs. I1119 14:19:00.599664 3098 master.cpp:2179] Received SUBSCRIBE call for framework 'Simple Spark' at scheduler-5f7621d4-f7e3-42d9-a34d-91c1bfdcb632@172.17.0.1:31705 I1119 14:19:00.599786 3098 master.cpp:2250] Subscribing framework Simple Spark with checkpointing disabled and capabilities [ ] E1119 14:19:00.599987 3101 socket.hpp:174] Shutdown failed on fd=32: Transport endpoint is not connected [107] Unfortunately from netstat: tcp6 0 0 [::] :31705 [::] :* LISTEN - And while i can do: telnet 0.0.0.0 31705 I cannot do telnet 172.17.0.1 31705 from my host machine. Some binds are on tcp6 i could tried to use export _JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true", but not so sure i should. Also i cannot pass the command line options for notebook metadata. I tried to set metadata to empty from the ui and it keeps creating its own on the fly and runs things with Local [*] . I had to add the metadata myself to make it run with mesos master where of course i had to put the actual values of the ports like: "-Dspark.replClassServer.advertisedPort=31709".
          Show
          radekg Radoslaw Gruchalski added a comment - Repeatable environment for verifying the patches: https://github.com/radekg/mesos-spark-docker-bridge Details: http://gruchalski.com/dcos/apache-spark-on-mesos-with-docker-bridge-networking.html
          Hide
          skonto Stavros Kontopoulos added a comment - - edited

          I verified the example for spark version spark 1.5.1. It seems to work fine. It makes possible the bidirectional communication between mesos master/notebook within a container to the outside of it, specifically the executors running on the host machine. There are some stuff to be addressed like: checking how it affects TorrentBroadcast communication (it is tested with HttpBroadcast).
          Also enough tests should be added and spark 1.6 should be supported and tested (is this finished?).

          Show
          skonto Stavros Kontopoulos added a comment - - edited I verified the example for spark version spark 1.5.1. It seems to work fine. It makes possible the bidirectional communication between mesos master/notebook within a container to the outside of it, specifically the executors running on the host machine. There are some stuff to be addressed like: checking how it affects TorrentBroadcast communication (it is tested with HttpBroadcast). Also enough tests should be added and spark 1.6 should be supported and tested (is this finished?).
          Hide
          radekg Radoslaw Gruchalski added a comment -

          I have added a change to enable spark.driver.advertisedPort support in NettyRpcEnv: https://github.com/radekg/spark/commit/b21aae1468169ee0a388d33ba6cebdb17b895956#diff-0c89b4a60c30a7cd2224bb64d93da942R125
          Not quite sure what the impact of this change is. It would be great if somebody could cross check.

          Show
          radekg Radoslaw Gruchalski added a comment - I have added a change to enable spark.driver.advertisedPort support in NettyRpcEnv : https://github.com/radekg/spark/commit/b21aae1468169ee0a388d33ba6cebdb17b895956#diff-0c89b4a60c30a7cd2224bb64d93da942R125 Not quite sure what the impact of this change is. It would be great if somebody could cross check.
          Hide
          radekg Radoslaw Gruchalski added a comment -

          Updated 1.6.0 patch to include NettyRpcEnv.

          Show
          radekg Radoslaw Gruchalski added a comment - Updated 1.6.0 patch to include NettyRpcEnv.
          Hide
          srowen Sean Owen added a comment -

          Radoslaw Gruchalski don't set Target version – this is for committers. It doesn't make sense to target released versions anyway

          Show
          srowen Sean Owen added a comment - Radoslaw Gruchalski don't set Target version – this is for committers. It doesn't make sense to target released versions anyway
          Hide
          radekg Radoslaw Gruchalski added a comment -

          No problem, wasn't aware. "New guy" syndrome.

          Show
          radekg Radoslaw Gruchalski added a comment - No problem, wasn't aware. "New guy" syndrome.
          Hide
          skonto Stavros Kontopoulos added a comment -

          Radoslaw Gruchalski Ara you planning to work on this on master? I have some spare cycles ...

          Show
          skonto Stavros Kontopoulos added a comment - Radoslaw Gruchalski Ara you planning to work on this on master? I have some spare cycles ...
          Hide
          radekg Radoslaw Gruchalski added a comment - - edited

          Stavros Kontopoulos, yes, sounds like a good idea.
          Would it be worth discussing exactly what the outcome of this feature should be?

          Show
          radekg Radoslaw Gruchalski added a comment - - edited Stavros Kontopoulos , yes, sounds like a good idea. Would it be worth discussing exactly what the outcome of this feature should be?
          Hide
          mandoskippy John Omernik added a comment -

          Another new guy, late to the game:

          Would work done on having Spark support the new Mesos HTTP API help with this? Basically, if Spark was able to communicate without having the thick native lib, could we make things easier from a networking perspective. I could be wrong on this part As I read it, the "connect back to the scheduler" that needs a JIRA like this for docker and bridged mode, goes away when you use the HTTP API because it's all scheduler to Mesos Master communication.

          I Think

          Thus, while this is good work, working on the 1.0 Mesos APIs could actually solve this problem, and open up new possibilities... Thoughts?

          Show
          mandoskippy John Omernik added a comment - Another new guy, late to the game: Would work done on having Spark support the new Mesos HTTP API help with this? Basically, if Spark was able to communicate without having the thick native lib, could we make things easier from a networking perspective. I could be wrong on this part As I read it, the "connect back to the scheduler" that needs a JIRA like this for docker and bridged mode, goes away when you use the HTTP API because it's all scheduler to Mesos Master communication. I Think Thus, while this is good work, working on the 1.0 Mesos APIs could actually solve this problem, and open up new possibilities... Thoughts?
          Hide
          skonto Stavros Kontopoulos added a comment - - edited
          Show
          skonto Stavros Kontopoulos added a comment - - edited Radoslaw Gruchalski Michael Gummelt what do you think?
          Hide
          mgummelt Michael Gummelt added a comment -

          This JIRA is complex and a lot of it is out of date. Can someone briefly explain to me what the problem is? Why do you want bridge networking?

          Show
          mgummelt Michael Gummelt added a comment - This JIRA is complex and a lot of it is out of date. Can someone briefly explain to me what the problem is? Why do you want bridge networking?
          Hide
          radekg Radoslaw Gruchalski added a comment -

          John Omernik I have not been following Mesos API between 0.28.2 and 1.0.0. No idea what changes would make it work without having to change LIBPROCESS env in Docker bridge networking. I wouldn't call using the documented LIBPROCESS functionality "tricking". Changing LIBPROCESS settings in Docker bridge is required because your Mesos in the container is bound to a different port than what Mesos cluster outside expects. A bit like what Akka was in the original 1.4 version of Spark. The difference was that Akka could not be changed like this (until 2.4).

          Michael Gummelt We used it at my previous place for starting Zeppelin, Jupyter and Spark notebook (the data fellas one) instances in a multi tenant Mesos environment with Docker. Now, why was it done in bridge networking? I implemented it so I guess I should know. But I don't. The answer is: nobody asked "why does it need bridge in the first place". And the answer to the question nobody asked is: "it doesn't need bridge mode, everything what we've done at the previous place, in bridge, can be done with host network". The only advantage we had was using the same configuration inside of the docker container.

          Show
          radekg Radoslaw Gruchalski added a comment - John Omernik I have not been following Mesos API between 0.28.2 and 1.0.0. No idea what changes would make it work without having to change LIBPROCESS env in Docker bridge networking. I wouldn't call using the documented LIBPROCESS functionality "tricking". Changing LIBPROCESS settings in Docker bridge is required because your Mesos in the container is bound to a different port than what Mesos cluster outside expects. A bit like what Akka was in the original 1.4 version of Spark. The difference was that Akka could not be changed like this (until 2.4). Michael Gummelt We used it at my previous place for starting Zeppelin, Jupyter and Spark notebook (the data fellas one) instances in a multi tenant Mesos environment with Docker. Now, why was it done in bridge networking? I implemented it so I guess I should know. But I don't. The answer is: nobody asked "why does it need bridge in the first place". And the answer to the question nobody asked is: "it doesn't need bridge mode, everything what we've done at the previous place, in bridge, can be done with host network". The only advantage we had was using the same configuration inside of the docker container.
          Hide
          mandoskippy John Omernik added a comment -

          I hadn't tried using the LIBPROCESS settings, I need to sit down and work with it again. The issue I was running into, was when I used docker containers for Spark, Bridged Networking caused failures because (likely the LIBPROCESS issue) and host networking caused issues for me, because I was trying to use a network client for MapR FS that didn't like running on the same "host" node (networking wise) as the client in the container. Now, options are good, bridge networking is also nice from a configuration of the container standpoint.

          As to the HTTP API for Mesos, this is something that could help reduce the dependancies and size of the container. Not that it matters TOO much, but shoveling around 1.5GB images (libmesos, spark, mapr client, etc) can get annoying

          I need to find some time to try building and using LIBPROCESS

          Show
          mandoskippy John Omernik added a comment - I hadn't tried using the LIBPROCESS settings, I need to sit down and work with it again. The issue I was running into, was when I used docker containers for Spark, Bridged Networking caused failures because (likely the LIBPROCESS issue) and host networking caused issues for me, because I was trying to use a network client for MapR FS that didn't like running on the same "host" node (networking wise) as the client in the container. Now, options are good, bridge networking is also nice from a configuration of the container standpoint. As to the HTTP API for Mesos, this is something that could help reduce the dependancies and size of the container. Not that it matters TOO much, but shoveling around 1.5GB images (libmesos, spark, mapr client, etc) can get annoying I need to find some time to try building and using LIBPROCESS
          Hide
          radekg Radoslaw Gruchalski added a comment -

          John Omernik No, HTTP API won't help anything. Spark still requires Mesos for the Mesos framework. Not quite sure what exactly you would like to achieve with it.
          As for bridged networking. I can see it being useful for different multi user scenarios but there would need to be some understanding on the maintainer's side.

          Show
          radekg Radoslaw Gruchalski added a comment - John Omernik No, HTTP API won't help anything. Spark still requires Mesos for the Mesos framework. Not quite sure what exactly you would like to achieve with it. As for bridged networking. I can see it being useful for different multi user scenarios but there would need to be some understanding on the maintainer's side.
          Hide
          mandoskippy John Omernik added a comment -

          So with the HTTP API, Spark would still require the heavy libmesos in order
          to work with Mesos? Even if all things in Spark shifted to using the HTTP
          API? My understanding was that part of the reason for the HTTPAPI was to
          remove the need for libmesos?

          John

          On Mon, Aug 8, 2016 at 7:31 AM, Radoslaw Gruchalski (JIRA) <jira@apache.org>

          Show
          mandoskippy John Omernik added a comment - So with the HTTP API, Spark would still require the heavy libmesos in order to work with Mesos? Even if all things in Spark shifted to using the HTTP API? My understanding was that part of the reason for the HTTPAPI was to remove the need for libmesos? John On Mon, Aug 8, 2016 at 7:31 AM, Radoslaw Gruchalski (JIRA) <jira@apache.org>
          Hide
          radekg Radoslaw Gruchalski added a comment -

          John Omernik Yes, there my lack of knowledge regarding the API can be seen. Just read the http://events.linuxfoundation.org/sites/events/files/slides/Mesos_HTTP_API.pdf. Considering that the Mesos scheduler was changed to take advantage, might be the case. Older versions of Mesos would still require native library.

          Show
          radekg Radoslaw Gruchalski added a comment - John Omernik Yes, there my lack of knowledge regarding the API can be seen. Just read the http://events.linuxfoundation.org/sites/events/files/slides/Mesos_HTTP_API.pdf . Considering that the Mesos scheduler was changed to take advantage, might be the case. Older versions of Mesos would still require native library.
          Hide
          mgummelt Michael Gummelt added a comment -

          Radoslaw Gruchalski

          > The only advantage we had was using the same configuration inside of the docker container.

          You mean you want to run the spark driver in a docker container? Which configuration did you have to change? I can look more into this, but I need a clear "It's easier/better to do X in bridge mode than in host mode".

          > So with the HTTP API, Spark would still require the heavy libmesos in order to work with Mesos?

          No. The HTTP API will remove the libmesos dependency, which is nice. It's not an urgent priority though.

          Show
          mgummelt Michael Gummelt added a comment - Radoslaw Gruchalski > The only advantage we had was using the same configuration inside of the docker container. You mean you want to run the spark driver in a docker container? Which configuration did you have to change? I can look more into this, but I need a clear "It's easier/better to do X in bridge mode than in host mode". > So with the HTTP API, Spark would still require the heavy libmesos in order to work with Mesos? No. The HTTP API will remove the libmesos dependency, which is nice. It's not an urgent priority though.
          Hide
          rxin Reynold Xin added a comment -

          Marking this as later since the original pull request has been closed.

          Show
          rxin Reynold Xin added a comment - Marking this as later since the original pull request has been closed.
          Hide
          lfisk Liam Fisk added a comment -

          This ticket appears to be related to SPARK-4563.

          Show
          lfisk Liam Fisk added a comment - This ticket appears to be related to SPARK-4563 .
          Hide
          lfisk Liam Fisk added a comment -

          Just mirroring what I said on SPARK-4563 about the lack of support for bridged networking:

          It also makes life difficult for OSX users. Docker for Mac uses xhyve to virtualize the docker engine (https://docs.docker.com/engine/installation/mac/), and thus `--net=host` binds to the VM's network instead of the true OSX host. The SPARK_LOCAL_IP ends up as 172.17.0.2, which is not externally contactable.

          The end result is OSX users cannot containerize Spark if Spark needs to contact a mesos cluster.

          While you are unlikely to have Spark running on an OSX machine in production, the development experience is a bit painful if you have to run a separate VM with public networking.

          Show
          lfisk Liam Fisk added a comment - Just mirroring what I said on SPARK-4563 about the lack of support for bridged networking: It also makes life difficult for OSX users. Docker for Mac uses xhyve to virtualize the docker engine ( https://docs.docker.com/engine/installation/mac/ ), and thus `--net=host` binds to the VM's network instead of the true OSX host. The SPARK_LOCAL_IP ends up as 172.17.0.2, which is not externally contactable. The end result is OSX users cannot containerize Spark if Spark needs to contact a mesos cluster. While you are unlikely to have Spark running on an OSX machine in production, the development experience is a bit painful if you have to run a separate VM with public networking.

            People

            • Assignee:
              Unassigned
              Reporter:
              radekg Radoslaw Gruchalski
            • Votes:
              3 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development