Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0, 1.3.0
    • Component/s: Mesos
    • Labels:
      None

      Description

      Flink's Mesos documentation could benefit from more details how to set things up and which parameters to use.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user tillrohrmann opened a pull request:

          https://github.com/apache/flink/pull/3236

          FLINK-5494 [docs] Add more details to the Mesos documentation

          Adds more details to the Mesos documentation so that setting up a Mesos cluster and running a Flink cluster in it becomes easier.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/tillrohrmann/flink mesosDocumentation

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3236.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3236


          commit 3839413caf6f0fec5e6820e57f43d318b716e72e
          Author: Till Rohrmann <trohrmann@apache.org>
          Date: 2017-01-30T17:19:07Z

          FLINK-5494 [docs] Add more details to the Mesos documentation


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/3236 FLINK-5494 [docs] Add more details to the Mesos documentation Adds more details to the Mesos documentation so that setting up a Mesos cluster and running a Flink cluster in it becomes easier. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink mesosDocumentation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3236.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3236 commit 3839413caf6f0fec5e6820e57f43d318b716e72e Author: Till Rohrmann <trohrmann@apache.org> Date: 2017-01-30T17:19:07Z FLINK-5494 [docs] Add more details to the Mesos documentation
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tzulitai commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99061517

          — Diff: docs/setup/mesos.md —
          @@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
          In the `/bin` directory of the Flink distribution, you find two startup scripts
          which manage the Flink processes in a Mesos cluster:

          -1. mesos-appmaster.sh

          • This starts the Mesos application master which will register the Mesos
          • scheduler. It is also responsible for starting up the worker nodes.
            +1. `mesos-appmaster.sh`
            + This starts the Mesos application master which will register the Mesos scheduler.
            + It is also responsible for starting up the worker nodes.

          -2. mesos-taskmanager.sh

          • The entry point for the Mesos worker processes. You don't need to explicitly
          • execute this script. It is automatically launched by the Mesos worker node to
          • bring up a new TaskManager.
            +2. `mesos-taskmanager.sh`
            + The entry point for the Mesos worker processes.
            + You don't need to explicitly execute this script.
            + It is automatically launched by the Mesos worker node to bring up a new TaskManager.
            +
            +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process.
            +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`.
            +This value can also be defined in the `flink-conf.yaml` or passed as a Java property.
            +
            +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script.
            +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster.
            +
            +#### General configuration
            +
            +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master.
            +This also allows to specify general Flink configuration parameters.
            +For example:
            +
            + bin/mesos-appmaster.sh \
            + -Dmesos.master=master.foobar.org:5050
            + -Djobmanager.heap.mb=1024 \
            + -Djobmanager.rpc.port=6123 \
            + -Djobmanager.web.port=8081 \
            + -Dmesos.initial-tasks=10 \
            + -Dmesos.resourcemanager.tasks.mem=4096 \
            + -Dtaskmanager.heap.mb=3500 \
            + -Dtaskmanager.numberOfTaskSlots=2 \
            + -Dparallelism.default=10
              1. High Availability

          -You will need to run a service like Marathon or Apache Aurora which takes care
          -of restarting the Flink master process in case of node or process failures. In
          -addition, Zookeeper needs to be configured like described in the
          -[High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)
          +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures.
          +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)

          -For the reconciliation of tasks to work correctly, please also set
          -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
          +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.

                1. Marathon

          -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In
          -particular, it should also adjust any configuration parameters for the Flink
          -cluster.
          +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
          +In particular, it should also adjust any configuration parameters for the Flink cluster.

          Here is an example configuration for Marathon:

          { - "id": "basic-0", - "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, - "mem": 2048, + "mem": 1024 }

          +When running Flink with Marathon, the whole Flink cluster including the job manager will be run as Mesos tasks in the Mesos cluster.
          +

              1. Configuration parameters
                1. Mesos configuration entries
              • End diff –

          Should we add the Mesos config to `setup/config.md` as well?

          Show
          githubbot ASF GitHub Bot added a comment - Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99061517 — Diff: docs/setup/mesos.md — @@ -145,60 +174,75 @@ If set to 'docker', specify the image name: In the `/bin` directory of the Flink distribution, you find two startup scripts which manage the Flink processes in a Mesos cluster: -1. mesos-appmaster.sh This starts the Mesos application master which will register the Mesos scheduler. It is also responsible for starting up the worker nodes. +1. `mesos-appmaster.sh` + This starts the Mesos application master which will register the Mesos scheduler. + It is also responsible for starting up the worker nodes. -2. mesos-taskmanager.sh The entry point for the Mesos worker processes. You don't need to explicitly execute this script. It is automatically launched by the Mesos worker node to bring up a new TaskManager. +2. `mesos-taskmanager.sh` + The entry point for the Mesos worker processes. + You don't need to explicitly execute this script. + It is automatically launched by the Mesos worker node to bring up a new TaskManager. + +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`. +This value can also be defined in the `flink-conf.yaml` or passed as a Java property. + +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script. +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster. + +#### General configuration + +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master. +This also allows to specify general Flink configuration parameters. +For example: + + bin/mesos-appmaster.sh \ + -Dmesos.master=master.foobar.org:5050 + -Djobmanager.heap.mb=1024 \ + -Djobmanager.rpc.port=6123 \ + -Djobmanager.web.port=8081 \ + -Dmesos.initial-tasks=10 \ + -Dmesos.resourcemanager.tasks.mem=4096 \ + -Dtaskmanager.heap.mb=3500 \ + -Dtaskmanager.numberOfTaskSlots=2 \ + -Dparallelism.default=10 High Availability -You will need to run a service like Marathon or Apache Aurora which takes care -of restarting the Flink master process in case of node or process failures. In -addition, Zookeeper needs to be configured like described in the - [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures. +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) -For the reconciliation of tasks to work correctly, please also set -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. Marathon -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In -particular, it should also adjust any configuration parameters for the Flink -cluster. +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. +In particular, it should also adjust any configuration parameters for the Flink cluster. Here is an example configuration for Marathon: { - "id": "basic-0", - "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, - "mem": 2048, + "mem": 1024 } +When running Flink with Marathon, the whole Flink cluster including the job manager will be run as Mesos tasks in the Mesos cluster. + Configuration parameters Mesos configuration entries End diff – Should we add the Mesos config to `setup/config.md` as well?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tzulitai commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99062197

          — Diff: docs/setup/mesos.md —
          @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime:

            1. Mesos without DC/OS

          -Let's take a look at how to setup Flink on Mesos without DC/OS.
          +You can also run Mesos without DC/OS.

                1. Prerequisites
                  +### Installing Mesos

          -Please follow the
          -[instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/).
          +Please follow the [instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/getting-started/).
          +
          +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`.
          +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes).
          +
          +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory.
          +In this file, you have to define
          +
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +and it is recommended to uncommment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          +
          +
          +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory.
          +You have to configure
          +
          + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
          +
          +and uncomment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +#### Mesos Library
          +
          +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
          — End diff –

          On Mac, the native library name seems to be `libmesos.dylib`. Might be good to inform in the docs what OS the instructions is assuming, or just add a small comment about MacOS here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99062197 — Diff: docs/setup/mesos.md — @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime: Mesos without DC/OS -Let's take a look at how to setup Flink on Mesos without DC/OS. +You can also run Mesos without DC/OS. Prerequisites +### Installing Mesos -Please follow the - [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/ ). +Please follow the [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/getting-started/ ). + +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`. +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes). + +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory. +In this file, you have to define + + export MESOS_work_dir=WORK_DIRECTORY + +and it is recommended to uncommment + + export MESOS_log_dir=LOGGING_DIRECTORY + + +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory. +You have to configure + + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT + +and uncomment + + export MESOS_log_dir=LOGGING_DIRECTORY + export MESOS_work_dir=WORK_DIRECTORY + +#### Mesos Library + +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`. — End diff – On Mac, the native library name seems to be `libmesos.dylib`. Might be good to inform in the docs what OS the instructions is assuming, or just add a small comment about MacOS here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tzulitai commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99063368

          — Diff: docs/setup/mesos.md —
          @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime:

            1. Mesos without DC/OS

          -Let's take a look at how to setup Flink on Mesos without DC/OS.
          +You can also run Mesos without DC/OS.

                1. Prerequisites
                  +### Installing Mesos

          -Please follow the
          -[instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/).
          +Please follow the [instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/getting-started/).
          +
          +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`.
          +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes).
          +
          +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory.
          +In this file, you have to define
          +
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +and it is recommended to uncommment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          +
          +
          +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory.
          +You have to configure
          +
          + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
          +
          +and uncomment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +#### Mesos Library
          +
          +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
          +
          +#### Deploying Mesos
          +
          +In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`.
          +In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`.
          +More information about the deployment scripts can be found [here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
          +
          +### Installing Marathon
          +
          +Optionally, you may also [install Marathon](https://mesosphere.github.io/marathon/docs/) which will be necessary to run Flink in high availability (HA) mode.

              1. Optional dependencies
              • End diff –

          This title is a bit not synced with the other title changes: "Installing Marathon" was singled out if "Optional dependencies", even though Marathon is still optional.

          Perhaps "Installing Maration (Optional)" and change this also to "Installing a Distributed Filesystem (Optional)".

          Show
          githubbot ASF GitHub Bot added a comment - Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99063368 — Diff: docs/setup/mesos.md — @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime: Mesos without DC/OS -Let's take a look at how to setup Flink on Mesos without DC/OS. +You can also run Mesos without DC/OS. Prerequisites +### Installing Mesos -Please follow the - [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/ ). +Please follow the [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/getting-started/ ). + +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`. +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes). + +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory. +In this file, you have to define + + export MESOS_work_dir=WORK_DIRECTORY + +and it is recommended to uncommment + + export MESOS_log_dir=LOGGING_DIRECTORY + + +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory. +You have to configure + + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT + +and uncomment + + export MESOS_log_dir=LOGGING_DIRECTORY + export MESOS_work_dir=WORK_DIRECTORY + +#### Mesos Library + +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`. + +#### Deploying Mesos + +In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`. +In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`. +More information about the deployment scripts can be found [here] ( http://mesos.apache.org/documentation/latest/deploy-scripts/ ). + +### Installing Marathon + +Optionally, you may also [install Marathon] ( https://mesosphere.github.io/marathon/docs/ ) which will be necessary to run Flink in high availability (HA) mode. Optional dependencies End diff – This title is a bit not synced with the other title changes: "Installing Marathon" was singled out if "Optional dependencies", even though Marathon is still optional. Perhaps "Installing Maration (Optional)" and change this also to "Installing a Distributed Filesystem (Optional)".
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tzulitai commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99063747

          — Diff: docs/setup/mesos.md —
          @@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
          In the `/bin` directory of the Flink distribution, you find two startup scripts
          which manage the Flink processes in a Mesos cluster:

          -1. mesos-appmaster.sh

          • This starts the Mesos application master which will register the Mesos
          • scheduler. It is also responsible for starting up the worker nodes.
            +1. `mesos-appmaster.sh`
            + This starts the Mesos application master which will register the Mesos scheduler.
            + It is also responsible for starting up the worker nodes.

          -2. mesos-taskmanager.sh

          • The entry point for the Mesos worker processes. You don't need to explicitly
          • execute this script. It is automatically launched by the Mesos worker node to
          • bring up a new TaskManager.
            +2. `mesos-taskmanager.sh`
            + The entry point for the Mesos worker processes.
            + You don't need to explicitly execute this script.
            + It is automatically launched by the Mesos worker node to bring up a new TaskManager.
            +
            +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process.
            +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`.
            +This value can also be defined in the `flink-conf.yaml` or passed as a Java property.
            +
            +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script.
            +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster.
            +
            +#### General configuration
            +
            +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master.
            +This also allows to specify general Flink configuration parameters.
            +For example:
            +
            + bin/mesos-appmaster.sh \
            + -Dmesos.master=master.foobar.org:5050
            + -Djobmanager.heap.mb=1024 \
            + -Djobmanager.rpc.port=6123 \
            + -Djobmanager.web.port=8081 \
            + -Dmesos.initial-tasks=10 \
            + -Dmesos.resourcemanager.tasks.mem=4096 \
            + -Dtaskmanager.heap.mb=3500 \
            + -Dtaskmanager.numberOfTaskSlots=2 \
            + -Dparallelism.default=10
              1. High Availability

          -You will need to run a service like Marathon or Apache Aurora which takes care
          -of restarting the Flink master process in case of node or process failures. In
          -addition, Zookeeper needs to be configured like described in the
          -[High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)
          +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures.
          +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)

          -For the reconciliation of tasks to work correctly, please also set
          -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
          +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.

                1. Marathon

          -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In
          -particular, it should also adjust any configuration parameters for the Flink
          -cluster.
          +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
          +In particular, it should also adjust any configuration parameters for the Flink cluster.

          Here is an example configuration for Marathon:

          { - "id": "basic-0", - "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, - "mem": 2048, + "mem": 1024 }

          +When running Flink with Marathon, the whole Flink cluster including the job manager will be run as Mesos tasks in the Mesos cluster.
          +

              1. Configuration parameters
                1. Mesos configuration entries

          +`mesos.initial-tasks`: The initial workers to bring up when the master starts (*DEFAULT*: The number of workers specified at cluster startup).

          -`mesos.initial-tasks`: The initial workers to bring up when the master

          • starts (*DEFAULT*: The number of workers specified at cluster startup).
            -
            -`mesos.maximum-failed-tasks`: The maximum number of failed workers before
          • the cluster fails (*DEFAULT*: Number of initial workers) May be set to -1
          • to disable this feature.
            +`mesos.maximum-failed-tasks`: The maximum number of failed workers before the cluster fails (*DEFAULT*: Number of initial workers).
            +May be set to -1 to disable this feature.

          -`mesos.master`: The Mesos master URL. The value should be in one of the

          • following forms: host:port, zk://host1:port1,host2:port2,.../path,
          • zk://username:password@host1:port1,host2:port2,.../path,
          • file:///path/to/file (where file contains one of the above)
            +`mesos.master`: The Mesos master URL. The value should be in one of the following forms: host:port, zk://host1:port1,host2:port2,.../path, zk://username:password@host1:port1,host2:port2,.../path, file:///path/to/file (where file contains one of the above)
              • End diff –

          I find this part a bit hard to read on the compiled docs. Perhaps use a bulleted list to show the different formats?

          Show
          githubbot ASF GitHub Bot added a comment - Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99063747 — Diff: docs/setup/mesos.md — @@ -145,60 +174,75 @@ If set to 'docker', specify the image name: In the `/bin` directory of the Flink distribution, you find two startup scripts which manage the Flink processes in a Mesos cluster: -1. mesos-appmaster.sh This starts the Mesos application master which will register the Mesos scheduler. It is also responsible for starting up the worker nodes. +1. `mesos-appmaster.sh` + This starts the Mesos application master which will register the Mesos scheduler. + It is also responsible for starting up the worker nodes. -2. mesos-taskmanager.sh The entry point for the Mesos worker processes. You don't need to explicitly execute this script. It is automatically launched by the Mesos worker node to bring up a new TaskManager. +2. `mesos-taskmanager.sh` + The entry point for the Mesos worker processes. + You don't need to explicitly execute this script. + It is automatically launched by the Mesos worker node to bring up a new TaskManager. + +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`. +This value can also be defined in the `flink-conf.yaml` or passed as a Java property. + +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script. +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster. + +#### General configuration + +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master. +This also allows to specify general Flink configuration parameters. +For example: + + bin/mesos-appmaster.sh \ + -Dmesos.master=master.foobar.org:5050 + -Djobmanager.heap.mb=1024 \ + -Djobmanager.rpc.port=6123 \ + -Djobmanager.web.port=8081 \ + -Dmesos.initial-tasks=10 \ + -Dmesos.resourcemanager.tasks.mem=4096 \ + -Dtaskmanager.heap.mb=3500 \ + -Dtaskmanager.numberOfTaskSlots=2 \ + -Dparallelism.default=10 High Availability -You will need to run a service like Marathon or Apache Aurora which takes care -of restarting the Flink master process in case of node or process failures. In -addition, Zookeeper needs to be configured like described in the - [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures. +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) -For the reconciliation of tasks to work correctly, please also set -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. Marathon -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In -particular, it should also adjust any configuration parameters for the Flink -cluster. +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. +In particular, it should also adjust any configuration parameters for the Flink cluster. Here is an example configuration for Marathon: { - "id": "basic-0", - "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, - "mem": 2048, + "mem": 1024 } +When running Flink with Marathon, the whole Flink cluster including the job manager will be run as Mesos tasks in the Mesos cluster. + Configuration parameters Mesos configuration entries +`mesos.initial-tasks`: The initial workers to bring up when the master starts (* DEFAULT *: The number of workers specified at cluster startup). -`mesos.initial-tasks`: The initial workers to bring up when the master starts (* DEFAULT *: The number of workers specified at cluster startup). - -`mesos.maximum-failed-tasks`: The maximum number of failed workers before the cluster fails (* DEFAULT *: Number of initial workers) May be set to -1 to disable this feature. +`mesos.maximum-failed-tasks`: The maximum number of failed workers before the cluster fails (* DEFAULT *: Number of initial workers). +May be set to -1 to disable this feature. -`mesos.master`: The Mesos master URL. The value should be in one of the following forms: host:port, zk://host1:port1,host2:port2,.../path, zk://username:password@host1:port1,host2:port2,.../path, file:///path/to/file (where file contains one of the above) +`mesos.master`: The Mesos master URL. The value should be in one of the following forms: host:port, zk://host1:port1,host2:port2,.../path, zk://username:password@host1:port1,host2:port2,.../path, file:///path/to/file (where file contains one of the above) End diff – I find this part a bit hard to read on the compiled docs. Perhaps use a bulleted list to show the different formats?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tzulitai commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99061454

          — Diff: docs/setup/mesos.md —
          @@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
          In the `/bin` directory of the Flink distribution, you find two startup scripts
          which manage the Flink processes in a Mesos cluster:

          -1. mesos-appmaster.sh

          • This starts the Mesos application master which will register the Mesos
          • scheduler. It is also responsible for starting up the worker nodes.
            +1. `mesos-appmaster.sh`
            + This starts the Mesos application master which will register the Mesos scheduler.
            + It is also responsible for starting up the worker nodes.

          -2. mesos-taskmanager.sh

          • The entry point for the Mesos worker processes. You don't need to explicitly
          • execute this script. It is automatically launched by the Mesos worker node to
          • bring up a new TaskManager.
            +2. `mesos-taskmanager.sh`
            + The entry point for the Mesos worker processes.
            + You don't need to explicitly execute this script.
            + It is automatically launched by the Mesos worker node to bring up a new TaskManager.
            +
            +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process.
            +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`.
            +This value can also be defined in the `flink-conf.yaml` or passed as a Java property.
            +
            +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script.
            +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster.
            +
            +#### General configuration
            +
            +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master.
            +This also allows to specify general Flink configuration parameters.
            +For example:
            +
            + bin/mesos-appmaster.sh \
            + -Dmesos.master=master.foobar.org:5050
            + -Djobmanager.heap.mb=1024 \
            + -Djobmanager.rpc.port=6123 \
            + -Djobmanager.web.port=8081 \
            + -Dmesos.initial-tasks=10 \
            + -Dmesos.resourcemanager.tasks.mem=4096 \
            + -Dtaskmanager.heap.mb=3500 \
            + -Dtaskmanager.numberOfTaskSlots=2 \
            + -Dparallelism.default=10
              1. High Availability

          -You will need to run a service like Marathon or Apache Aurora which takes care
          -of restarting the Flink master process in case of node or process failures. In
          -addition, Zookeeper needs to be configured like described in the
          -[High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)
          +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures.
          +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)

          -For the reconciliation of tasks to work correctly, please also set
          -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
          +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.

                1. Marathon

          -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In
          -particular, it should also adjust any configuration parameters for the Flink
          -cluster.
          +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
          +In particular, it should also adjust any configuration parameters for the Flink cluster.

          Here is an example configuration for Marathon:

          {

          • "id": "basic-0",
          • "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...",
            + "id": "flink",
            + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
            "cpus": 1.0,
          • "mem": 2048,
            + "mem": 1024
              • End diff –

          The keys in this configuration example is missing indentation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99061454 — Diff: docs/setup/mesos.md — @@ -145,60 +174,75 @@ If set to 'docker', specify the image name: In the `/bin` directory of the Flink distribution, you find two startup scripts which manage the Flink processes in a Mesos cluster: -1. mesos-appmaster.sh This starts the Mesos application master which will register the Mesos scheduler. It is also responsible for starting up the worker nodes. +1. `mesos-appmaster.sh` + This starts the Mesos application master which will register the Mesos scheduler. + It is also responsible for starting up the worker nodes. -2. mesos-taskmanager.sh The entry point for the Mesos worker processes. You don't need to explicitly execute this script. It is automatically launched by the Mesos worker node to bring up a new TaskManager. +2. `mesos-taskmanager.sh` + The entry point for the Mesos worker processes. + You don't need to explicitly execute this script. + It is automatically launched by the Mesos worker node to bring up a new TaskManager. + +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`. +This value can also be defined in the `flink-conf.yaml` or passed as a Java property. + +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script. +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster. + +#### General configuration + +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master. +This also allows to specify general Flink configuration parameters. +For example: + + bin/mesos-appmaster.sh \ + -Dmesos.master=master.foobar.org:5050 + -Djobmanager.heap.mb=1024 \ + -Djobmanager.rpc.port=6123 \ + -Djobmanager.web.port=8081 \ + -Dmesos.initial-tasks=10 \ + -Dmesos.resourcemanager.tasks.mem=4096 \ + -Dtaskmanager.heap.mb=3500 \ + -Dtaskmanager.numberOfTaskSlots=2 \ + -Dparallelism.default=10 High Availability -You will need to run a service like Marathon or Apache Aurora which takes care -of restarting the Flink master process in case of node or process failures. In -addition, Zookeeper needs to be configured like described in the - [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures. +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) -For the reconciliation of tasks to work correctly, please also set -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. Marathon -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In -particular, it should also adjust any configuration parameters for the Flink -cluster. +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. +In particular, it should also adjust any configuration parameters for the Flink cluster. Here is an example configuration for Marathon: { "id": "basic-0", "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, "mem": 2048, + "mem": 1024 End diff – The keys in this configuration example is missing indentation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tzulitai commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99063127

          — Diff: docs/setup/mesos.md —
          @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime:

            1. Mesos without DC/OS

          -Let's take a look at how to setup Flink on Mesos without DC/OS.
          +You can also run Mesos without DC/OS.

                1. Prerequisites
                  +### Installing Mesos

          -Please follow the
          -[instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/).
          +Please follow the [instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/getting-started/).
          +
          +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`.
          +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes).
          +
          +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory.
          +In this file, you have to define
          +
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +and it is recommended to uncommment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          +
          +
          +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory.
          +You have to configure
          +
          + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
          +
          +and uncomment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +#### Mesos Library
          +
          +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
          +
          +#### Deploying Mesos
          +
          +In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`.
          +In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`.
          +More information about the deployment scripts can be found [here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
          +
          +### Installing Marathon
          +
          +Optionally, you may also [install Marathon](https://mesosphere.github.io/marathon/docs/) which will be necessary to run Flink in high availability (HA) mode.

              1. Optional dependencies

          -Optionally,
          -you may also install [Marathon](https://mesosphere.github.io/marathon/) which
          -will be necessary if you want your Flink cluster to be highly available in the
          -presence of master node failures. Additionally, you probably want to install a
          -distributed file system to share data across nodes and make use of Flink's
          -checkpointing mechanism.
          +Additionally, you probably want to install a distributed file system to share data across nodes and make use of Flink's checkpointing mechanism.
          — End diff –

          I bumped a bit on this sentence. The relationship between a distributed file system and Flink's checkpointing is a bit unclear here. Perhaps change to something like "use filesystem backend for Flink's checkpointing", and also link to the related documents.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99063127 — Diff: docs/setup/mesos.md — @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime: Mesos without DC/OS -Let's take a look at how to setup Flink on Mesos without DC/OS. +You can also run Mesos without DC/OS. Prerequisites +### Installing Mesos -Please follow the - [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/ ). +Please follow the [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/getting-started/ ). + +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`. +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes). + +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory. +In this file, you have to define + + export MESOS_work_dir=WORK_DIRECTORY + +and it is recommended to uncommment + + export MESOS_log_dir=LOGGING_DIRECTORY + + +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory. +You have to configure + + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT + +and uncomment + + export MESOS_log_dir=LOGGING_DIRECTORY + export MESOS_work_dir=WORK_DIRECTORY + +#### Mesos Library + +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`. + +#### Deploying Mesos + +In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`. +In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`. +More information about the deployment scripts can be found [here] ( http://mesos.apache.org/documentation/latest/deploy-scripts/ ). + +### Installing Marathon + +Optionally, you may also [install Marathon] ( https://mesosphere.github.io/marathon/docs/ ) which will be necessary to run Flink in high availability (HA) mode. Optional dependencies -Optionally, -you may also install [Marathon] ( https://mesosphere.github.io/marathon/ ) which -will be necessary if you want your Flink cluster to be highly available in the -presence of master node failures. Additionally, you probably want to install a -distributed file system to share data across nodes and make use of Flink's -checkpointing mechanism. +Additionally, you probably want to install a distributed file system to share data across nodes and make use of Flink's checkpointing mechanism. — End diff – I bumped a bit on this sentence. The relationship between a distributed file system and Flink's checkpointing is a bit unclear here. Perhaps change to something like "use filesystem backend for Flink's checkpointing", and also link to the related documents.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99219038

          — Diff: docs/setup/mesos.md —
          @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime:

            1. Mesos without DC/OS

          -Let's take a look at how to setup Flink on Mesos without DC/OS.
          +You can also run Mesos without DC/OS.

                1. Prerequisites
                  +### Installing Mesos

          -Please follow the
          -[instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/).
          +Please follow the [instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/getting-started/).
          +
          +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`.
          +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes).
          +
          +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory.
          +In this file, you have to define
          +
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +and it is recommended to uncommment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          +
          +
          +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory.
          +You have to configure
          +
          + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
          +
          +and uncomment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +#### Mesos Library
          +
          +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
          — End diff –

          Good idea. Will add this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99219038 — Diff: docs/setup/mesos.md — @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime: Mesos without DC/OS -Let's take a look at how to setup Flink on Mesos without DC/OS. +You can also run Mesos without DC/OS. Prerequisites +### Installing Mesos -Please follow the - [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/ ). +Please follow the [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/getting-started/ ). + +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`. +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes). + +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory. +In this file, you have to define + + export MESOS_work_dir=WORK_DIRECTORY + +and it is recommended to uncommment + + export MESOS_log_dir=LOGGING_DIRECTORY + + +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory. +You have to configure + + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT + +and uncomment + + export MESOS_log_dir=LOGGING_DIRECTORY + export MESOS_work_dir=WORK_DIRECTORY + +#### Mesos Library + +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`. — End diff – Good idea. Will add this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99220004

          — Diff: docs/setup/mesos.md —
          @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime:

            1. Mesos without DC/OS

          -Let's take a look at how to setup Flink on Mesos without DC/OS.
          +You can also run Mesos without DC/OS.

                1. Prerequisites
                  +### Installing Mesos

          -Please follow the
          -[instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/).
          +Please follow the [instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/getting-started/).
          +
          +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`.
          +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes).
          +
          +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory.
          +In this file, you have to define
          +
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +and it is recommended to uncommment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          +
          +
          +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory.
          +You have to configure
          +
          + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
          +
          +and uncomment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +#### Mesos Library
          +
          +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
          +
          +#### Deploying Mesos
          +
          +In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`.
          +In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`.
          +More information about the deployment scripts can be found [here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
          +
          +### Installing Marathon
          +
          +Optionally, you may also [install Marathon](https://mesosphere.github.io/marathon/docs/) which will be necessary to run Flink in high availability (HA) mode.

              1. Optional dependencies
              • End diff –

          Will change it that way.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99220004 — Diff: docs/setup/mesos.md — @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime: Mesos without DC/OS -Let's take a look at how to setup Flink on Mesos without DC/OS. +You can also run Mesos without DC/OS. Prerequisites +### Installing Mesos -Please follow the - [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/ ). +Please follow the [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/getting-started/ ). + +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`. +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes). + +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory. +In this file, you have to define + + export MESOS_work_dir=WORK_DIRECTORY + +and it is recommended to uncommment + + export MESOS_log_dir=LOGGING_DIRECTORY + + +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory. +You have to configure + + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT + +and uncomment + + export MESOS_log_dir=LOGGING_DIRECTORY + export MESOS_work_dir=WORK_DIRECTORY + +#### Mesos Library + +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`. + +#### Deploying Mesos + +In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`. +In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`. +More information about the deployment scripts can be found [here] ( http://mesos.apache.org/documentation/latest/deploy-scripts/ ). + +### Installing Marathon + +Optionally, you may also [install Marathon] ( https://mesosphere.github.io/marathon/docs/ ) which will be necessary to run Flink in high availability (HA) mode. Optional dependencies End diff – Will change it that way.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99220223

          — Diff: docs/setup/mesos.md —
          @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime:

            1. Mesos without DC/OS

          -Let's take a look at how to setup Flink on Mesos without DC/OS.
          +You can also run Mesos without DC/OS.

                1. Prerequisites
                  +### Installing Mesos

          -Please follow the
          -[instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/).
          +Please follow the [instructions on how to setup Mesos on the official website](http://mesos.apache.org/documentation/latest/getting-started/).
          +
          +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`.
          +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes).
          +
          +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory.
          +In this file, you have to define
          +
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +and it is recommended to uncommment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          +
          +
          +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory.
          +You have to configure
          +
          + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
          +
          +and uncomment
          +
          + export MESOS_log_dir=LOGGING_DIRECTORY
          + export MESOS_work_dir=WORK_DIRECTORY
          +
          +#### Mesos Library
          +
          +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
          +
          +#### Deploying Mesos
          +
          +In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`.
          +In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`.
          +More information about the deployment scripts can be found [here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
          +
          +### Installing Marathon
          +
          +Optionally, you may also [install Marathon](https://mesosphere.github.io/marathon/docs/) which will be necessary to run Flink in high availability (HA) mode.

              1. Optional dependencies

          -Optionally,
          -you may also install [Marathon](https://mesosphere.github.io/marathon/) which
          -will be necessary if you want your Flink cluster to be highly available in the
          -presence of master node failures. Additionally, you probably want to install a
          -distributed file system to share data across nodes and make use of Flink's
          -checkpointing mechanism.
          +Additionally, you probably want to install a distributed file system to share data across nodes and make use of Flink's checkpointing mechanism.
          — End diff –

          Have removed the sentence.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99220223 — Diff: docs/setup/mesos.md — @@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the meantime: Mesos without DC/OS -Let's take a look at how to setup Flink on Mesos without DC/OS. +You can also run Mesos without DC/OS. Prerequisites +### Installing Mesos -Please follow the - [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/ ). +Please follow the [instructions on how to setup Mesos on the official website] ( http://mesos.apache.org/documentation/latest/getting-started/ ). + +After installation you have to configure the set of master and agent nodes by creating the files `MESOS_HOME/etc/mesos/masters` and `MESOS_HOME/etc/mesos/slaves`. +These files contain in each row a single hostname on which the respective component will be started (assuming SSH access to these nodes). + +Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use the template found in the same directory. +In this file, you have to define + + export MESOS_work_dir=WORK_DIRECTORY + +and it is recommended to uncommment + + export MESOS_log_dir=LOGGING_DIRECTORY + + +In order to configure the Mesos agents, you have to create `MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same directory. +You have to configure + + export MESOS_master=MASTER_HOSTNAME:MASTER_PORT + +and uncomment + + export MESOS_log_dir=LOGGING_DIRECTORY + export MESOS_work_dir=WORK_DIRECTORY + +#### Mesos Library + +In order to run Java applications with Mesos you have to export `MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`. + +#### Deploying Mesos + +In order to start your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-start-cluster.sh`. +In order to stop your mesos cluster, use the deployment script `MESOS_HOME/sbin/mesos-stop-cluster.sh`. +More information about the deployment scripts can be found [here] ( http://mesos.apache.org/documentation/latest/deploy-scripts/ ). + +### Installing Marathon + +Optionally, you may also [install Marathon] ( https://mesosphere.github.io/marathon/docs/ ) which will be necessary to run Flink in high availability (HA) mode. Optional dependencies -Optionally, -you may also install [Marathon] ( https://mesosphere.github.io/marathon/ ) which -will be necessary if you want your Flink cluster to be highly available in the -presence of master node failures. Additionally, you probably want to install a -distributed file system to share data across nodes and make use of Flink's -checkpointing mechanism. +Additionally, you probably want to install a distributed file system to share data across nodes and make use of Flink's checkpointing mechanism. — End diff – Have removed the sentence.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99220661

          — Diff: docs/setup/mesos.md —
          @@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
          In the `/bin` directory of the Flink distribution, you find two startup scripts
          which manage the Flink processes in a Mesos cluster:

          -1. mesos-appmaster.sh

          • This starts the Mesos application master which will register the Mesos
          • scheduler. It is also responsible for starting up the worker nodes.
            +1. `mesos-appmaster.sh`
            + This starts the Mesos application master which will register the Mesos scheduler.
            + It is also responsible for starting up the worker nodes.

          -2. mesos-taskmanager.sh

          • The entry point for the Mesos worker processes. You don't need to explicitly
          • execute this script. It is automatically launched by the Mesos worker node to
          • bring up a new TaskManager.
            +2. `mesos-taskmanager.sh`
            + The entry point for the Mesos worker processes.
            + You don't need to explicitly execute this script.
            + It is automatically launched by the Mesos worker node to bring up a new TaskManager.
            +
            +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process.
            +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`.
            +This value can also be defined in the `flink-conf.yaml` or passed as a Java property.
            +
            +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script.
            +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster.
            +
            +#### General configuration
            +
            +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master.
            +This also allows to specify general Flink configuration parameters.
            +For example:
            +
            + bin/mesos-appmaster.sh \
            + -Dmesos.master=master.foobar.org:5050
            + -Djobmanager.heap.mb=1024 \
            + -Djobmanager.rpc.port=6123 \
            + -Djobmanager.web.port=8081 \
            + -Dmesos.initial-tasks=10 \
            + -Dmesos.resourcemanager.tasks.mem=4096 \
            + -Dtaskmanager.heap.mb=3500 \
            + -Dtaskmanager.numberOfTaskSlots=2 \
            + -Dparallelism.default=10
              1. High Availability

          -You will need to run a service like Marathon or Apache Aurora which takes care
          -of restarting the Flink master process in case of node or process failures. In
          -addition, Zookeeper needs to be configured like described in the
          -[High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)
          +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures.
          +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)

          -For the reconciliation of tasks to work correctly, please also set
          -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
          +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.

                1. Marathon

          -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In
          -particular, it should also adjust any configuration parameters for the Flink
          -cluster.
          +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
          +In particular, it should also adjust any configuration parameters for the Flink cluster.

          Here is an example configuration for Marathon:

          {

          • "id": "basic-0",
          • "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...",
            + "id": "flink",
            + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1",
            "cpus": 1.0,
          • "mem": 2048,
            + "mem": 1024
              • End diff –

          Added indentation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99220661 — Diff: docs/setup/mesos.md — @@ -145,60 +174,75 @@ If set to 'docker', specify the image name: In the `/bin` directory of the Flink distribution, you find two startup scripts which manage the Flink processes in a Mesos cluster: -1. mesos-appmaster.sh This starts the Mesos application master which will register the Mesos scheduler. It is also responsible for starting up the worker nodes. +1. `mesos-appmaster.sh` + This starts the Mesos application master which will register the Mesos scheduler. + It is also responsible for starting up the worker nodes. -2. mesos-taskmanager.sh The entry point for the Mesos worker processes. You don't need to explicitly execute this script. It is automatically launched by the Mesos worker node to bring up a new TaskManager. +2. `mesos-taskmanager.sh` + The entry point for the Mesos worker processes. + You don't need to explicitly execute this script. + It is automatically launched by the Mesos worker node to bring up a new TaskManager. + +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`. +This value can also be defined in the `flink-conf.yaml` or passed as a Java property. + +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script. +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster. + +#### General configuration + +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master. +This also allows to specify general Flink configuration parameters. +For example: + + bin/mesos-appmaster.sh \ + -Dmesos.master=master.foobar.org:5050 + -Djobmanager.heap.mb=1024 \ + -Djobmanager.rpc.port=6123 \ + -Djobmanager.web.port=8081 \ + -Dmesos.initial-tasks=10 \ + -Dmesos.resourcemanager.tasks.mem=4096 \ + -Dtaskmanager.heap.mb=3500 \ + -Dtaskmanager.numberOfTaskSlots=2 \ + -Dparallelism.default=10 High Availability -You will need to run a service like Marathon or Apache Aurora which takes care -of restarting the Flink master process in case of node or process failures. In -addition, Zookeeper needs to be configured like described in the - [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures. +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) -For the reconciliation of tasks to work correctly, please also set -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. Marathon -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In -particular, it should also adjust any configuration parameters for the Flink -cluster. +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. +In particular, it should also adjust any configuration parameters for the Flink cluster. Here is an example configuration for Marathon: { "id": "basic-0", "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, "mem": 2048, + "mem": 1024 End diff – Added indentation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99221601

          — Diff: docs/setup/mesos.md —
          @@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
          In the `/bin` directory of the Flink distribution, you find two startup scripts
          which manage the Flink processes in a Mesos cluster:

          -1. mesos-appmaster.sh

          • This starts the Mesos application master which will register the Mesos
          • scheduler. It is also responsible for starting up the worker nodes.
            +1. `mesos-appmaster.sh`
            + This starts the Mesos application master which will register the Mesos scheduler.
            + It is also responsible for starting up the worker nodes.

          -2. mesos-taskmanager.sh

          • The entry point for the Mesos worker processes. You don't need to explicitly
          • execute this script. It is automatically launched by the Mesos worker node to
          • bring up a new TaskManager.
            +2. `mesos-taskmanager.sh`
            + The entry point for the Mesos worker processes.
            + You don't need to explicitly execute this script.
            + It is automatically launched by the Mesos worker node to bring up a new TaskManager.
            +
            +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process.
            +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`.
            +This value can also be defined in the `flink-conf.yaml` or passed as a Java property.
            +
            +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script.
            +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster.
            +
            +#### General configuration
            +
            +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master.
            +This also allows to specify general Flink configuration parameters.
            +For example:
            +
            + bin/mesos-appmaster.sh \
            + -Dmesos.master=master.foobar.org:5050
            + -Djobmanager.heap.mb=1024 \
            + -Djobmanager.rpc.port=6123 \
            + -Djobmanager.web.port=8081 \
            + -Dmesos.initial-tasks=10 \
            + -Dmesos.resourcemanager.tasks.mem=4096 \
            + -Dtaskmanager.heap.mb=3500 \
            + -Dtaskmanager.numberOfTaskSlots=2 \
            + -Dparallelism.default=10
              1. High Availability

          -You will need to run a service like Marathon or Apache Aurora which takes care
          -of restarting the Flink master process in case of node or process failures. In
          -addition, Zookeeper needs to be configured like described in the
          -[High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)
          +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures.
          +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)

          -For the reconciliation of tasks to work correctly, please also set
          -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
          +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.

                1. Marathon

          -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In
          -particular, it should also adjust any configuration parameters for the Flink
          -cluster.
          +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
          +In particular, it should also adjust any configuration parameters for the Flink cluster.

          Here is an example configuration for Marathon:

          { - "id": "basic-0", - "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, - "mem": 2048, + "mem": 1024 }

          +When running Flink with Marathon, the whole Flink cluster including the job manager will be run as Mesos tasks in the Mesos cluster.
          +

              1. Configuration parameters
                1. Mesos configuration entries
              • End diff –

          Yes they should also go there.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99221601 — Diff: docs/setup/mesos.md — @@ -145,60 +174,75 @@ If set to 'docker', specify the image name: In the `/bin` directory of the Flink distribution, you find two startup scripts which manage the Flink processes in a Mesos cluster: -1. mesos-appmaster.sh This starts the Mesos application master which will register the Mesos scheduler. It is also responsible for starting up the worker nodes. +1. `mesos-appmaster.sh` + This starts the Mesos application master which will register the Mesos scheduler. + It is also responsible for starting up the worker nodes. -2. mesos-taskmanager.sh The entry point for the Mesos worker processes. You don't need to explicitly execute this script. It is automatically launched by the Mesos worker node to bring up a new TaskManager. +2. `mesos-taskmanager.sh` + The entry point for the Mesos worker processes. + You don't need to explicitly execute this script. + It is automatically launched by the Mesos worker node to bring up a new TaskManager. + +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`. +This value can also be defined in the `flink-conf.yaml` or passed as a Java property. + +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script. +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster. + +#### General configuration + +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master. +This also allows to specify general Flink configuration parameters. +For example: + + bin/mesos-appmaster.sh \ + -Dmesos.master=master.foobar.org:5050 + -Djobmanager.heap.mb=1024 \ + -Djobmanager.rpc.port=6123 \ + -Djobmanager.web.port=8081 \ + -Dmesos.initial-tasks=10 \ + -Dmesos.resourcemanager.tasks.mem=4096 \ + -Dtaskmanager.heap.mb=3500 \ + -Dtaskmanager.numberOfTaskSlots=2 \ + -Dparallelism.default=10 High Availability -You will need to run a service like Marathon or Apache Aurora which takes care -of restarting the Flink master process in case of node or process failures. In -addition, Zookeeper needs to be configured like described in the - [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures. +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) -For the reconciliation of tasks to work correctly, please also set -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. Marathon -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In -particular, it should also adjust any configuration parameters for the Flink -cluster. +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. +In particular, it should also adjust any configuration parameters for the Flink cluster. Here is an example configuration for Marathon: { - "id": "basic-0", - "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, - "mem": 2048, + "mem": 1024 } +When running Flink with Marathon, the whole Flink cluster including the job manager will be run as Mesos tasks in the Mesos cluster. + Configuration parameters Mesos configuration entries End diff – Yes they should also go there.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3236#discussion_r99223588

          — Diff: docs/setup/mesos.md —
          @@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
          In the `/bin` directory of the Flink distribution, you find two startup scripts
          which manage the Flink processes in a Mesos cluster:

          -1. mesos-appmaster.sh

          • This starts the Mesos application master which will register the Mesos
          • scheduler. It is also responsible for starting up the worker nodes.
            +1. `mesos-appmaster.sh`
            + This starts the Mesos application master which will register the Mesos scheduler.
            + It is also responsible for starting up the worker nodes.

          -2. mesos-taskmanager.sh

          • The entry point for the Mesos worker processes. You don't need to explicitly
          • execute this script. It is automatically launched by the Mesos worker node to
          • bring up a new TaskManager.
            +2. `mesos-taskmanager.sh`
            + The entry point for the Mesos worker processes.
            + You don't need to explicitly execute this script.
            + It is automatically launched by the Mesos worker node to bring up a new TaskManager.
            +
            +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process.
            +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`.
            +This value can also be defined in the `flink-conf.yaml` or passed as a Java property.
            +
            +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script.
            +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster.
            +
            +#### General configuration
            +
            +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master.
            +This also allows to specify general Flink configuration parameters.
            +For example:
            +
            + bin/mesos-appmaster.sh \
            + -Dmesos.master=master.foobar.org:5050
            + -Djobmanager.heap.mb=1024 \
            + -Djobmanager.rpc.port=6123 \
            + -Djobmanager.web.port=8081 \
            + -Dmesos.initial-tasks=10 \
            + -Dmesos.resourcemanager.tasks.mem=4096 \
            + -Dtaskmanager.heap.mb=3500 \
            + -Dtaskmanager.numberOfTaskSlots=2 \
            + -Dparallelism.default=10
              1. High Availability

          -You will need to run a service like Marathon or Apache Aurora which takes care
          -of restarting the Flink master process in case of node or process failures. In
          -addition, Zookeeper needs to be configured like described in the
          -[High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)
          +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures.
          +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs]({{ site.baseurl }}/setup/jobmanager_high_availability.html)

          -For the reconciliation of tasks to work correctly, please also set
          -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
          +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.

                1. Marathon

          -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In
          -particular, it should also adjust any configuration parameters for the Flink
          -cluster.
          +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
          +In particular, it should also adjust any configuration parameters for the Flink cluster.

          Here is an example configuration for Marathon:

          { - "id": "basic-0", - "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, - "mem": 2048, + "mem": 1024 }

          +When running Flink with Marathon, the whole Flink cluster including the job manager will be run as Mesos tasks in the Mesos cluster.
          +

              1. Configuration parameters
                1. Mesos configuration entries

          +`mesos.initial-tasks`: The initial workers to bring up when the master starts (*DEFAULT*: The number of workers specified at cluster startup).

          -`mesos.initial-tasks`: The initial workers to bring up when the master

          • starts (*DEFAULT*: The number of workers specified at cluster startup).
            -
            -`mesos.maximum-failed-tasks`: The maximum number of failed workers before
          • the cluster fails (*DEFAULT*: Number of initial workers) May be set to -1
          • to disable this feature.
            +`mesos.maximum-failed-tasks`: The maximum number of failed workers before the cluster fails (*DEFAULT*: Number of initial workers).
            +May be set to -1 to disable this feature.

          -`mesos.master`: The Mesos master URL. The value should be in one of the

          • following forms: host:port, zk://host1:port1,host2:port2,.../path,
          • zk://username:password@host1:port1,host2:port2,.../path,
          • file:///path/to/file (where file contains one of the above)
            +`mesos.master`: The Mesos master URL. The value should be in one of the following forms: host:port, zk://host1:port1,host2:port2,.../path, zk://username:password@host1:port1,host2:port2,.../path, file:///path/to/file (where file contains one of the above)
              • End diff –

          Yes that is a good idea. Will change it accordingly.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/3236#discussion_r99223588 — Diff: docs/setup/mesos.md — @@ -145,60 +174,75 @@ If set to 'docker', specify the image name: In the `/bin` directory of the Flink distribution, you find two startup scripts which manage the Flink processes in a Mesos cluster: -1. mesos-appmaster.sh This starts the Mesos application master which will register the Mesos scheduler. It is also responsible for starting up the worker nodes. +1. `mesos-appmaster.sh` + This starts the Mesos application master which will register the Mesos scheduler. + It is also responsible for starting up the worker nodes. -2. mesos-taskmanager.sh The entry point for the Mesos worker processes. You don't need to explicitly execute this script. It is automatically launched by the Mesos worker node to bring up a new TaskManager. +2. `mesos-taskmanager.sh` + The entry point for the Mesos worker processes. + You don't need to explicitly execute this script. + It is automatically launched by the Mesos worker node to bring up a new TaskManager. + +In order to run the `mesos-appmaster.sh` script you have to define `mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to the Java process. +Additionally, you should define the number of task managers which are started by Mesos via `mesos.initial-tasks`. +This value can also be defined in the `flink-conf.yaml` or passed as a Java property. + +When executing `mesos-appmaster.sh`, it will create a job manager on the machine where you executed the script. +In contrast to that, the task managers will be run as Mesos tasks in the Mesos cluster. + +#### General configuration + +It is possible to completely parameterize a Mesos application through Java properties passed to the Mesos application master. +This also allows to specify general Flink configuration parameters. +For example: + + bin/mesos-appmaster.sh \ + -Dmesos.master=master.foobar.org:5050 + -Djobmanager.heap.mb=1024 \ + -Djobmanager.rpc.port=6123 \ + -Djobmanager.web.port=8081 \ + -Dmesos.initial-tasks=10 \ + -Dmesos.resourcemanager.tasks.mem=4096 \ + -Dtaskmanager.heap.mb=3500 \ + -Dtaskmanager.numberOfTaskSlots=2 \ + -Dparallelism.default=10 High Availability -You will need to run a service like Marathon or Apache Aurora which takes care -of restarting the Flink master process in case of node or process failures. In -addition, Zookeeper needs to be configured like described in the - [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) +You will need to run a service like Marathon or Apache Aurora which takes care of restarting the Flink master process in case of node or process failures. +In addition, Zookeeper needs to be configured like described in the [High Availability section of the Flink docs] ({{ site.baseurl }}/setup/jobmanager_high_availability.html) -For the reconciliation of tasks to work correctly, please also set -`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. +For the reconciliation of tasks to work correctly, please also set `recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path. Marathon -Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. In -particular, it should also adjust any configuration parameters for the Flink -cluster. +Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. +In particular, it should also adjust any configuration parameters for the Flink cluster. Here is an example configuration for Marathon: { - "id": "basic-0", - "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue -DanotherEntry=anotherValue ...", + "id": "flink", + "cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 -Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 -Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 -Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 -Dmesos.resourcemanager.tasks.cpus=1", "cpus": 1.0, - "mem": 2048, + "mem": 1024 } +When running Flink with Marathon, the whole Flink cluster including the job manager will be run as Mesos tasks in the Mesos cluster. + Configuration parameters Mesos configuration entries +`mesos.initial-tasks`: The initial workers to bring up when the master starts (* DEFAULT *: The number of workers specified at cluster startup). -`mesos.initial-tasks`: The initial workers to bring up when the master starts (* DEFAULT *: The number of workers specified at cluster startup). - -`mesos.maximum-failed-tasks`: The maximum number of failed workers before the cluster fails (* DEFAULT *: Number of initial workers) May be set to -1 to disable this feature. +`mesos.maximum-failed-tasks`: The maximum number of failed workers before the cluster fails (* DEFAULT *: Number of initial workers). +May be set to -1 to disable this feature. -`mesos.master`: The Mesos master URL. The value should be in one of the following forms: host:port, zk://host1:port1,host2:port2,.../path, zk://username:password@host1:port1,host2:port2,.../path, file:///path/to/file (where file contains one of the above) +`mesos.master`: The Mesos master URL. The value should be in one of the following forms: host:port, zk://host1:port1,host2:port2,.../path, zk://username:password@host1:port1,host2:port2,.../path, file:///path/to/file (where file contains one of the above) End diff – Yes that is a good idea. Will change it accordingly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on the issue:

          https://github.com/apache/flink/pull/3236

          Thanks for your review @tzulitai. I've addressed your comments. Merging this PR now.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3236 Thanks for your review @tzulitai. I've addressed your comments. Merging this PR now.
          Hide
          till.rohrmann Till Rohrmann added a comment -

          1.3.0: f8f9cae191a5504450cfc1b141b114641c2216b3
          1.2.0: 8d81bfdaa2cd18ca9ef84c1618fc8c94e0f0e199

          Show
          till.rohrmann Till Rohrmann added a comment - 1.3.0: f8f9cae191a5504450cfc1b141b114641c2216b3 1.2.0: 8d81bfdaa2cd18ca9ef84c1618fc8c94e0f0e199
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3236

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3236

            People

            • Assignee:
              till.rohrmann Till Rohrmann
              Reporter:
              till.rohrmann Till Rohrmann
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development