Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-7241

Fix YARN high availability documentation

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0, 1.3.1
    • Fix Version/s: 1.4.0, 1.3.2
    • Component/s: Documentation, YARN
    • Labels:
      None

      Description

      The documentation (jobmanager_high_availability.md) incorrectly suggests this configuration template when running on YARN:

      high-availability: zookeeper
      high-availability.zookeeper.quorum: localhost:2181
      high-availability.zookeeper.storageDir: hdfs:///flink/recovery
      high-availability.zookeeper.path.root: /flink
      high-availability.zookeeper.path.namespace: /cluster_one # important: customize per cluster
      yarn.application-attempts: 10
      

      while above it says that the namespace should not be set on YARN because it will be automatically generated.

      Also, the documentation still refers to namespace while this has been renamed to cluster-id.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha closed the pull request at:

          https://github.com/apache/flink/pull/4382

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/4382
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/4382

          Merged

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4382 Merged
          Hide
          aljoscha Aljoscha Krettek added a comment -

          Fixed on master in
          a671035fa4e52cdf3540fa865473aa9404a841c4
          d628e7a7b45aa3d82a9195dc227c1067151d9c99
          640cc4ee600d83cdcadabbf4193ad016d36bf4c8

          Fixed on release-1.3 in
          6fa4383ba2d5d21c43cd21871fe5c505ce261311
          1810d200a3c32a7dd27b9de4dcadfe9fed83e7b8
          7180ceb2dd3f96526019914b6101ceee845be270

          Show
          aljoscha Aljoscha Krettek added a comment - Fixed on master in a671035fa4e52cdf3540fa865473aa9404a841c4 d628e7a7b45aa3d82a9195dc227c1067151d9c99 640cc4ee600d83cdcadabbf4193ad016d36bf4c8 Fixed on release-1.3 in 6fa4383ba2d5d21c43cd21871fe5c505ce261311 1810d200a3c32a7dd27b9de4dcadfe9fed83e7b8 7180ceb2dd3f96526019914b6101ceee845be270
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user aljoscha commented on the issue:

          https://github.com/apache/flink/pull/4382

          Thanks 👌 Will update

          Show
          githubbot ASF GitHub Bot added a comment - Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4382 Thanks 👌 Will update
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/4382

          I would go a bit further in updating this:

          • The ZooKeeper root node has a default value of `/flink` (I think). You only need to set this when you want to organize your ZooKeeper node tree in a specific way.
          • The cluster-id docs should be changed to say that you do not need to set this manually in Yarn / Mesos mode and that it generates a new subtree per launch. You only need to set this if you want to manually recover an earlier HA job (which is different from re-launching the job from a savepoint or externalized checkpoint)
          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/4382 I would go a bit further in updating this: The ZooKeeper root node has a default value of `/flink` (I think). You only need to set this when you want to organize your ZooKeeper node tree in a specific way. The cluster-id docs should be changed to say that you do not need to set this manually in Yarn / Mesos mode and that it generates a new subtree per launch. You only need to set this if you want to manually recover an earlier HA job (which is different from re-launching the job from a savepoint or externalized checkpoint)
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user aljoscha opened a pull request:

          https://github.com/apache/flink/pull/4382

          FLINK-7241 Fix YARN high availability documentation

          R: @rmetzger (or anyone really)

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/aljoscha/flink jira-7241-fix-yarn-ha-doc

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/4382.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #4382


          commit e9e9c50d19bb0d6a69aef791cd189ec40217f0ec
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-07-21T15:45:24Z

          FLINK-7241 Rename namespace to cluster-id in HA doc

          This was renamed a while ago and namespace is now deprecated.

          commit 6b15600baa706cf89a585bd0a1bb5cd3e0df8969
          Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
          Date: 2017-07-21T15:46:25Z

          FLINK-7241 Remove cluster-id from YARN HA setup template in doc

          When running a YARN cluster the cluster-id is automatically derived.
          Acutally setting a cluster-id can lead to problems if you start multiple
          per-job YARN sessions with the same cluster-id by accident.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/4382 FLINK-7241 Fix YARN high availability documentation R: @rmetzger (or anyone really) You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink jira-7241-fix-yarn-ha-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4382.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4382 commit e9e9c50d19bb0d6a69aef791cd189ec40217f0ec Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-07-21T15:45:24Z FLINK-7241 Rename namespace to cluster-id in HA doc This was renamed a while ago and namespace is now deprecated. commit 6b15600baa706cf89a585bd0a1bb5cd3e0df8969 Author: Aljoscha Krettek <aljoscha.krettek@gmail.com> Date: 2017-07-21T15:46:25Z FLINK-7241 Remove cluster-id from YARN HA setup template in doc When running a YARN cluster the cluster-id is automatically derived. Acutally setting a cluster-id can lead to problems if you start multiple per-job YARN sessions with the same cluster-id by accident.

            People

            • Assignee:
              aljoscha Aljoscha Krettek
              Reporter:
              aljoscha Aljoscha Krettek
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development