Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5074

Implement a RunningJobRegistry based on Zookeeper

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Cluster Management
    • Labels:

      Description

      For flip-6, it has implemented the ZookeeperHaServices, but ZookeeperHaServices does not support getRunningJobsRegistry. So need to implement a ZK based running job registry.

        Issue Links

          Activity

          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed in 544f53467b901e6e891a23fc4f2ef3a6be229718

          Show
          StephanEwen Stephan Ewen added a comment - Fixed in 544f53467b901e6e891a23fc4f2ef3a6be229718
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/2903

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2903
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/2903

          I think this looks good, thanks!
          Merging this...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2903 I think this looks good, thanks! Merging this...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user shuai-xu commented on the issue:

          https://github.com/apache/flink/pull/2903

          @StephanEwen I have modified according to your comments. Can this pr be merged, I have a new jira need to based on this one. Thank you very much

          Show
          githubbot ASF GitHub Bot added a comment - Github user shuai-xu commented on the issue: https://github.com/apache/flink/pull/2903 @StephanEwen I have modified according to your comments. Can this pr be merged, I have a new jira need to based on this one. Thank you very much
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user shuai-xu reopened a pull request:

          https://github.com/apache/flink/pull/2903

          FLINK-5074 [runtime] add a zookeeper based running job registry

          This pr is for jira #[5074](https://issues.apache.org/jira/browse/FLINK-5074).

          Implement a zookeeper based running job registry for flip-6. It will write the running job info to zookeeper and support querying job status.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/shuai-xu/flink jira-5074

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/2903.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2903


          commit 18d9e17f10cf98d9fe351da181cae57b52957f6f
          Author: shuai.xus <shuai.xus@alibaba-inc.com>
          Date: 2016-11-30T08:05:55Z

          FLINK-5704 [runtime] add a zookeeper based running job registry

          commit 5a623063e2f9a39e6e0eb32319faa7e9d564cf3d
          Author: shuai.xus <shuai.xus@alibaba-inc.com>
          Date: 2016-12-01T07:27:11Z

          FLINK-5074 [runtime] HighAvailabilityServicesUtils support new ZookeeperHaServices

          commit 2f42ec657eb69197ef5280c114d486ccd1271c44
          Author: shuai.xus <shuai.xus@alibaba-inc.com>
          Date: 2016-12-05T06:12:39Z

          FLINK-5074 [runtime] fix get zk path bug and add unittest

          commit 0fa9a8d7e4d31451f72177f93e3fba3ca5ad3593
          Author: shuai.xus <shuai.xus@alibaba-inc.com>
          Date: 2016-12-21T09:44:44Z

          FLINK-5074 [runtime] add throws Exception to before and after


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user shuai-xu reopened a pull request: https://github.com/apache/flink/pull/2903 FLINK-5074 [runtime] add a zookeeper based running job registry This pr is for jira # [5074] ( https://issues.apache.org/jira/browse/FLINK-5074 ). Implement a zookeeper based running job registry for flip-6. It will write the running job info to zookeeper and support querying job status. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shuai-xu/flink jira-5074 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2903.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2903 commit 18d9e17f10cf98d9fe351da181cae57b52957f6f Author: shuai.xus <shuai.xus@alibaba-inc.com> Date: 2016-11-30T08:05:55Z FLINK-5704 [runtime] add a zookeeper based running job registry commit 5a623063e2f9a39e6e0eb32319faa7e9d564cf3d Author: shuai.xus <shuai.xus@alibaba-inc.com> Date: 2016-12-01T07:27:11Z FLINK-5074 [runtime] HighAvailabilityServicesUtils support new ZookeeperHaServices commit 2f42ec657eb69197ef5280c114d486ccd1271c44 Author: shuai.xus <shuai.xus@alibaba-inc.com> Date: 2016-12-05T06:12:39Z FLINK-5074 [runtime] fix get zk path bug and add unittest commit 0fa9a8d7e4d31451f72177f93e3fba3ca5ad3593 Author: shuai.xus <shuai.xus@alibaba-inc.com> Date: 2016-12-21T09:44:44Z FLINK-5074 [runtime] add throws Exception to before and after
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user shuai-xu closed the pull request at:

          https://github.com/apache/flink/pull/2903

          Show
          githubbot ASF GitHub Bot added a comment - Github user shuai-xu closed the pull request at: https://github.com/apache/flink/pull/2903
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user shuai-xu commented on the issue:

          https://github.com/apache/flink/pull/2903

          @StephanEwen done

          Show
          githubbot ASF GitHub Bot added a comment - Github user shuai-xu commented on the issue: https://github.com/apache/flink/pull/2903 @StephanEwen done
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on a diff in the pull request:

          https://github.com/apache/flink/pull/2903#discussion_r92608715

          — Diff: flink-runtime/src/test/java/org/apache/flink/runtime/highavailability/ZooKeeperRegistryTest.java —
          @@ -0,0 +1,87 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.flink.runtime.highavailability;
          +
          +import org.apache.curator.test.TestingServer;
          +import org.apache.flink.api.common.JobID;
          +import org.apache.flink.configuration.Configuration;
          +import org.apache.flink.configuration.HighAvailabilityOptions;
          +import org.apache.flink.util.TestLogger;
          +import org.junit.After;
          +import org.junit.Before;
          +import org.junit.Test;
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import static org.junit.Assert.*;
          +
          +public class ZooKeeperRegistryTest extends TestLogger {
          + private TestingServer testingServer;
          +
          + private static Logger LOG = LoggerFactory.getLogger(ZooKeeperRegistryTest.class);
          +
          + @Before
          + public void before() {
          + try

          { + testingServer = new TestingServer(); + }

          catch (Exception e) {
          — End diff –

          I would just add `throws Exception` to the method signature here, rather than catching and wrapping the exception. Makes error stack traces easier to read.

          As a general rule: I think one should never need to explicitly throw a `RuntimeException`.
          (I know Flink has a few places where it does it, but these are in my opinion errors from the past, during the time when we were still learning)

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2903#discussion_r92608715 — Diff: flink-runtime/src/test/java/org/apache/flink/runtime/highavailability/ZooKeeperRegistryTest.java — @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.highavailability; + +import org.apache.curator.test.TestingServer; +import org.apache.flink.api.common.JobID; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.configuration.HighAvailabilityOptions; +import org.apache.flink.util.TestLogger; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static org.junit.Assert.*; + +public class ZooKeeperRegistryTest extends TestLogger { + private TestingServer testingServer; + + private static Logger LOG = LoggerFactory.getLogger(ZooKeeperRegistryTest.class); + + @Before + public void before() { + try { + testingServer = new TestingServer(); + } catch (Exception e) { — End diff – I would just add `throws Exception` to the method signature here, rather than catching and wrapping the exception. Makes error stack traces easier to read. As a general rule: I think one should never need to explicitly throw a `RuntimeException`. (I know Flink has a few places where it does it, but these are in my opinion errors from the past, during the time when we were still learning)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/2903

          Looks good to me!

          I think it would be good to add a test to this, based on a Curator TestingServer.
          This test here gives an example how to use the TestingServer:
          https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/ZooKeeperLeaderElectionTest.java

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2903 Looks good to me! I think it would be good to add a test to this, based on a Curator TestingServer. This test here gives an example how to use the TestingServer: https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/ZooKeeperLeaderElectionTest.java

            People

            • Assignee:
              tiemsn shuai.xu
              Reporter:
              tiemsn shuai.xu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development