Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5941

Let handlers take part in job archiving

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Webfrontend
    • Labels:
      None

      Description

      The key idea behind the HistoryServer is to pre-compute all JSON responses which the WebFrontend could request and store them as files in a directory structure resembling the REST-API.

      For this require a mechanism to generate the responses and their corresponding REST URL.

      FLINK-5852 made it easier to re-use the JSON generation code, while FLINK-5870 made handlers aware of the REST URLs that they are registered one.

      The aim of this JIRA is to extend job-related handlers, building on the above JIRAs, enabling them to generate a number of (Path, Json) pairs for a given ExecutionGraph, containing all responses that they could generate for the given graph and their respective REST URL..

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3444

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3444
          Hide
          Zentol Chesnay Schepler added a comment -

          Implemented in 7fe0eb477df52cfd7254695a67d41f3cba34ef0a.

          Show
          Zentol Chesnay Schepler added a comment - Implemented in 7fe0eb477df52cfd7254695a67d41f3cba34ef0a.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on the issue:

          https://github.com/apache/flink/pull/3444

          merging.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on the issue: https://github.com/apache/flink/pull/3444 merging.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on the issue:

          https://github.com/apache/flink/pull/3444

          Thanks! +1 to merge

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on the issue: https://github.com/apache/flink/pull/3444 Thanks! +1 to merge
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on the issue:

          https://github.com/apache/flink/pull/3444

          @uce I've addresse all your comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on the issue: https://github.com/apache/flink/pull/3444 @uce I've addresse all your comments.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103918560

          — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/CurrentJobsOverviewHandler.java —
          @@ -121,6 +125,26 @@ public String handleJsonRequest(Map<String, String> pathParams, Map<String, Stri
          }
          }

          + public static class CurrentJobsOverviewArchiver implements Archiver {
          +
          + @Override
          + public ArchivedJson[] archiveJsonWithPath(AccessExecutionGraph graph) throws IOException {
          — End diff –

          OK sorry I was confused by the manual JSON creation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103918560 — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/CurrentJobsOverviewHandler.java — @@ -121,6 +125,26 @@ public String handleJsonRequest(Map<String, String> pathParams, Map<String, Stri } } + public static class CurrentJobsOverviewArchiver implements Archiver { + + @Override + public ArchivedJson[] archiveJsonWithPath(AccessExecutionGraph graph) throws IOException { — End diff – OK sorry I was confused by the manual JSON creation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103918296

          — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebRuntimeMonitor.java —
          @@ -424,6 +425,33 @@ protected void initChannel(SocketChannel ch)

          { LOG.info("Web frontend listening at " + address + ':' + port); }

          + public static Archiver[] getArchivers() {
          — End diff –

          I see... changing that will probably be an unrelated refactoring. So let's leave it as is and add a comment. I think it won't even hurt to add a note why it is a static method etc.

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103918296 — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebRuntimeMonitor.java — @@ -424,6 +425,33 @@ protected void initChannel(SocketChannel ch) { LOG.info("Web frontend listening at " + address + ':' + port); } + public static Archiver[] getArchivers() { — End diff – I see... changing that will probably be an unrelated refactoring. So let's leave it as is and add a comment. I think it won't even hurt to add a note why it is a static method etc.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103917220

          — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/CurrentJobsOverviewHandler.java —
          @@ -121,6 +125,26 @@ public String handleJsonRequest(Map<String, String> pathParams, Map<String, Stri
          }
          }

          + public static class CurrentJobsOverviewArchiver implements Archiver {
          +
          + @Override
          + public ArchivedJson[] archiveJsonWithPath(AccessExecutionGraph graph) throws IOException {
          — End diff –

          I don't quite understand, we are using it? The archiver follows the code path of the handler in case ```includeRunningJobs``` and ```includeFinishedJobs``` are both true, since this is the only path that the WebFrontend actually calls.

          I don't remember what in the end cause the job status to not appear, it is fixed now though.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103917220 — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/CurrentJobsOverviewHandler.java — @@ -121,6 +125,26 @@ public String handleJsonRequest(Map<String, String> pathParams, Map<String, Stri } } + public static class CurrentJobsOverviewArchiver implements Archiver { + + @Override + public ArchivedJson[] archiveJsonWithPath(AccessExecutionGraph graph) throws IOException { — End diff – I don't quite understand, we are using it? The archiver follows the code path of the handler in case ```includeRunningJobs``` and ```includeFinishedJobs``` are both true, since this is the only path that the WebFrontend actually calls. I don't remember what in the end cause the job status to not appear, it is fixed now though.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103916234

          — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebRuntimeMonitor.java —
          @@ -424,6 +425,33 @@ protected void initChannel(SocketChannel ch)

          { LOG.info("Web frontend listening at " + address + ':' + port); }

          + public static Archiver[] getArchivers() {
          — End diff –

          i wanted to do something like this originally. However, since the archiving is done in the ``┬┤MemoryArchivist``` i would have to pass the ```WebRuntimeMonitor``` reference to it. This is problematic since you now add an order in which they have to created. This specifically caused issues with the Yarn/Mesos runner that create the JobManager before the WebRuntimeMonitor.

          It's a stupid problem and i agree that the static method approach is rather brittle, but i couldn't find a good solution for the above issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103916234 — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebRuntimeMonitor.java — @@ -424,6 +425,33 @@ protected void initChannel(SocketChannel ch) { LOG.info("Web frontend listening at " + address + ':' + port); } + public static Archiver[] getArchivers() { — End diff – i wanted to do something like this originally. However, since the archiving is done in the ``┬┤MemoryArchivist``` i would have to pass the ```WebRuntimeMonitor``` reference to it. This is problematic since you now add an order in which they have to created. This specifically caused issues with the Yarn/Mesos runner that create the JobManager before the WebRuntimeMonitor. It's a stupid problem and i agree that the static method approach is rather brittle, but i couldn't find a good solution for the above issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103903528

          — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/history/ArchivedJson.java —
          @@ -0,0 +1,28 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.flink.runtime.webmonitor.history;
          +
          +public class ArchivedJson {
          + public final String path;
          + public final String json;
          +
          + public ArchivedJson(String path, String json) {
          — End diff –

          Could you add `checkNotNull` checks here? And we usually have getters instead of public fields. Would be nice to follow that convention (of course opinions will differ about the benefit of this).

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103903528 — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/history/ArchivedJson.java — @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.runtime.webmonitor.history; + +public class ArchivedJson { + public final String path; + public final String json; + + public ArchivedJson(String path, String json) { — End diff – Could you add `checkNotNull` checks here? And we usually have getters instead of public fields. Would be nice to follow that convention (of course opinions will differ about the benefit of this).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103903740

          — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/history/Archiver.java —
          @@ -0,0 +1,26 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.flink.runtime.webmonitor.history;
          +
          +import org.apache.flink.runtime.executiongraph.AccessExecutionGraph;
          +
          +import java.io.IOException;
          +
          +public interface Archiver {
          + ArchivedJson[] archiveJsonWithPath(AccessExecutionGraph graph) throws IOException;
          — End diff –

          I think we can get rid of most toArray conversions by returning `Collection<ArchivedJson>` here. And also let's add Javadocs please.

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103903740 — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/history/Archiver.java — @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.runtime.webmonitor.history; + +import org.apache.flink.runtime.executiongraph.AccessExecutionGraph; + +import java.io.IOException; + +public interface Archiver { + ArchivedJson[] archiveJsonWithPath(AccessExecutionGraph graph) throws IOException; — End diff – I think we can get rid of most toArray conversions by returning `Collection<ArchivedJson>` here. And also let's add Javadocs please.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103903655

          — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/history/Archiver.java —
          @@ -0,0 +1,26 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.flink.runtime.webmonitor.history;
          +
          +import org.apache.flink.runtime.executiongraph.AccessExecutionGraph;
          +
          +import java.io.IOException;
          +
          +public interface Archiver {
          — End diff –

          I would call this `JsonArchiver` or something that gives it more context. Or it may become obsolete with the suggestions for the `ArchivableHandler`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103903655 — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/history/Archiver.java — @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.runtime.webmonitor.history; + +import org.apache.flink.runtime.executiongraph.AccessExecutionGraph; + +import java.io.IOException; + +public interface Archiver { — End diff – I would call this `JsonArchiver` or something that gives it more context. Or it may become obsolete with the suggestions for the `ArchivableHandler`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103901870

          — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebRuntimeMonitor.java —
          @@ -424,6 +425,33 @@ protected void initChannel(SocketChannel ch)

          { LOG.info("Web frontend listening at " + address + ':' + port); }

          + public static Archiver[] getArchivers() {
          — End diff –

          What do you think about making this "dynamic" and moving it to the handlers. This would of course only work if this method is not static (which should not be a problem if the JobManager does the archiving).

          The handlers implement an interface:

          ```java
          interface ArchivableJsonHandler

          { ArchivedJson[] createArchivedJsons(AccessExecutionGraph eg); }

          ```

          The router helpers that register the handlers (`GET()`, etc.) check:
          ```java
          if (handler instanceof ArchivableHandler)

          { archivable.add(handler.getJsonArchiver()); }

          ```

          This methods then bcomes:
          ```java
          public Collection<ArchivableJsonHandler> getAchivableHanlders()

          { return archivable; }

          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103901870 — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/WebRuntimeMonitor.java — @@ -424,6 +425,33 @@ protected void initChannel(SocketChannel ch) { LOG.info("Web frontend listening at " + address + ':' + port); } + public static Archiver[] getArchivers() { — End diff – What do you think about making this "dynamic" and moving it to the handlers. This would of course only work if this method is not static (which should not be a problem if the JobManager does the archiving). The handlers implement an interface: ```java interface ArchivableJsonHandler { ArchivedJson[] createArchivedJsons(AccessExecutionGraph eg); } ``` The router helpers that register the handlers (`GET()`, etc.) check: ```java if (handler instanceof ArchivableHandler) { archivable.add(handler.getJsonArchiver()); } ``` This methods then bcomes: ```java public Collection<ArchivableJsonHandler> getAchivableHanlders() { return archivable; } ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103902087

          — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/CurrentJobsOverviewHandler.java —
          @@ -121,6 +125,26 @@ public String handleJsonRequest(Map<String, String> pathParams, Map<String, Stri
          }
          }

          + public static class CurrentJobsOverviewArchiver implements Archiver {
          +
          + @Override
          + public ArchivedJson[] archiveJsonWithPath(AccessExecutionGraph graph) throws IOException {
          — End diff –

          Why don't we use `writeJobDetailOverviewAsJson` here? Maybe was this the problem before with not listing the final job status?

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103902087 — Diff: flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/CurrentJobsOverviewHandler.java — @@ -121,6 +125,26 @@ public String handleJsonRequest(Map<String, String> pathParams, Map<String, Stri } } + public static class CurrentJobsOverviewArchiver implements Archiver { + + @Override + public ArchivedJson[] archiveJsonWithPath(AccessExecutionGraph graph) throws IOException { — End diff – Why don't we use `writeJobDetailOverviewAsJson` here? Maybe was this the problem before with not listing the final job status?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user uce commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3444#discussion_r103903357

          — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/history/ArchivedJson.java —
          @@ -0,0 +1,28 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.flink.runtime.webmonitor.history;
          +
          +public class ArchivedJson {
          — End diff –

          Could you please add a high level comment about what this is, why it exists and where it is used?

          Show
          githubbot ASF GitHub Bot added a comment - Github user uce commented on a diff in the pull request: https://github.com/apache/flink/pull/3444#discussion_r103903357 — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/history/ArchivedJson.java — @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.runtime.webmonitor.history; + +public class ArchivedJson { — End diff – Could you please add a high level comment about what this is, why it exists and where it is used?
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zentol opened a pull request:

          https://github.com/apache/flink/pull/3444

          FLINK-5941 Integrate Archiver pattern into handlers

          This PR adds the last piece to the foundation of the HistoryServer. It builds on top of #3365 and #3376, meaning that only 58649fb is relevant for this PR.

          The key addition is the ```Archiver``` interface. An ```Archiver``` generates Json responses and their respective REST paths for a given ```AccessExecutionGraph```.

          The goal here is to allow writing these responses to files contained in a directory structure resembling the REST API, without defining the structure in 2 different places. With the additions made here the writing of files can be done in a simple loop:
          ```
          AccessExecutiongraph graph = ...
          for (Archiver archiver : WebRuntimeMonitor.getArchivers()) {
          for (ArchivedJson archive : archiver.archive(graph))

          { File f = createFile(basePath, archive.path); writeJson(f, archive.json); }

          }
          ```

          Instead of extending existing job-related handlers I've added a new class in each of them that implements the new interface. This makes testing easier and also allows for easier refactoring/relocation.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zentol/flink 5941_handler_archiver

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3444.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3444


          commit 61688e45a461da717c220891b60f3329f202ad7a
          Author: zentol <chesnay@apache.org>
          Date: 2017-03-01T11:23:15Z

          FLINK-5870 Handlers define REST URLs

          commit 8eb29ab3b6ef87fe9a1b00ad5ac08b6e7725c2f4
          Author: zentol <chesnay@apache.org>
          Date: 2017-02-28T17:01:14Z

          FLINK-5852 Move handler JSON generation code into static methods

          commit 58649fb99370b896c0c7bdcc81b674c7b7a81c56
          Author: zentol <chesnay@apache.org>
          Date: 2017-02-20T15:30:35Z

          FLINK-5941 Integrate Archiver pattern into handlers


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/3444 FLINK-5941 Integrate Archiver pattern into handlers This PR adds the last piece to the foundation of the HistoryServer. It builds on top of #3365 and #3376, meaning that only 58649fb is relevant for this PR. The key addition is the ```Archiver``` interface. An ```Archiver``` generates Json responses and their respective REST paths for a given ```AccessExecutionGraph```. The goal here is to allow writing these responses to files contained in a directory structure resembling the REST API, without defining the structure in 2 different places. With the additions made here the writing of files can be done in a simple loop: ``` AccessExecutiongraph graph = ... for (Archiver archiver : WebRuntimeMonitor.getArchivers()) { for (ArchivedJson archive : archiver.archive(graph)) { File f = createFile(basePath, archive.path); writeJson(f, archive.json); } } ``` Instead of extending existing job-related handlers I've added a new class in each of them that implements the new interface. This makes testing easier and also allows for easier refactoring/relocation. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 5941_handler_archiver Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3444.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3444 commit 61688e45a461da717c220891b60f3329f202ad7a Author: zentol <chesnay@apache.org> Date: 2017-03-01T11:23:15Z FLINK-5870 Handlers define REST URLs commit 8eb29ab3b6ef87fe9a1b00ad5ac08b6e7725c2f4 Author: zentol <chesnay@apache.org> Date: 2017-02-28T17:01:14Z FLINK-5852 Move handler JSON generation code into static methods commit 58649fb99370b896c0c7bdcc81b674c7b7a81c56 Author: zentol <chesnay@apache.org> Date: 2017-02-20T15:30:35Z FLINK-5941 Integrate Archiver pattern into handlers

            People

            • Assignee:
              Zentol Chesnay Schepler
              Reporter:
              Zentol Chesnay Schepler
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development