Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6531

Deserialize checkpoint hooks with user classloader

    Details

      Description

      The checkpoint hooks introduced in FLINK-6390 aren't being deserialized with the user classloader, breaking remote execution.

      Remote execution produces a `ClassNotFoundException` as the job graph is transferred from the client to the JobManager.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user EronWright opened a pull request:

          https://github.com/apache/flink/pull/3867

          FLINK-6531 Deserialize checkpoint hooks with user classloader

          • enhance MasterTriggerRestoreHook.Factory::create to take user-code classloader
          • deserialize WithMasterCheckpointHook instance with supplied classloader
          • throw JobExecutionException when hook cannot be instantiated

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/EronWright/flink FLINK-6531

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3867.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3867


          commit 7e94a9be59bff643150c08e73a292436c557cd8a
          Author: Wright, Eron <eron.wright@emc.com>
          Date: 2017-05-10T17:14:54Z

          FLINK-6531

          • enhance MasterTriggerRestoreHook.Factory::create to take user-code classloader
          • deserialize WithMasterCheckpointHook instance with supplied classloader
          • throw JobExecutionException when hook cannot be instantiated

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user EronWright opened a pull request: https://github.com/apache/flink/pull/3867 FLINK-6531 Deserialize checkpoint hooks with user classloader enhance MasterTriggerRestoreHook.Factory::create to take user-code classloader deserialize WithMasterCheckpointHook instance with supplied classloader throw JobExecutionException when hook cannot be instantiated You can merge this pull request into a Git repository by running: $ git pull https://github.com/EronWright/flink FLINK-6531 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3867.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3867 commit 7e94a9be59bff643150c08e73a292436c557cd8a Author: Wright, Eron <eron.wright@emc.com> Date: 2017-05-10T17:14:54Z FLINK-6531 enhance MasterTriggerRestoreHook.Factory::create to take user-code classloader deserialize WithMasterCheckpointHook instance with supplied classloader throw JobExecutionException when hook cannot be instantiated
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3867

          Thanks for this patch, this is crucial!

          There is a slightly different approach I would suggest, to not assume that the `Factory` is always in the core classpath. I have the code for that almost ready, plus a test. Can open a PR for your review shortly...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3867 Thanks for this patch, this is crucial! There is a slightly different approach I would suggest, to not assume that the `Factory` is always in the core classpath. I have the code for that almost ready, plus a test. Can open a PR for your review shortly...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3867

          @EronWright Here is a slightly modified version of your code: https://github.com/apache/flink/pull/3868

          I added a small test to also validate the class loading.

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3867 @EronWright Here is a slightly modified version of your code: https://github.com/apache/flink/pull/3868 I added a small test to also validate the class loading.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user EronWright closed the pull request at:

          https://github.com/apache/flink/pull/3867

          Show
          githubbot ASF GitHub Bot added a comment - Github user EronWright closed the pull request at: https://github.com/apache/flink/pull/3867
          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed in

          • 1.3.0 via 11a7f466ea14478970723704d4477afea41b4e43
          • 1.4.0 via aa8a90a588b4d72fc585731bea233495f0690364
          Show
          StephanEwen Stephan Ewen added a comment - Fixed in 1.3.0 via 11a7f466ea14478970723704d4477afea41b4e43 1.4.0 via aa8a90a588b4d72fc585731bea233495f0690364

            People

            • Assignee:
              eronwright Eron Wright
              Reporter:
              eronwright Eron Wright
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development