Uploaded image for project: 'Apache Twill'
  1. Apache Twill
  2. TWILL-230

Retrieval of ResourceReport via TwillController may fail

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      When starting a new YarnTwillRunnerService, it discovers all existing applications and get the tracker URL from the application report. However, the operation might fail if the application was started with a different user than the one who starts the YarnTwillRunnerService in a Kerberos cluster, resulting in failure to get resource report.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user chtyim opened a pull request:

          https://github.com/apache/twill/pull/53

          (TWILL-230) Get resource report based on the caller user

          • Also by default get the resource report from the tracking url, then fall back to the original tracking url.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/chtyim/twill feature/TWILL-230

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/twill/pull/53.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #53


          commit 239619d8a1ccb08dae3c3262f380b87759a574f6
          Author: Terence Yim <chtyim@apache.org>
          Date: 2017-04-03T23:49:52Z

          (TWILL-230) Get resource report based on the caller user

          • Also by default get the resource report from the tracking url, then fall back to the original tracking url.

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user chtyim opened a pull request: https://github.com/apache/twill/pull/53 ( TWILL-230 ) Get resource report based on the caller user Also by default get the resource report from the tracking url, then fall back to the original tracking url. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chtyim/twill feature/ TWILL-230 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/twill/pull/53.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #53 commit 239619d8a1ccb08dae3c3262f380b87759a574f6 Author: Terence Yim <chtyim@apache.org> Date: 2017-04-03T23:49:52Z ( TWILL-230 ) Get resource report based on the caller user Also by default get the resource report from the tracking url, then fall back to the original tracking url.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/53#discussion_r109552373

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillController.java —
          @@ -322,7 +314,46 @@ private boolean hasRun(YarnApplicationState state) {

          @Override
          public ResourceReport getResourceReport() {

          • // in case the user calls this before starting, return null
            + // Only has resource report if the app is running.
            + if (state() != State.RUNNING) { + return null; + }

            + ResourceReportClient resourcesClient = getResourcesClient();
            return (resourcesClient == null) ? null : resourcesClient.get();
            }
            +
            + /**
            + * Returns the

            {@link ResourceReportClient}

            for fetching resource report from the AM.
            + * It first consults the RM for the tracking URL and get the resource report from there.
            + */
            + @Nullable
            + private ResourceReportClient getResourcesClient() {
            + YarnApplicationReport report = processController.getReport();
            + List<URL> urls = new ArrayList<>(2);

              • End diff –

          Why will this url list has initial capacity of 2?

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/53#discussion_r109552373 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillController.java — @@ -322,7 +314,46 @@ private boolean hasRun(YarnApplicationState state) { @Override public ResourceReport getResourceReport() { // in case the user calls this before starting, return null + // Only has resource report if the app is running. + if (state() != State.RUNNING) { + return null; + } + ResourceReportClient resourcesClient = getResourcesClient(); return (resourcesClient == null) ? null : resourcesClient.get(); } + + /** + * Returns the {@link ResourceReportClient} for fetching resource report from the AM. + * It first consults the RM for the tracking URL and get the resource report from there. + */ + @Nullable + private ResourceReportClient getResourcesClient() { + YarnApplicationReport report = processController.getReport(); + List<URL> urls = new ArrayList<>(2); End diff – Why will this url list has initial capacity of 2?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/53#discussion_r109553180

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillController.java —
          @@ -322,7 +314,46 @@ private boolean hasRun(YarnApplicationState state) {

          @Override
          public ResourceReport getResourceReport() {

          • // in case the user calls this before starting, return null
            + // Only has resource report if the app is running.
            + if (state() != State.RUNNING) { + return null; + }

            + ResourceReportClient resourcesClient = getResourcesClient();
            return (resourcesClient == null) ? null : resourcesClient.get();
            }
            +
            + /**
            + * Returns the

            {@link ResourceReportClient}

            for fetching resource report from the AM.
            + * It first consults the RM for the tracking URL and get the resource report from there.
            + */
            + @Nullable
            + private ResourceReportClient getResourcesClient() {
            + YarnApplicationReport report = processController.getReport();
            + List<URL> urls = new ArrayList<>(2);

              • End diff –

          Because we store the tracking URL and the original tracking URL.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/53#discussion_r109553180 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillController.java — @@ -322,7 +314,46 @@ private boolean hasRun(YarnApplicationState state) { @Override public ResourceReport getResourceReport() { // in case the user calls this before starting, return null + // Only has resource report if the app is running. + if (state() != State.RUNNING) { + return null; + } + ResourceReportClient resourcesClient = getResourcesClient(); return (resourcesClient == null) ? null : resourcesClient.get(); } + + /** + * Returns the {@link ResourceReportClient} for fetching resource report from the AM. + * It first consults the RM for the tracking URL and get the resource report from there. + */ + @Nullable + private ResourceReportClient getResourcesClient() { + YarnApplicationReport report = processController.getReport(); + List<URL> urls = new ArrayList<>(2); End diff – Because we store the tracking URL and the original tracking URL.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anwar6953 commented on the issue:

          https://github.com/apache/twill/pull/53

          LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user anwar6953 commented on the issue: https://github.com/apache/twill/pull/53 LGTM
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anwar6953 commented on the issue:

          https://github.com/apache/twill/pull/53

          LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user anwar6953 commented on the issue: https://github.com/apache/twill/pull/53 LGTM
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/twill/pull/53

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/twill/pull/53

            People

            • Assignee:
              chtyim Terence Yim
              Reporter:
              chtyim Terence Yim
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development